CSCI572 - Homework 5 - Solved

Adding Spell Checking and AutoComplete to Your Search Engine Objectives
o   Experience using a third-party spell program 

o   Developing efficient methods for accomplishing autocomplete 

In the previous document (AutocompleteInSolr.pdf) you saw how to enhance the Solr program with spelling correction and an autocomplete (suggest) function. In this exercise you are asked to use an external spelling correction program in conjunction with Solr and to enhance the autocomplete functionality of Solr. For spelling correction, you may use an existing third-party program adapted to your downloaded files. In the case of autocomplete you will need to enhance your client program that communicates with Solr to deliver autocomplete suggestions to the web interface you created in an earlier homework  Description of the Exercise 

Spelling Correction: in the class lecture you saw a complete spelling correction program developed by Peter Norvig. The program was written in Python. For this exercise you are welcome to use whatever third-party spelling program you wish, or you may even write your own. Since most of you wrote your previous homework client using PhP, you may want to adopt a version of Norvig’s spelling program written in PhP and run it on your server. You can download the PhP version of Norvig’s spelling corrector from here:  

http://www.phpclasses.org/package/4859-PHP-Suggest-corrected-spelling-text-in-pure-PHP.html#download 

(you will have to register at the site before being able to download the software, registration is free)

If you prefer to use Norvig’s program in a different language, a wide variety of implementations can be found at the bottom of this page, http://norvig.com/spell-correct.html 

You should make sure to enhance your spelling correction program with a set of terms that are specific to the news website that you are responsible for. You should make sure that common terms such as climate, election, etc., and the terms used in the queries of homework #4 are handled. Norvig’s spell correction program uses a text file(‘’big.txt”) to get set of words to calculate edit distance. For this you should create your own “big.txt” for your specified news website. You can use any parser (our suggestion - Apache Tika) and Instructions on using apache Tika for this purpose can be found here, (https://tika.apache.org/1.5/gettingstarted.html 

). 

Autocomplete: for the autocomplete portion of the exercise, you will have to modify your client program, so it accepts single character insertions to the text box, and returns a list of completions/suggestions.  

There are several ways to implement the autocomplete functionality while using Solr. One possible way is to use the FuzzyLookupFactory

 (https://lucene.apache.org/solr/guide/6_6/suggester.html) feature of Solr/Lucene. The FuzzyLookupFactory creates suggestions for misspelled words in fields. It assumes that what you’re sending as the suggest.query parameter is the beginning of the suggestion. It will match terms in your index starting with the provided characters. So, if the query is "ca" it will return all the words starting with "ca", e.g.  "california" and “carolina" etc.  For the first character and second character that is entered, some autocomplete suggestions should appear. 

For this to work you need to enable the suggest component as described in the tutorial but add some options.

Note: with respect to specific issues about how spelling corrections are displayed or how autocomplete corrections are displayed you should imitate the way Google handles both. For example, while typing in the search box, the top suggestions should automatically appear and be updated as the user keeps typing. The spellcheck suggestion should appear at the top of the retrieved results. If the word typed is correct no suggestion should appear at the top

 

 Appendix 

Note 1: 

Suggested config change for making 'AND' as default instead of 'OR' for multi-word queries in solr: 

Solr default boolean model uses OR instead of AND. So, if your query is "Elon Musk", then the result will match all pages which either have " Elon " OR " Musk " present and not the entire query " Elon Musk”.To solve this problem, please do as following to set up the standard Query Parser Parameters:  

In solrconfig.xml add this line:   <str name="q.op">AND</str> within this tag: <requestHandler name="/select" class="solr.SearchHandler">  and Inside <lst> default tag within requestHandler tag:  <lst name="defaults"> Remember to reload after editing.  

 



1.      Show result for misspelled word. Just below the text box, you can display the correct spelling which is clickable. Upon clicking the correct word, it should perform a search and display the new results.

2.      Show results for the spell corrected term. Just below the text box, display the spell corrected term and the initial misspelled term. Make the misspelled term clickable. Upon clicking, it should perform a search and display the new results.