2023-10-07

Dictionary

Tokens

Question 1: What is token

A token is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing

Question 2: What is a type

Class of all token containing the same sequence of characters

Question 3: What is a term

It is a type that is indexed

Question 4: What are some of the problems in tokenization

Question 5: Why would you eliminate stop words?

Question 6: Why is the trend of eliminating stop words dying out?

Question 7: What is the need for normalization?

Question 8: What is an important criterion when doing normalization?

Question 9: Is the normalization for the indexed text and the query the same?

Question 10: The general convention is to convert everything to lowercase. But how would you differentiate between FED and fed? Give an example to support the claim that it is ok to convert everything to lowercase

Question 11: What is the alternative to normalization

Question 12: How would you handle synonyms and homonyms

Question 13: How to deal with spelling mistakes?

Stemming and Lemmatization

Question 14: What is the difference between stemming and lemmatization?

Question 15: What is the most common stemming algorithm for English?

Question 16: Does stemming help?

Skip Pointers

Question 17: What is the tradeoff associated in the number of skip pointers used?

Question 18: What is the general heuristic for skip distance?

Question 19: What is the problem with the $\sqrt{L}$ heuristic?

Question 20: Suppose the skip distance is $n$. How many posting are in between the skip pointer and the destination