[Feature]: Language / Error model: Improve spellchecking

[crazoter]

NLP can be used for two cases:

1. Word suggestions (predicting the next word) e.g. distilbert (potentially tricky as NLP models predict the next token, not the next word. Generating a list of "potential" words will need some engineering.
2. Word corrections (can also complement word suggestions) e.g. textdistance (https://github.com/life4/textdistance). https://stackoverflow.com/questions/1661434/algorithm-wanted-find-all-words-of-a-dictionary-that-are-similar-to-words-in-a or thefuzz https://github.com/seatgeek/thefuzz with the introduction of https://github.com/wolfgarbe/SymSpell, this has been included.

The idea is that word correction can correct the word, but not take the context into account, thus correcting incorrectly typed words wrongly e.g. "i was quit" --> quit instead of quite.

This of course requires the pre-requisites:

1. that I record the context while typing out the sentence.
2. that the words are not prematurely filtered out (this will likely affect performance)

The feature of allowing users to input multiple keys which may not be in correct order also poses an interesting problem which will complicate the search process.

The current solution is simple and efficient but limited in simplifying keystrokes.

The solution will need to balance performance, quality of life and computation resources.

Tasks:

• Add contextual spell checking
• Change dictionary to use word frequency dictionary
• Involve frequency in the ranking of words

reference: #7 (comment)

[crazoter]

• Possible paper for exploration (freq. dictionaries & n-gram conditional probability):
  • https://arxiv.org/pdf/1910.11242.pdf
  • https://paperswithcode.com/paper/a-context-sensitive-real-time-spell-checker
  • https://github.com/gosom/context-spell-correct (will need to port from golang)
  • Wait. this is not even related to real word errors lMAO
• SpaCy, BERT:
  • https://spacy.io/universe/project/contextualspellcheck/.
  • https://github.com/R1j1t/contextualSpellCheck
  • problematic because it doesn't do any contextual spell checking if the word is already correct.
• spark libraries & pipelines:
  • https://medium.com/spark-nlp/applying-context-aware-spell-checking-in-spark-nlp-3c29c46963bc
• symspell doesn't work in this context (it will perform spell checking without context), although this might change depending on the dictionary used

[crazoter]

Real word errors.

RWE.pdf

https://medium.com/analytics-vidhya/real-word-spell-correction-c64a3a02c64d
https://norvig.com/spell-correct.html
https://towardsdatascience.com/spelling-correction-how-to-make-an-accurate-and-fast-corrector-dc6d0bcbba5f

[crazoter]

Extra reference on using symspell
https://stackoverflow.com/questions/66556613/is-there-a-way-to-force-symspell-python-to-return-more-than-one-correction-recom