nltk.tokenize.moses no longer available

[petri]

Trying to use finnlem results in error:

Traceback (most recent call last):
  File "test.py", line 1, in <module>
    from model_wrappers import Seq2Seq
  File "/Users/petri/Code/finnlem/src/model_wrappers.py", line 13, in <module>
    from nltk.tokenize.moses import MosesDetokenizer
ModuleNotFoundError: No module named 'nltk.tokenize.moses'

This is because MosesTokenizer has been moved out of NLTK due to licensing issues. See for example
pytorch/text#306 .

Apparently sacremoses package is now recommended as replacement.

[jmyrberg]

Hi Petri,

Sorry for the late reply, and thanks for reporting this. Since this was just a hobby project, I was expecting that something would break up at some point :)

But luckily, I was happy to see that the Turku NLP Group had just recently released a paper, and the related neural parser, which (afaik) uses a similar approach for lemmatization. So, unless you just want to learn how the seq2seq model works here, I would recommend using their parser for actual lemmatization tasks.

Hope this helps!