-
-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
中英混合句子分詞嗰陣會將所有英文單詞連埋一齊 #32
Comments
呢個技術上應該可以喺 In [1]: import pycantonese
In [2]: pycantonese.pos_tag(['我', '今晚', '會', 'have', 'dinner', 'at', 'home'])
Out[2]:
[('我', 'PRON'),
('今晚', 'ADV'),
('會', 'AUX'),
('have', 'VERB'),
('dinner', 'ADP'),
('at', 'ADV'),
('home', 'VERB')] 因為 pycantonese 專係處理廣東話,噉啲英文嘅 POS tagging 就會唔啱,呢個喺你嘅情況會唔會係大問題? 如果淨係分詞想保留運用空格而暫時唔理標詞問題住先,我可以睇下點做。 |
唔該晒,即係話我喺做 |
如果我冇理解錯嘅話,你嘅意思係咪即係類似呢個做法? In [1]: import pycantonese
In [2]: import itertools
In [3]: user_input = "我今晚會 have dinner at home"
In [4]: list(itertools.chain.from_iterable(pycantonese.segment(x) for x in user_input.split()))
Out[4]: ['我', '今晚', '會', 'have', 'dinner', 'at', 'home'] |
係嘅冇錯,我後尾自己實現咗。只不過想知pycantonese有冇可能順便做到噉?唔使另外再實現。定係話噉樣將啲單詞連起身有其他目的? |
In [1]: import pycantonese
In [2]: pycantonese.segment('我 今晚會') # with an accidental space
Out[2]: ['我', ' ', '今晚', '會'] # note: not the behavior in production now. I saw outputs like this and decided to sanitize the user input by removing all whitespace before applying word segmentataion. Now that I'm looking into the implementation of |
明白嘞,原來係呢個原因。噉至少我哋可以確定,將英文單詞合併成 |
I've just resolved this issue by updating the upstream In [1]: import pycantonese
In [2]: pycantonese.segment("我今晚會 have dinner at home")
Out[2]: ['我', '今晚', '會', 'have', 'dinner', 'at', 'home']
In [3]: pycantonese.segment("我今 晚會 have dinner at home")
Out[3]: ['我', '今晚', '會', 'have', 'dinner', 'at', 'home'] I was thinking of making a new release after resolving this issue, but on second thought, I'm gonna hold it off a bit, because #33 is still up in the air, also because the new Python 3.11 is coming in a month or so and I'd like to wait on its Docker images etc to be available for CI build support. |
I forgot to mention that you've been acknowledged in the readme. Thanks for reporting this issue! |
輸入
輸出係
可以睇到
havedinnerathome
成個變成咗一個動詞。如果想還原句子就做唔到。可唔可以喺保留英文單詞之間空格嘅前提下分詞?The text was updated successfully, but these errors were encountered: