[Core tokenization] add_dummy_prefix_space option to help with latest issues
#28010
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Just wanted to say this would be hugely helpful for us over at https://github.com/probcomp/hfppl ! |
|
Likewise the ability to not include an extra SPIECE_UNDERLINE / Llama token 29871 when encoding a word with a space in front ( |
…s into add-prefix-space
There was a problem hiding this comment.
I'll let @Lysandre decide, but instead of following what we do with bloom I'd rather we convert from slow. Bit slower but at least we are sure we use the correct logic.
This is done with a warning.
|
Failing test is unrelated 😉 |
| if add_prefix_space is not None: | ||
| logger.warning_once( | ||
| "You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers" | ||
| ) | ||
| kwargs["from_slow"] = True |
There was a problem hiding this comment.
How long does it take to convert the tokenizer from slow? If it's quick we can move it to info
There was a problem hiding this comment.
Around 10 seconds I believe!
…test issues (#28010) * add add_dummy_prefix_space option to slow * checking kwargs might be better. Should be there for all spm tokenizer IMO * nits * fix copies * more copied * nits * add prefix space * nit * nits * Update src/transformers/convert_slow_tokenizer.py * fix inti * revert wrong styling * fix * nits * style * updates * make sure we use slow tokenizer for conversion instead of looking for the decoder * support llama ast well * update llama tokenizer fast * nits * nits nits nits * update the doc * update * update to fix tests * skip unrelated tailing test * Update src/transformers/convert_slow_tokenizer.py * add proper testing * test decode as well * more testing * format * fix llama test * Apply suggestions from code review
What does this PR do?
Allows users to use
tokenizer.tokenizecontrolling the addition of prefix space. Let's also update fast!fixes #28622