-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
can eagle support #90
Comments
Only EAGLE-1 supports bs > 1 now. |
Is it because of the dynamic tree attention mechanism that Eagle2 does not support batch size>1? |
EAGLE-2 requires some code modifications to support bs>1, so it temporarily does not support bs>1. |
can you show me an example to use bs = 2 ? I create model and prompts this way but it raise this error input_ids=model.tokenizer([prompt1,prompt2]).input_ids RuntimeError: output with shape [1, 32, 44, 128] doesn't match the broadcast shape [2, 32, 44, 128] |
You can go to the v1 branch and check here: https://github.com/SafeAILab/EAGLE/tree/v1?tab=readme-ov-file#batch-size--1 |
Can you give some hints about how to make EAGLE2 for batch inference? If the tree mask is not the same for different batch, how can we modify this? |
when I inference multi batch, it raise an Error
RuntimeError: output with shape [1, 32, 84, 128] doesn't match the broadcast shape [2, 32, 84, 128]
The text was updated successfully, but these errors were encountered: