-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EAGLE-2 is slower than EAGlE-1 #88
Comments
I make a mistake above, the inference speed of EAGLE-2 is 66 and EAGLE-1 is 69. Besides, temperature is 0 |
It is not normal. Can you provide more details (e.g., if you are running something else on your machine, what is your environment, which codes are your running)? Without them, it is hard to debug your code. |
Some details of my environment are listed as follow: I just run the evaluation script gen_ea_answer_vicuna.py as suggested in Readme |
The possible reason is that total_token was not set correctly. |
Thanks for the response. Can you provide me some advise to set total_token? Should I set larger or smaller? |
HI, the benchmark can't record new token correctly, I don't know if that's normal.
|
Overall, the smaller the model and the more powerful the computational capacity, the larger this value should be. |
It is normal for the baseline not to return new tokens. |
Thanks for this great repo. I have test EAGLE-1 and EAGLE-2 on vicuna-7b.
But I found that EAGLE-2 is slower than EAGLE-1, 69 tokens/s and 66 tokens/s respectively.
The inference test is on MT-bentch, using V100 with 32G memory, and batch size is 1.
Is it normal?
Best regards
The text was updated successfully, but these errors were encountered: