-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question about EAGLE-2 #110
Comments
When using top-k, the actual sampling distribution is not q, and the probability of the first draft token being "am" is 1.0. |
@Yipeng1994 I think your question sounds like not for EAGLE-2, it's more like for the speculative sampling which was proposed in https://arxiv.org/pdf/2211.17192 |
I see. One more question: for the remaining tokens, which in this case is "have", does EAGLE-2 resort to naive sampling instead of using the formula
any longer? @Liyuhui-12
Thanks for your suggestion. I will read through it. @yanjunplay |
You can refer to Appendix B of the EAGLE paper https://arxiv.org/abs/2401.15077 for the specific sampling algorithm. It is a recursive version of the original speculative sampling paper https://arxiv.org/pdf/2211.17192 |
EAGLE employs top-k sampling for verification, sampling tokens in order from left to right. Therefore, in the multi-round speculative sampling (EAGLE Appendix B method) for loop (i ≤ k), Is this correct? |
How to prove that EAGLE-2 ensures that the distribution of the generated text remains unchange?
Suppose you use a draft model that is identical to the target model. We have a distribution p(.|.) for the target model and q(.|.) for the draft model.
Suppose the probability for the next token after "I" is "am" (0.51) and "have" (0.49). After top-k and reranking, we attempt to accept the first token "am" with a probability:
In this scenario, the target model has a probability of 0.49 to accept the token "have." However, when EAGLE-2 is applied, the probability of accepting "have" becomes 0, illustrating how EAGLE-2 modifies the sampling behavior of the target model.
It seems that EAGLE-2 turns the sampling strategy of the target model into the greedy sampling strategy of the draft model.
I am uncertain if I have overlooked any details in the paper that would clarify this conclusion. Could you assist me in understanding the precise mechanics of how EAGLE-2 operates to preserve the distribution while implementing top-k filtering and reranking? Your insight would be greatly appreciated in elucidating this aspect of EAGLE-2's functionality.
The text was updated successfully, but these errors were encountered: