Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

difference between the target input_ids and original LLMs in function getkacc() #70

Open
kafkayu opened this issue May 7, 2024 · 0 comments

Comments

@kafkayu
Copy link

kafkayu commented May 7, 2024

image
Thanks for your excellent work.I am confused about the target input_ids mentioned above /eagle/train/main.py .It seems that the target input_ids are sourced from dataset instead of the target LLM.
image
But I find that function getkacc() calculates the draft token acc with the hidden_states of target input_ids directly. For example ,
the dataset input_ids is "Deep learning is a powerful approach to machine learning" ,
the orginal LLM output is " Deep learning is a subfield of machine learning" .
so the target input_ids in dataset is calculated as follows:
"Deep learning is a powerful approach to machine learning" -> "learning is a subfield approach to machine learning"(target input_ids).
It should be noted that the next token of "Deep learning is a powerful" in original LLM is "approach", hypothetically.

the EAGLE draft prediction is as follows in getkacc() :
"Deep learning is a"-> "subfield"
"Deep learning is a subfield"-> "approach"
image

In fact , the real EAGLE inference in original LLM is as follows:
"Deep learning is a"-> "subfield"
"Deep learning is a subfield"-> "of"

I would like to know whether there's an issue with the getkacc() function calculation or if my understanding is flawed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant