You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As we discussed, I trained a EAGLE-2 model with ShareGPT data my fine tuned Llama 70B model.
I got reasonable speed up and accept length, but the numbers were still lower than the baseline (setup 1 mentioned below). Any hint for me to future tune the models to maximize the impact?
The following are the results with MT-bench (I just used the code and data from the repo with 80 questions)
Accept length for setup 1: 2.996 (12234 total decoding steps and 36611 total accept length)
Accept length for setup 2: ~2.34
This is the code I used to calculate average accept length yanjunplay@6f6201e
BTW, my calculation formula of accept length might be different from other benchmarks, so I think I care more about the relative number here. :-D
Thanks again!
Update:
BTW, my numbers are different from SpecBench since SpecBench also count the token generated by base(target) model.
So if if we use the SpecBench method, the accept length should +1 and be 3.996 and 3.34 accordingly.
The text was updated successfully, but these errors were encountered:
I suspect one possible reason is that your fine-tuned LLAMA 70B's distribution differs significantly from ShareGPT, more than the original LLAMA 70B, possibly due to fine-tuning in a specific domain. One potential solution is to generate data using your fine-tuned LLAMA 70B instead of using the fixed text from ShareGPT.
QQ, is running python -m eagle.ge_data.allocation --outdir [path of data] with my fine-tuned LLAMA 70 base model going to do the trick? I actually did that to generate the training data.
Hello, me again. :-D
As we discussed, I trained a EAGLE-2 model with ShareGPT data my fine tuned Llama 70B model.
I got reasonable speed up and accept length, but the numbers were still lower than the baseline (setup 1 mentioned below). Any hint for me to future tune the models to maximize the impact?
Comparison setup:
1 (baseline). Open sourced llama3 70B https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct + https://huggingface.co/yuhuili/EAGLE-LLaMA3-Instruct-70B
2. My fine tuned llama3 70B + my trained EAGLE-2 (with fine tuned llama weights)
The following are the results with MT-bench (I just used the code and data from the repo with 80 questions)
This is the code I used to calculate average accept length yanjunplay@6f6201e
BTW, my calculation formula of accept length might be different from other benchmarks, so I think I care more about the relative number here. :-D
Thanks again!
Update:
BTW, my numbers are different from SpecBench since SpecBench also count the token generated by base(target) model.
So if if we use the SpecBench method, the accept length should +1 and be 3.996 and 3.34 accordingly.
The text was updated successfully, but these errors were encountered: