Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Guided decoding #476

Merged
merged 12 commits into from
Mar 21, 2024
Merged

Guided decoding #476

merged 12 commits into from
Mar 21, 2024

Conversation

yunfeng-scale
Copy link
Collaborator

@yunfeng-scale yunfeng-scale commented Mar 20, 2024

Pull Request Summary

Support guided_json, guided_regex, guided_choice from vLLM

Test Plan and Usage Guide

tested within vllm entrypoint and from model engine

curl localhost:5005/predict -d '{"prompt":"hello", "guided_regex":"a beautiful world .*"}'
{"text": "a beautiful world is fuxix", "count_prompt_tokens": 2, "count_output_tokens": 16, "log_probs": null, "tokens": ["a", " ", "b", "ea", "u", "t", "iful", " ", "wo", "r", "l", "d", " is", " f", "ux", "ix"]}

@yunfeng-scale yunfeng-scale requested a review from a team March 20, 2024 19:02
@yunfeng-scale yunfeng-scale merged commit 5f6cd32 into main Mar 21, 2024
5 checks passed
@yunfeng-scale yunfeng-scale deleted the yunfeng-guided-decoding branch March 21, 2024 04:47
This was referenced Mar 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants