Ianmacleod/add mistral #307
Conversation
| @@ -117,6 +117,8 @@ | |||
| "falcon-7b-instruct": "tiiuae/falcon-7b-instruct", | |||
| "falcon-40b": "tiiuae/falcon-40b", | |||
| "falcon-40b-instruct": "tiiuae/falcon-40b-instruct", | |||
| "mistral-7b": "mistralai/Mistral-7B-v0.1", | |||
There was a problem hiding this comment.
Do we need to add additional validation/tests? Also, think we need to update the EGP APIs?
Also we may want to update our Model Zoo docs to have some more granularity, to reflect the fact that not all models are supported for all frameworks.
There was a problem hiding this comment.
Oh yes, the EGP APIs will also need to change. And I will add the model-zoo comment and add a supported frameworks column to the docs.
There was a problem hiding this comment.
as far as testing, we can add synthetic monitors for completions for the new models. I will take out a new pr for this.
model-engine/model_engine_server/domain/use_cases/llm_model_endpoint_use_cases.py
Show resolved
Hide resolved
| @@ -117,6 +117,8 @@ | |||
| "falcon-7b-instruct": "tiiuae/falcon-7b-instruct", | |||
| "falcon-40b": "tiiuae/falcon-40b", | |||
| "falcon-40b-instruct": "tiiuae/falcon-40b-instruct", | |||
| "mistral-7b": "mistralai/Mistral-7B-v0.1", | |||
There was a problem hiding this comment.
update L494 about context window?
There was a problem hiding this comment.
added this in, also added some logic to make the output dir "mistral_files" and added the max_model_len param
adding mistral endpoint support for completions