-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Logging I/O] Post inference hooks as background tasks #422
Conversation
model-engine/model_engine_server/inference/forwarding/http_forwarder.py
Outdated
Show resolved
Hide resolved
model-engine/model_engine_server/inference/forwarding/celery_forwarder.py
Show resolved
Hide resolved
This pull request has been linked to Shortcut Story #832920: Post inference hooks as background tasks. |
) | ||
request_params = args[0] | ||
request_params_pydantic = EndpointPredictV1Request.parse_obj(request_params) | ||
forwarder.post_inference_hooks_handler.handle(request_params_pydantic, retval, task_id) # type: ignore |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any way to pass in forwarder to the class rather than using encapsulation? maybe the @app.task
decorator allows custom args to be passed into the base class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah I originally tried that approach but sadly you can't pass in custom args into the @app.task
decorator: https://docs.celeryq.dev/en/stable/userguide/tasks.html#list-of-options
model-engine/model_engine_server/inference/forwarding/http_forwarder.py
Outdated
Show resolved
Hide resolved
model-engine/model_engine_server/inference/forwarding/http_forwarder.py
Outdated
Show resolved
Hide resolved
c477762
to
932bcc9
Compare
Pull Request Summary
Making post inference hooks execute as background tasks in the forwarder (
http_forwarder
for sync,celery_forwarder
for async).Test Plan and Usage Guide
Local testing (unit tests are follow ups):
Sync:
Async:
sc-832920