Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: New Feature: Cancel token generation #7027

Closed
Tyler-R-Kendrick opened this issue Jul 1, 2024 · 1 comment
Closed

.Net: New Feature: Cancel token generation #7027

Tyler-R-Kendrick opened this issue Jul 1, 2024 · 1 comment
Assignees
Labels
filters Anything related to filters

Comments

@Tyler-R-Kendrick
Copy link

Based on the discussion here, I'd like to be able to cancel the llm request from the function filter (ideally using a cancellation token).

I can see two solutions that make sense here.

1.) The cancellation token from the originating HttpRequest is passed through the context as a new property.
2.) The "Result" property cancels the originating request when it is a Task or IAsyncEnumerable that has been cancelled.

@dmytrostruk dmytrostruk self-assigned this Jul 1, 2024
@dmytrostruk dmytrostruk changed the title New Feature: Cancel token generation .Net: New Feature: Cancel token generation Jul 1, 2024
@dmytrostruk dmytrostruk added filters Anything related to filters and removed triage labels Jul 1, 2024
@dmytrostruk
Copy link
Member

Closing this issue as per internal discussion:

  1. It's possible to stop a request to LLM from filter by throwing an exception at specific point.
  2. Cancellation token is still required to monitor for cancellations and pass original token to subsequent async operations inside filters. PR: .Net: Added cancellation token to filter context types #7033

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
filters Anything related to filters
Projects
Status: Sprint: Done
Development

No branches or pull requests

3 participants