Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add retry logic and proxy support to the NeMo LLM Service #1544

Merged
merged 8 commits into from
Mar 11, 2024
Merged

Add retry logic and proxy support to the NeMo LLM Service #1544

merged 8 commits into from
Mar 11, 2024

Conversation

mdemoret-nv
Copy link
Contributor

@mdemoret-nv mdemoret-nv commented Mar 4, 2024 •
edited
Loading

Description

  • Adds the ability to retry NeMo failures more than one time
  • Adds an argument to configure the retry count
  • Adds support for proxying requests to the NeMo service using the NGC_API_BASE environment variable
  • Changes the API for the base LLMService to improve the type hints and allow arguments other than strings to be used.

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@mdemoret-nv mdemoret-nv added non-breaking Non-breaking change improvement Improvement to existing functionality labels Mar 4, 2024
@mdemoret-nv mdemoret-nv requested a review from a team as a code owner March 4, 2024 20:17
@dagardner-nv dagardner-nv mentioned this pull request Mar 6, 2024
Draft
@mdemoret-nv mdemoret-nv added breaking Breaking change and removed non-breaking Non-breaking change labels Mar 7, 2024
@mdemoret-nv mdemoret-nv requested a review from a team as a code owner March 8, 2024 16:13
dagardner-nv
dagardner-nv approved these changes Mar 8, 2024
View reviewed changes
Copy link
Contributor

@dagardner-nv dagardner-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few questions

@mdemoret-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 744ba79 into nv-morpheus:branch-24.03 Mar 11, 2024
9 checks passed
@mdemoret-nv mdemoret-nv deleted the mdd_nemo-service-retry branch March 11, 2024 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dagardner-nv dagardner-nv dagardner-nv approved these changes

Assignees
No one assigned
Labels
breaking Breaking change improvement Improvement to existing functionality
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@mdemoret-nv @dagardner-nv