Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: inference fault tolerance #108

Merged
merged 23 commits into from
Nov 2, 2023
Merged

fix: inference fault tolerance #108

merged 23 commits into from
Nov 2, 2023

Conversation

ishaansehgal99
Copy link
Collaborator

@ishaansehgal99 ishaansehgal99 commented Oct 26, 2023 •
edited
Loading

This PR introduces a couple changes:

  1. Minor fixes on the e2e preset build pipeline
  2. Updating image version of pytorch/cuda/nccl from 23.06 to 23.10
  3. Update a docker file path (/home/llama -> /llama and /home/falcon -> /falcon) to prevent conflicting dir paths (host-volume: /home) mounting over one another
  4. Add torch rdzv and headless service for inference fault tolerance
  5. Inference code increased error handling, resiliency, fault tolerance, timeouts

@ishaansehgal99 ishaansehgal99 changed the title fix: worker networking issue inference fix: inference fault tolerance Nov 2, 2023
Fei-Guo
Fei-Guo reviewed Nov 2, 2023
View reviewed changes
Copy link
Collaborator

@Fei-Guo Fei-Guo left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename createHeadlessService to useHeadlessService in all cases.

presets/llama-2-chat/inference-api.py Outdated Show resolved Hide resolved
presets/llama-2-chat/inference-api.py Outdated Show resolved Hide resolved
presets/llama-2/inference-api.py Outdated Show resolved Hide resolved
@Fei-Guo Fei-Guo merged commit d0ba4d5 into main Nov 2, 2023
5 checks passed
@Fei-Guo Fei-Guo deleted the Ishaan/fix-worker-networking-inference branch November 2, 2023 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Fei-Guo Fei-Guo Fei-Guo approved these changes

@helayoty helayoty Awaiting requested review from helayoty helayoty is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ishaansehgal99 @Fei-Guo