-
Notifications
You must be signed in to change notification settings - Fork 56
GPU Resource Management on the VDK
GPUs are very expensive to acquire. If you are using Kubeflow to run your ML training you will need to have GPUs in your k8s cluster.
We want to ensure the following:
- Teams are only using their fare share of the available resources 🤝
- Available GPU don't become fragmented across nodes 💻🧩
- Resources should never sit idle if there is actual demand for them 🚫 🏭
There are 4 GPU nodes with 10 cores each on your k8s cluster. The total available GPU cores is 40. There are 2 teams sharing this cluster. you tell them to only use 20 cores each. However team 1 consumes 30 cores leaving team 2 to only be able to access 10 of its allocated 20.
This is really frustrating for Team 2 and there is no automated or obvious recourse they can take to reclaim the 10 cores the are missing
There are 4 GPU nodes with 10 cores each on your k8s cluster. The total available GPU cores is 40. There are 2 teams sharing this cluster. you tell them to only use 20 cores each. Both teams are respectful of this. Team 1 works with smaller models and team 2 works with bigger models. Team 1 deploy 6 training each using 3 cores for a total of 18 consumed cores. Team 2 goes to deploy a model which needs 8 cores. Of course they expect this to be able to run. Officially they have 20 cores free. However this job fails to start. Why does it fail to start?
When we look at the pod layout on the nodes we see
Node name | pod state | resource state | has resources to run team 2s job |
---|---|---|---|
Node 1 | Running 2 Team 1 jobs | (6 cores in use, 4 free) | ❌ |
Node 2 | Running 2 Team 1 jobs | (6 cores in use, 4 free) | ❌ |
Node 3 | Running 1 Team 1 job | (3 cores in use, 7 free) | ❌ |
Node 4 | Running 1 Team 1 job | (3 cores in use, 7 free) | ❌ |
This is really frustrating because we want it to pack jobs onto the same nodes where possible so that there is a single block of 8 GPUs free for Team 2 to consume. The ideal pod layout would look like
Node name | pod state | resource state | has resources to run team 2s job |
---|---|---|---|
Node 1 | Running 3 Team 1 jobs | (9 cores in use, 1 free) | ❌ |
Node 2 | Running 2 Team 1 jobs | (6 cores in use, 1 free) | ❌ |
Node 3 | Empty, this is able to take a job from team 2 | (0 cores in use, 10 free) | ✅ |
Node 4 | Empty, this is able to take a job from team 2 | (0 cores in use, 10 free) | ✅ |
There are 4 GPU nodes with 10 cores each on your k8s cluster. The total available GPU cores is 40. There are 2 teams sharing this cluster. you tell them to only use 20 cores each. Team 1 has used up all of its GPUs on training. Team 2 are focusing on one model at the moment and often aren't using most of their allocated resources. It is senseless to leave the GPU sitting idle. Team 1 should be able to consume the resources that belong to the other team under the knowledge that they could be pushed off them at anytime. As long as they are checkpointing the training often this shouldn't be an issue.
RUN:AI (https://www.run.ai/)
Run AI supports all these use cases. But its not open source. We would be an open source alternative.
Some of this functionality could be build by a k8s admin. For example they could assign namespaces to teams and create a resourceQuote per namespace. This addresses problem 1. But it doesn't address fragmentation issues or allowing over quota.
**This solution will cost your company a lot of money holding onto extra resources because of fragementation and not supporting over quota **
Current VDK control plane impl:
- Currently it is a system that monitors your kubernetes cluster. It listens for jobs created, finished etc..
- When it is deploying a job it dynamically builds the k8s yaml files
These is powerful functionality to build on:
- This allows it to dynamically deal with fragmentation. At deploy time we can look at which nodes have free resources and select the most appropriate node by dynamically adding node affinity to the job right before it is deployed.
- The same is true for over quota. If there are free resources we can let a team run extra work. When a team which has free resources goes to deploy a job and there are no free resources on the cluster we can scan the cluster and kill a job of a team which is over quota.
SDK - Develop Data Jobs
SDK Key Concepts
Control Service - Deploy Data Jobs
Control Service Key Concepts
- Scheduling a Data Job for automatic execution
- Deployment
- Execution
- Production
- Properties and Secrets
Operations UI
Community
Contacts