TGIS_README.md

Repo organization and building the TGIS-vLLM image

This fork attempts to remain aligned with the vLLM repo as much as possible, while also containing a set of permanent changes to add:

• A TGIS api adapter layer (see TGIS)
• A RedHat UBI-based Docker image delivery

Given the fast pace of vLLM development, we also provide builds that include yet-to-be-merged PRs to vLLM by squash-merging open vLLM PRs onto a release branch on top of main that is continually reconstructed as we make more contributions.

See a sketch of the commit graph

Contributing changes for vLLM

To contribute improvements to vLLM that would have utility for the whole community, don't base them on the main branch in this repo. Either contribute to your own fork of vLLM, or create a branch in this repo at the latest vllm-project/vllm:main commit.

Open all PRs into vLLM

Once you have opened a PR, follow the steps for Reconstructing the release branch

Contributing changes specific to the TGIS adapter or IBM delivery

Contributing changes to the TGIS adapter or IBM delivery process is business as usual, make changes on a branch or fork directly from this repo's main branch and PR it back in.

NB: When there are pending PRs to vLLM that are squashed onto the release branch, you will likely want to also apply your changes on a branch off of the latest release.xxx tag to build and test.

Most of the IBM-specific changes in this repo are located in these files/packages:

• vllm/tgis_utils contains many custom implementations of classes required for feature parity with TGIS
• vllm/entrypoints/openai/api_server.py contains changes to parse TGIS specific command line args
• vllm/entrypoints/openai/cli_args.py contains changes to inject TGIS specific command line args
• vllm/entrypoints/grpc is entirely our own package for implementing a grpc server with the TGIS api
• proto contains the TGIS api defs
• Dockerfile.ubi is the UBI-based dockerfile that we build and ship
• .github/workflows/build.yml contains our workflow for building and pushing the UBI-based image

Main branch rebasing procedure

Rebasing vllm:main onto ibm:main is pretty straightforward. Assuming you have vllm-project/vllm as the upstream remote and ibm/vllm as the origin remote, one way to do this is:

# fetch latest ibm main
git fetch origin main
# fetch latest vllm main
git fetch upstream main

# Check out IBM main and cherry pick all new vllm commits here
# NB: This works because the vllm main uses squash commits for a linear history.
# Rebasing vllm main onto ibm main creates some small issues where manual conflict resolution causes commits to differ, and must be skipped or re-applied with every rebase
git checkout origin/main
git cherry-pick $(cat vllm_main_commit.txt)..upstream/main
# Store the new latest vllm main commit
echo "$(git rev-parse --short upstream/main)" > vllm_main_commit.txt
git add vllm_main_commit.txt
git commit -s -m "Update vLLM to $(git rev-parse --short upstream/main)"
# Push to origin/main
git push origin HEAD:main

Release branch building procedure

To rebuild the release branch, we want to squash each pending vLLM PR into a commit on top of main.

Assuming this repo is the origin remote and vllm-project/vllm is upstream, this looks something like:

# Fetch latest mains
git fetch upstream main
git fetch origin main
git checkout origin/main

# Start a new release branch here at ibm:main
git branch -f release HEAD
git checkout release

# for each ${PR_NUMBER} to squash in:
# We first fetch the PR head from vLLM
git branch -D ${PR_NUMBER}
git fetch upstream pull/${PR_NUMBER}/head:${PR_NUMBER}
# Then we want to squash-merge on top of vLLM:main
git checkout upstream/main
git merge --squash ${PR_NUMBER}
# (Resolve any conflicts here)
git commit -m "Squash ${PR_NUMBER}"
# Then we want to apply that squash commit with only that PR's changes to `release`
export SQUASH_HEAD=$(git rev-parse --short HEAD)
git checkout release
git cherry-pick $SQUASH_HEAD
# Merge conflicts should be minimal if all PRs are in a mergeable state with vllm:main
# But pending PRs may create conflicting changes with each other 

# force-push: we're overwriting the `release` branch
git push -f origin HEAD:release

# Create a tag to save this build
git tag release.$(git rev-parse --short HEAD)
git push origin release.$(git rev-parse --short HEAD)