[WIP] AI video prototype #2959
Draft
[WIP] AI video prototype #2959
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yondonfu
force-pushed
the
ai-video
branch
5 times, most recently
from
feb3f3e
to
563d199
Compare
|
d32b1b5 temporarily disables Linux arm64 builds because they fail due to an error related to not being able to find zlib during ffmpeg compilation. This error doesn't occur for amd64 builds so I suspect the issue is related to the amd64 -> arm64 cross-compilation process. I noticed that we compile an arm64 specific version of x264 before compiling ffmpeg - perhaps we have to do something similar with zlib? zlib is currently required as a dependency as of 133050d#diff-4ae778054809274731b9da0c6a5a869c0bd214e92f954a5c9c39181748c2f175 which enabled the png decoder and image2 muxer which are used to demux + decode a sequence of PNG files so they can be encoded into an mp4 file. Ideally, we would replace the PNG demux/decode component by passing tensors (that represent frames) outputted by a model directly from GPU memory to NVENC using torchaudio.StreamWriter, but torchaudio.StreamWriter doesn't support RGB -> YUV conversion on the GPU yet - it can still encode a larger, less-streaming friendly (my understanding is yuv420p is preferred for streaming) RGB output, but I didn't jump to implement this yet due to current limitations. Until this replacement happens, zlib would be a required dependency to support the temporary PNG demux + decode component. |
|
@yondonfu Weird - on release go-livepeer right now zlib is dynamically linked. I'll have a look. |
This commit removes the tempoary AI subnet docs now that the final docs have been deployed on https://docs.livepeer.ai/ai/introduction.
This commit prevents the orchestrator selection go routine from staying in a infinite loop when no Orchestrators can be found.
) This commit introduces latency consideration into the orchestrator selection process, addressing two key issues. Firstly, it resolves a minor bug where the algorithm consistently selected known orchestrators due to a condition that never evaluated to true (see [this condition](https://github.com/livepeer/go-livepeer/blob/1239b4e56133003fe6a98a863cce6bdd6b5f2532/server/selection.go#L110)). Secondly, this change ensures that, once all orchestrators have been evaluated, the one with the fastest response time for a specific job is chosen. While the current method for calculating latency is somewhat basic, it sets the foundation for more sophisticated enhancements in the future. Co-authored-by: Brad P <[email protected]>
This commit updates the 'ai-worker' dependency to the latest commit.
This commit adds the `gateway` flag and deprecates the `broadcaster` flag per core team decision (details: https://discord.com/channels/423160867534929930/1051963444598943784/1210356864643109004).
* Remove -pricePerUnit requirement for orchestrator with -AIWorker flag * refactor: add PricePerUnit comment This commit reintroduces the previously omitted comment for the PricePerUnit variable, improving code readability and maintainability. * refactor: simplify PricePerUnit flag check condition This commit simplifies the conditional check used to check if the `PricePerUnit` flag is needed. --------- Co-authored-by: Rick Staa <[email protected]>
This commit updates the https://github.com/livepeer/ai-worker to the latest version so that Orchestrators can enable the [DeepCache](https://github.com/horseee/DeepCache) optimization. This optimization will provide a 50% speedup for multi-step inference requests.
This commit ensures that the global https://pkg.go.dev/github.com/golang/mock/Mockgen package is correctly found when the binary is built using the makescript.
This commit enables the NSFW filter on the AI Subnet that has been implement at the runner side in livepeer/ai-worker#76. BREAKING CHANGE: Depending on how dApps interact with the subnet this could be a breaking change given that we return an extra `nsfw` property.
This commit updates the ai-worker so that the right go bindings are available and no nil errors are thrown.
This commit ensures that the livepeer builder is triggered when AI-version tags are used (e.g., `v0.7.2-ai-video-1`).
This commit ensures that the ai-worker is up to date so that no `nil` pointer runtime error is thrown when the runner container returns a empty response.
* refactor(census): rename Broadcaster metrics to Gateway This commit renames the metrics related to Broadcaster to Gateway, following a team decision. More details can be found in the discussion here: [Team Discussion Link](.com/channels/423160867534929930/1051963444598943784/1210356864643109004). * chore: update pending changelog
) This commit adds the `pricePerGateway` flag and deprecates the `pricePerBroadcaster` flag per core team decision (details: https://discord.com/channels/423160867534929930/1051963444598943784/1210356864643109004).
This commit introduces a safeguard to ensure that the Docker image tagged as 'stable' is only pushed when a new tag is created on the stable branch. This prevents unintended updates to the stable Docker image, ensuring consistency and reliability for users relying on the stable tag.
This commit addresses a syntax error in the Docker image tag creation step.
) * Fix nil baseprice when pricePerUnit is unused in aiWorker * fix: fix priceInfo 'nil' error on discovery This commit ensures that when the `transcodePrice` is not set by the AI orchestrator no `nil` error is thrown when a Gateway requests the orchestrators OrchInfo. * fix(ai): fix incorrect transcodePrice condition This commit fixes the check that is performed to check if transcodePrice is set. --------- Co-authored-by: Rick Staa <[email protected]>
This commit ensures that the livepeer_cli does not throw a `nil` error when it tries to retrieve the orchestrator base price.
This commit allows orchestrators to pin the https://hub.docker.com/r/livepeer/ai-runner image, preventing disruptions from breaking changes in the latest tag.
This commit updates the https://github.com/livepeer/ai-worker to the latest commit.
This commit ensures that the stable tag is created on the master branch.
* add safety check to image-to-video input image * refactor(ai): improve code syntax This commit improves the code syntax by making the output format generation step consistent between pipelines. It also updates the ai-worker to the latest version. --------- Co-authored-by: Brad P <[email protected]>
This commit updates the ai-worker dependency to the latest version (i.e. v0.0.4).
This commit updates the AI worker to v0.0.5 so that people can use the new I2I pix2pix model.
This commit updates the ai-worker to the latest version (i.e. v0.0.6) in order to fix a syntax error that was introduced due to an upstream dependency in v0.0.4 and v0.0.5.
…re calculation (#3074) * fix(ai): Fix accuracy of T2I latency score when num_inference_steps provided * refactor(ai): update numInferenceSteps default This commit ensures that the same numInferenceSteps default value is used as the one set in https://github.com/livepeer/ai-worker/blob/31fe460a45e1d9e908d3a1bdcfdd8822c3889214/runner/app/routes/text_to_image.py#L28. --------- Co-authored-by: Elite Encoder <[email protected]>
This commit ensures that the go-livepeer ai-video branch uses the latest ai-worker dependeny (i.e. v0.0.7).
* add upscale image support using stabilityai/stable-diffusion-x4-upscaler model * fix(ai): fix ai-worker client bindings This commit ensures that the right golang client bindings response and request types are used. It also cleans up the codebase a bit. --------- Co-authored-by: Mike Zupper <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Opening this draft PR to kick off CI build process for the WIP
ai-videobranch. This PR should not be merged as-is and the code on this branch will likely be refactored + cleaned up separately later on.