-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
drivers: Refactor package as a more generic and performant library #2223
Conversation
5da8ff8
to
ad83025
Compare
ad83025
to
32501a0
Compare
a33e9b4
to
9f8c762
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added 2 comments, other than that LGTM 👍
Also remember to:
- Update CHANGELOG_PENDING.md
- Consider squashing some commits before merging
9f8c762
to
8726c58
Compare
8726c58
to
f142c67
Compare
The Filebase S3-compatible API does not return a file size in the response for files greater than 255 bytes. That caused a panic on s3.go:267 when reading recordings. With this change, the problem won't happen anymore cause we don't even try to dereference the ContentLength, but hold it as a nilable pointer that should be checked wherever it is used (right now: nowhere).
We want to use this lib for saving large files to ObjectStore as well for the VOD tasks. Since they can be really large, we don't want to have to buffer the whole files in memory for saving. So changing this API to receive a Reader instead makes more sense.
Avoid the need to buffer the whole input in memory before saving. Will also be more performant for large files.
Increase the concurrency and the part size so the uploads are more performant in our production environment. These are also more compatible with Storj's configuration which saves files in 64MB chunks on the decentralized network, so that part size also works well for them. But it still improves upload performance to gcloud in around 10x as well.
f142c67
to
b76b14f
Compare
Codecov Report
@@ Coverage Diff @@
## master #2223 +/- ##
===================================================
- Coverage 54.93891% 54.87805% -0.06086%
===================================================
Files 93 93
Lines 19316 19352 +36
===================================================
+ Hits 10612 10620 +8
- Misses 8115 8140 +25
- Partials 589 592 +3
Continue to review full report at Codecov.
|
It returns an error so had to change a few signatures.
Also add comments with context about how those numbers came up.
What does this pull request do? Explain your changes. (required)
This is to make some improvements to the
drivers
package so that it can also be usedas a library from other projects, specifically the
task-runner
for VOD tasks. It is currentlyusing the drivers library for a platform-agnostic way of accessing Object Stores for reading
and writing VOD assets.
The greatest advantage of that is that we can just use the same Object Store API from
livepeer-com
to declare where assets should be stored as well as stream recordings,keeping the whole interface leaner.
Specific updates (required)
Size
property optional in ourFile
representation. It is optional in the AWS SDK and is not returned in all implementations of it, like previous versions of Filebase, so it was required at some point (not anymore) but I still think it makes sense to keep it.io.Reader
instead of the whole buffered file. The reader is compatible with clients that have the whole bytes in memory, but requiring all clients to buffer the entire contents in memory is not ideal. This would have made the VOD tasks much more inefficient, needing to read the entire files in memory before sending to the OS.io.ReadSeeker
due to hashing purposes. The uploader is the suggested solution in the open issue in their SDK repo.How did you test each of these updates (required)
go-livepeer
drivers
lib in thetask-runner
for VOD and already deployed it in production, it's working greatDoes this pull request close any open issues?
Part of livepeer/task-runner#14
Checklist:
make
runs successfully./test.sh
pass(does it need a changelog update?)