Skip to content

Commit

Permalink
drivers: Tune s3 uploader for better performance
Browse files Browse the repository at this point in the history
Increase the concurrency and the part size so the uploads
are more performant in our production environment.

These are also more compatible with Storj's configuration
which saves files in 64MB chunks on the decentralized network,
so that part size also works well for them. But it still improves
upload performance to gcloud in around 10x as well.
  • Loading branch information
victorges committed Mar 29, 2022
1 parent ce9347c commit a33e9b4
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion drivers/s3.go
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,8 @@ func (os *s3Session) saveDataPut(ctx context.Context, name string, data io.Reade
}

uploader := s3manager.NewUploader(os.s3sess, func(u *s3manager.Uploader) {
u.Concurrency = 2
u.Concurrency = 8
u.PartSize = 32 * 1024 * 1024
u.RequestOptions = append(u.RequestOptions, request.WithLogLevel(aws.LogDebug))
})
params := &s3manager.UploadInput{
Expand Down

0 comments on commit a33e9b4

Please sign in to comment.