S3 upload with boto3
Nov 10, 2020
S3 can handle really big files. You can upload them with Python one by one or piece by piece. The latter one is considered as scalable and safer, because you can apply parallelism and individual piece resumption.
One by one
Each file may be uploaded individually. It took 1m25s.
Piece by piece
Splitting into 32MB pieces and sending them sequentially took 1m21s.
Built-in
The file-like upload can be managed by boto3 and the best method will be chosen including appropriate parallelism. It took 1m12s.
On-the-fly
The multipart upload may give your additional benefit like generating or forwarding the stream without using your local filesystem. In took 1m47s.
