S3 upload with boto3

S3 can handle really big files. You can upload them with Python one by one or piece by piece. The latter one is considered as scalable and safer, because you can apply parallelism and individual piece resumption.

One by one

Each file may be uploaded individually. It took 1m25s.

Piece by piece

Splitting into 32MB pieces and sending them sequentially took 1m21s.

Built-in

The file-like upload can be managed by boto3 and the best method will be chosen including appropriate parallelism. It took 1m12s.

On-the-fly

The multipart upload may give your additional benefit like generating or forwarding the stream without using your local filesystem. In took 1m47s.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store