Nowadays most of the data can be accessed over HTTP. You can browse it and eventually fetch it. In the past HTTP served hypertext transmission and files, archives were mainly accessible using FTP protocol. Python still supports it.
FTP allows you to login, switch current directory and fetch all filenames in the current directory.
Knowing the filename you can just download it to the local filesystem.
When dealing with each chunk is required you can provide the callback.