Request for user feedback: the state of S3 support #7607

There are questions for the S3 component users at the end of this post.

Currently Scrapy has 3 components that talk to S3, and they don't share all of the code and settings. A short comparison:

How do they work:

• S3DownloadHandler takes s3:// URLs, converts them into http://{bucket}.s3.amazonaws.com{path} URLs (simple public ones if the keys are not provided, signed using botocore if they are) and passes them to the HTTP download handler.
• S3FeedStorage uses boto3.session.Session() and upload_fileobj() to upload a file.
• S3FilesStore uses botocore.session.get_session() and put_object() to upload a file.

Supported settings:

• S3DownloadHandler only supports AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN.
• S3FeedStorage additionally supports AWS_ENDPOINT_URL, AWS_REGION_NAME and FEED_STORAGE_S3_ACL.
• S3FilesStore additionally supports AWS_USE_SSL and AWS_VERIFY and instead of FEED_STORAGE_S3_ACL it supports FILES_STORE_S3_ACL

One reason for these inconsistencies is that most of the more recent changes to S3 support were done for specific user requests, and as the code isn't shared, they were done to only one component. botocore vs boto3 is more complicated, it seems that boto3 was considered an extra unneeded dep in some discussions (but was brought in for multipart uploads in S3FeedStorage).

It would be nice to unify all of this, to add modern features (I'm sure there are many tunables that we don't support at all that some users may want, and S3DownloadHandler is especially bad as it doesn't even use https:// and uses some old signature algo), but I want to try collecting user feedback:

1. If you are using S3DownloadHandler, or would want to use it if it gained some additional features/support, what do/would you use it for?
2. Are there any missing features in any of these components (including ones with already open issues) that you need or want?
3. Do you run into any problems with any of those components related to S3 regions, ACLs, endpoints, tokens etc. (including ones with already open issues)?