Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Enable providing endpoint_url through environment variables #704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 23, 2023
Merged

Enable providing endpoint_url through environment variables #704

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
129eea7
Enable providing `endpoint_url` through environment variables
elephantum Feb 19, 2023
653a8a1
Update docstring
elephantum Feb 22, 2023
2eba516
Add `endpoint_url` as a top-level configuration parameter for S3FileS…
elephantum Feb 22, 2023
42d1f6a
Fix black formatting
elephantum Feb 23, 2023
b97b668
fix incorrect drop of `endpoint_url` from client_kwargs
elephantum Feb 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 17 additions & 14 deletions docs/source/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,37 +198,40 @@ For a self-hosted MinIO instance:
# When relying on auto discovery for credentials
>>> s3 = s3fs.S3FileSystem(
anon=False,
client_kwargs={
'endpoint_url': 'https://...'
}
endpoint_url='https://...'
)
# Or passing the credentials directly
>>> s3 = s3fs.S3FileSystem(
key='miniokey...',
secret='asecretkey...',
client_kwargs={
'endpoint_url': 'https://...'
}
endpoint_url='https://...'
)

It is also possible to set credentials through envrironment variables:

.. code-block:: python
# export FSSPEC_S3_ENDPOINT_URL=https://...
# export FSSPEC_S3_KEY='miniokey...'
# export FSSPEC_S3_SECRET='asecretkey...'
>>> s3 = s3fs.S3FileSystem()
# or ...
>>> f = fsspec.open("s3://minio-bucket/...")


For Storj DCS via the `S3-compatible Gateway <https://docs.storj.io/dcs/getting-started/quickstart-aws-sdk-and-hosted-gateway-mt>`_:

.. code-block:: python

# When relying on auto discovery for credentials
>>> s3 = s3fs.S3FileSystem(
anon=False,
client_kwargs={
'endpoint_url': 'https://gateway.storjshare.io'
}
endpoint_url='https://gateway.storjshare.io'
)
# Or passing the credentials directly
>>> s3 = s3fs.S3FileSystem(
key='accesskey...',
secret='asecretkey...',
client_kwargs={
'endpoint_url': 'https://gateway.storjshare.io'
}
endpoint_url='https://gateway.storjshare.io'
)

For a Scaleway s3-compatible storage in the ``fr-par`` zone:
Expand All @@ -238,8 +241,8 @@ For a Scaleway s3-compatible storage in the ``fr-par`` zone:
>>> s3 = s3fs.S3FileSystem(
key='scaleway-api-key...',
secret='scaleway-secretkey...',
endpoint_url='https://s3.fr-par.scw.cloud',
client_kwargs={
'endpoint_url': 'https://s3.fr-par.scw.cloud',
'region_name': 'fr-par'
}
)
Expand All @@ -251,8 +254,8 @@ For an OVH s3-compatible storage in the ``GRA`` zone:
>>> s3 = s3fs.S3FileSystem(
key='ovh-s3-key...',
secret='ovh-s3-secretkey...',
endpoint_url='https://s3.GRA.cloud.ovh.net',
client_kwargs={
'endpoint_url': 'https://s3.GRA.cloud.ovh.net',
'region_name': 'GRA'
},
config_kwargs={
Expand Down
13 changes: 11 additions & 2 deletions s3fs/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,10 +184,15 @@ class S3FileSystem(AsyncFileSystem):
Whether to use anonymous connection (public buckets only). If False,
uses the key/secret given, or boto's credential resolver (client_kwargs,
environment, variables, config files, EC2 IAM server, in that order)
endpoint_url : string (None)
Use this endpoint_url, if specified. Needed for connecting to non-AWS
S3 buckets. Takes precedence over `endpoint_url` in client_kwargs.
key : string (None)
If not anonymous, use this access key ID, if specified
If not anonymous, use this access key ID, if specified. Takes precedence
over `aws_access_key_id` in client_kwargs.
secret : string (None)
If not anonymous, use this secret access key, if specified
If not anonymous, use this secret access key, if specified. Takes
precedence over `aws_secret_access_key` in client_kwargs.
token : string (None)
If not anonymous, use this security token, if specified
use_ssl : bool (True)
Expand Down Expand Up @@ -257,6 +262,7 @@ class S3FileSystem(AsyncFileSystem):
def __init__(
self,
anon=False,
endpoint_url=None,
key=None,
secret=None,
token=None,
Expand Down Expand Up @@ -286,6 +292,8 @@ def __init__(
if password:
secret = password

self.endpoint_url = endpoint_url

self.anon = anon
self.key = key
self.secret = secret
Expand Down Expand Up @@ -460,6 +468,7 @@ async def set_session(self, refresh=False, kwargs={}):
aws_access_key_id=self.key,
aws_secret_access_key=self.secret,
aws_session_token=self.token,
endpoint_url=self.endpoint_url,
)
init_kwargs = {
key: value
Expand Down