feat: update s3 data source for us-isof partition #311
Conversation
src/fmeval/constants.py
Outdated
| BUILT_IN_DATASET_PREFIX = "s3://fmeval/datasets" | ||
| BUILT_IN_DATASET_DEFAULT_REGION = "us-west-2" | ||
|
|
||
| ISOF_BUILT_IN_DATASET_REGION = "us-isof-south-1" |
There was a problem hiding this comment.
Let's create a mapping from iso region -> iso region that holds the builtin datasets so that we can easily extend the code to support other iso regions in the future.
BUILT_IN_DATASET_ISO_REGIONS = {
"us-isof-south-1": "us-isof-south-1",
"us-isof-east-1": "us-isof-south-1"
}
ISO_REGIONS = ["us-isof-south-1", "us-isof-east-1"] # to be extended with other iso regions later
| region = session.region_name | ||
| if region in ISOF_REGIONS: | ||
| s3_client = ( | ||
| boto3.client("s3", region_name=ISOF_BUILT_IN_DATASET_REGION, verify=False) |
There was a problem hiding this comment.
If we use the mapping described above, this would change to if region in ISO_REGIONS and boto3.client("s3", region_name= BUILT_IN_DATASET_ISO_REGIONS[region], verify=False)
| dataset_uri = DATASET_CONFIGS[TREX].dataset_uri | ||
| s3_client = get_s3_client(dataset_uri) | ||
| mock_client.assert_called_once_with("s3", region_name="us-west-2") | ||
| if dataset_region == "us-isof-south-1": |
There was a problem hiding this comment.
This would get updated to if dataset_region in BUILTIN_DATASET_ISO_REGIONS.values()
| dataset_uri = dataset_uri = S3_PREFIX + DATASET_URI | ||
| s3_client = get_s3_client(dataset_uri) | ||
| mock_client.assert_called_once_with("s3") | ||
| if region.startswith("us-isof"): |
There was a problem hiding this comment.
Similar kind of update here.
| the bucket is not accessible in opt-in regions. | ||
| Util method to return boto3 s3 client. For built-in datasets, the boto3 client region is default to us-west-2 for | ||
| commercial regions as the bucket is not accessible in opt-in regions. | ||
| For us-isof partition, built-in datasets are located in us-isof-south-1 region. |
There was a problem hiding this comment.
Non-blocking nit: update docstring to describe the iso region case in general
| ) | ||
| session = boto3.session.Session() | ||
| region = session.region_name | ||
| if region in BUILT_IN_DATASET_ISO_REGIONS.keys(): |
There was a problem hiding this comment.
Non-blocking nit: no need to use .keys() when checking dict membership
| s3_client = ( | ||
| boto3.client("s3", region_name=BUILT_IN_DATASET_ISO_REGIONS[region], verify=False) | ||
| if uri.startswith(BUILT_IN_DATASET_PREFIX) | ||
| else boto3.client("s3", verify=False) |
There was a problem hiding this comment.
Non-blocking nit: Should we specify region_name when calling boto3.client just to be safe/make this detail more explicit?
Marking this as a nit since based on the docs, whatever configuration files/env vars are used to create session should be used to create the boto3 client as well, so the current code should work.
|
Will address non-blocking comments in a separate PR. |
| ) | ||
| session = boto3.session.Session() | ||
| region = session.region_name | ||
| if region in BUILT_IN_DATASET_ISO_REGIONS.keys(): |
There was a problem hiding this comment.
Question: if later we want to support other ISO regions, are we going to keep updating BUILT_IN_DATASET_ISO_REGIONS?
I'd prefer use prefix check so it will be automatically expand to other regions.
There was a problem hiding this comment.
For new iso regions in other iso partitions we'll need to add the built-in datasets to a region in that partition and update the BUILD_IN_DATASET_ISO_REGIONS map, since iso partitions (eg. iso-f and iso-e) can't access each other.
Issue #, if available:
Description of changes:
get_s3_clientfor us-isof partition. Built-in datasets are located inus-isof-south-1region. (verifyis set to false because there are issues with SSL verification in the partition).By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.