Question about s3 code to allow access b2 (backblaze) #1595
Description
Hi,
I fail to use b2 as replication using the simple line "b2": "accountid:applicationid:bucket" as explained https://perkeep.org/doc/server-config. I got the same issue as #1385, since b2 changed their access.
So i tried to use AWS go SDK since it is supported by backblaze (https://help.backblaze.com/hc/en-us/articles/360047629713-Using-the-AWS-Go-SDK-with-B2) without succeed, but I got deeper.
Here is my config for b2:
[...]
"/bs2/": {
"handler": "storage-s3",
"handlerArgs":{
"bucket":"<BUCKET_NAME>",
"aws_region":"us-west-004",
"aws_access_key":"<API_KEY>",
"aws_secret_access_key":"<APPLICATION_KEY>",
"hostname":"s3.us-west-004.backblazeb2.com"
}
}
[...]Using actual S3 code does not work for B2, "because" the following line return an empty region:
perkeep/pkg/blobserver/s3/s3_preflight.go
Line 83 in 508fe2c
For example I got the error when executing perkeepd:
2022/03/04 14:33:02 sync: /bs/ -> /bs2/: error copying sha224-1c4c590fc4cc1a5216047d8906bd378c39b1294e5887c8d07bfb9c4a
: dest write: MissingRegion: could not find region configuration
With the patch below, I can use b2 as a replica:
diff --git a/pkg/blobserver/s3/s3.go b/pkg/blobserver/s3/s3.go
index d63fd66..b0da9c8 100644
--- a/pkg/blobserver/s3/s3.go
+++ b/pkg/blobserver/s3/s3.go
@@ -136,7 +136,7 @@ func newFromConfigWithTransport(_ blobserver.Loader, config jsonconfig.Obj, tran
ctx := context.TODO() // TODO: 5 min timeout or something?
if !skipStartupCheck {
- info, err := normalizeBucketLocation(ctx, awsSession, hostname, bucket)
+ info, err := normalizeBucketLocation(ctx, awsSession, hostname, bucket, region)
if err != nil {
return nil, err
}
diff --git a/pkg/blobserver/s3/s3_preflight.go b/pkg/blobserver/s3/s3_preflight.go
index 512f931..af5a7ca 100644
--- a/pkg/blobserver/s3/s3_preflight.go
+++ b/pkg/blobserver/s3/s3_preflight.go
@@ -60,13 +60,13 @@ type bucketInfo struct {
// "test.s3-us-west-1.amazonaws.com", and it would return
// endpoint=s3.us-west-1.amazonaws.com, isAWS=true, region=us-west-1 (assuming,
// of course, the bucket is in us-west-1).
-func normalizeBucketLocation(ctx context.Context, cfg client.ConfigProvider, endpoint string, bucket string) (bucketInfo, error) {
+func normalizeBucketLocation(ctx context.Context, cfg client.ConfigProvider, endpoint string, bucket string, configRegion string) (bucketInfo, error) {
if strings.HasPrefix(endpoint, "https://") || strings.HasPrefix(endpoint, "http://") {
return bucketInfo{}, fmt.Errorf("invalid s3 endpoint: must not include uri scheme")
}
svc := s3.New(cfg)
- endpoint, region, err := determineEndpoint(ctx, svc, endpoint, bucket, "")
+ endpoint, region, err := determineEndpoint(ctx, svc, endpoint, bucket, configRegion)
if err != nil {
return bucketInfo{}, err
}
@@ -77,6 +77,14 @@ func normalizeBucketLocation(ctx context.Context, cfg client.ConfigProvider, end
if err != nil {
return bucketInfo{}, err
}
+ // if isAWS is false, this is potentially b2 related, region should be provided
+ if !isAWS {
+ return bucketInfo{
+ endpoint: endpoint,
+ isAWS: isAWS,
+ region: region,
+ }, nil
+ }
// the endpoint should be corrected before being used to determine a region
// or else the region request can fail spuriously
svc.Config.WithEndpoint(endpoint)This code could appears pretty naive and maybe it is and i'm sorry about this, but the idea is to saying that interacting with B2 using S3 code with only few changes is possible. And if there is a better way to fix this, please, any advice or remarks are welcome.