Description
I am having trouble getting both community and pro localstack s3 working using code on talking to localhost:4566. I just started a trial of the pro version and at this point.
No matter what incantation of my key I instantiate as an environment variable or directly into the docker compose file (below) or on the command line, I cannot get code to put an object into localstack s3.
I have tried running with docker compose with the following docker compose file:
version: '3.7'
services:
localstack:
image: localstack/localstack:latest
container_name: localstack_local
ports:
- '4563-4565:4563-4565'
- '4566:4566'
- '4567-4584:4567-4584'
- '8050:8080'
environment:
- MAIN_CONTAINER_NAME=localstack_local
- AWS_DEFAULT_REGION=ca-central-1
- EDGE_PORT=4566
- SERVICES=s3
- DATA_DIR=/var/lib/localstack/data
- DEBUG=1
- PORT_WEB_UI=8080
- PERSISTENCE=1
- DOCKER_HOST=unix:///var/run/docker.sock
- LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY- }
volumes:
- ./development:/var/lib/localstack
- /var/run/docker.sock:/var/run/docker.sock
The docker version seems to run, but my code cannot seem to upload objects to s3. I have this working on another computer, but I can’t get to that computer and I need to get other developers up and running with their computers.
I have also tried:
docker run
--rm -it
-p 4566:4566
-p 4510-4559:4510-4559
-e LOCALSTACK_API_KEY=xxxxxxxxx
localstack/localstack-pro
Same problem.
OS: mac intel (I have it working on a mac M1).
localstack --version -> 1.3.1
I have done
localstack update all
localstack update docker-images
localstack update localstack-cli
I have also tried both python and brew installs of localstack.
I realize there are multiple problems here. The main thing I need to get working is my code uploading to s3 localstack. The code:
const endpoint = isDevelopment() ? "http://localhost:4566" : undefined;
export const combine = (...args: any[]) => {
return args.reduce((acc, obj) => {
if (!obj) return acc;
const temp:any = Object.keys(obj).reduce((acc: any, key: string) => obj[key] === undefined ? {...acc} : {...acc, [key] : obj[key]} , {});
return { ...acc, ...temp };
}, {});
};
const combineForEnv = (s3InitParams: string, options: any = {}) => {
if (isDevelopment()) {
if (options?.region) {
delete options.region;
}
return combine(s3InitParams, { ...options, endpoint, s3ForcePathStyle: true });
}
return combine(s3InitParams, options);
};
export const s3ShimGetSignedUrlPromise = async (
operation: string,
params: any,
s3InitParams?: any
) => {
console.log("s3ShimGetSignedUrlPromise - ENTRY: ", operation, params, combine(s3InitParams));
try {
console.log("S3 Init Params: ", s3InitParams);
const initParams = combineForEnv(s3InitParams);
console.log("Init Params: ", initParams);
const s3 = new AWS.S3(initParams);
const result = await s3.getSignedUrlPromise(operation, params);
console.group("s3ShimGetSignedUrlPromise - RESULT:", result);
return result;
} catch (e) {
console.log("s3ShimGetSignedUrlPromise error:", e);
throw e;
}
};
export async function generateUploads(fileNames: string[]) {
const uploads = fileNames.map((name) => ({
fileName: name,
type: lookup((mime as any).getType(name))
})),
types = uploads.reduce(
(memo, u) => {
memo[u.type.toLowerCase() + "s"]++;
return memo;
},
{ images: 0, videos: 0 } as { [k: string]: number }
);
// generate counter
const ids = await getSourceIdsByType(types);
// associate and set id
const result = await Promise.all(
uploads.map(async ({ fileName, type }: { fileName: string; type: string }) => {
const sourceId = ids[type.toLowerCase() + "s"].shift();
if (!sourceId) throw new Error("Error generating source id for upload");
console.log("GENERATED IDs: " + JSON.stringify(ids));
console.log("GENERATED SOURCE ID: " + sourceId.toString());
const convertedFileName = sanitizeFileName(fileName);
const path = `${process.env.SERVICE_ENV}_images/${sourceId}/${convertedFileName}`;
const location = await s3ShimGetSignedUrlPromise("putObject",
{
Bucket: `${process.env.SERVICE_ENV}-${process.env.SERVICE_REGION}-media`,
Key: path,
Expires: 8 * 60 * 60 // 8 hours
},
{
region: process.env.AWS_REGION_DEFAULT,
useAccelerateEndpoint: true,
signatureVersion: "v4"
});
console.log("S3: upload: ", fileName, sourceId, type, location);
return { fileName, source: "THING", sourceId, type, location };
})
);
console.log("generateUploads - RESULT: ", result);
return result;
}
export default async function createUpload(
_: undefined,
{ input }: { input: { fileNames: string[] } },
// eslint-disable-next-line
__: Context
): Promise<MutationResponse & { uploads: any[] }> {
const uploads = await generateUploads(input.fileNames);
console.log("createUpload - uploads: ", uploads);
return {
affectedCount: uploads.length,
success: true,
message: "Created Upload Successfully",
uploads
};
}
The code runs without error and prints out:
GENERATED IDs: {"images":[]}
GENERATED SOURCE ID: development_images_28710
s3ShimGetSignedUrlPromise - ENTRY: putObject {
Bucket: 'development-media,
Key: 'development_images/development_images_28710/thing.jpg',
Expires: 28800
} { useAccelerateEndpoint: true, signatureVersion: 'v4' }
S3 Init Params: {
region: undefined,
useAccelerateEndpoint: true,
signatureVersion: 'v4'
}
Init Params: {
useAccelerateEndpoint: true,
signatureVersion: 'v4',
endpoint: 'http://localhost:4566',
s3ForcePathStyle: true
}
s3ShimGetSignedUrlPromise - RESULT: http://localhost:4566/development-media/development_images/development_images_28710/thing.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxxxx&X-Amz-Date=20230114T171804Z&X-Amz-Expires=28800&X-Amz-Signature=xxxx&X-Amz-SignedHeaders=host
S3: upload: thing.jpg development_images_28710 IMAGE http://localhost:4566/development-media/development_images/development_images_28710/thing.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxxx&X-Amz-Date=20230114T171804Z&X-Amz-Expires=28800&X-Amz-Signature=xxxx-Amz-SignedHeaders=host
generateUploads - RESULT: [
{
fileName: 'thing.jpg',
source: 'LOCAL',
sourceId: 'development_images_28710',
type: 'IMAGE',
location: 'http://localhost:4566/development-media/development_images/development_images_28710/thing.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxxx&X-Amz-Date=20230114T171804Z&X-Amz-Expires=28800&X-Amz-Signature=xxxx-Amz-SignedHeaders=host'
}
]
createUpload - uploads: [
{
fileName: 'thing.jpg',
source: 'LOCAL',
sourceId: 'development_images_28710',
type: 'IMAGE',
location: 'http://localhost:4566/development/development_images/development_images_28710/thing.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxxx&X-Amz-Date=20230114T171804Z&X-Amz-Expires=28800&X-Amz-Signature=xxxx&X-Amz-SignedHeaders=host'
}
]
I am able to use the awslocal s3 mb s3://bucketname command with no errors, but awslocal s3 ls gives no result.
The localstack output looks like this:
localstack_local | 2023-01-14T19:18:02.679 INFO --- [functhread54] l.services.motoserver : starting moto server on http://0.0.0.0:54359
localstack_local | 2023-01-14T19:18:02.681 INFO --- [ asgi_gw_0] localstack.services.infra : Starting mock S3 service on http port 4566 ...
localstack_local | 2023-01-14T19:18:02.707 INFO --- [ asgi_gw_0] botocore.credentials : Found credentials in environment variables.
localstack_local | 2023-01-14T19:18:03.182 INFO --- [ asgi_gw_0] localstack.utils.bootstrap : Execution of "require" took 746.16ms
localstack_local | 2023-01-14T19:18:03.186 DEBUG --- [ asgi_gw_0] l.services.s3.s3_utils : Received presign S3 URL: http://localhost:4566/development-media/development_images/development_images_28715/thing.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxxx&X-Amz-Date=20230114T191802Z&X-Amz-Expires=28800&X-Amz-Signature=xxxx&X-Amz-SignedHeaders=host
localstack_local | 2023-01-14T19:18:03.190 WARN --- [ asgi_gw_0] l.services.s3.s3_utils : Signatures do not match, but not raising an error, as S3_SKIP_SIGNATURE_VALIDATION=1
localstack_local | 2023-01-14T19:18:03.193 DEBUG --- [ asgi_gw_0] l.services.s3.s3_utils : Valid presign url.
localstack_local | 2023-01-14T19:18:03.334 INFO --- [ asgi_gw_0] localstack.request.aws : AWS s3.GetObject => 200
There’s a get, but no puts. On the machine that works, there are puts in the output.
The warning Signatures do not match, but not raising an error
occurs on the machine that works, so I don’t think that is an issue.