What does this PR do?

• In Use aws-sdk v3 for secrets manager #616, we upgraded aws-sdk to v3 for SecretsManager
• In Lambda, v3 of SecretsManager can cause a SignatureDoesNotMatch error in Lambda: SignatureDoesNotMatch when migrating from AWS SDK v2 to v3 aws/aws-sdk-js-v3#5192 (comment)
• Due to proactive initialization, a busy Lambda might have a container initialize more than 5 minutes before the first invocation
• Originally, we were creating the SecretsManager client during initialization. Since getAPIKey is an async function, it may get paused after importing SecretsManager and may not continue with the getSecretValue call until the first invocation
• If there's a >5 min gap, we get errors
• To resolve this, we can just get the API key during the first invocation

    if (config.apiKeySecretARN !== "") {
      try {
        // Runs on init, which may be a proactive init.
        const { SecretsManager } = await import("@aws-sdk/client-secrets-manager");

        // Runs after the promise resolves, which may be during init or may be later (on the first invocation)
        const secretRegion = config.apiKeySecretARN.split(":")[3];
        const secretsManager = new SecretsManager({
          useFipsEndpoint: FIPS_MODE_ENABLED,
          region: secretRegion,
        });

        // May not run until the first invocation, possibly leading to a >5 min gap
        const secret = await secretsManager.getSecretValue({ SecretId: config.apiKeySecretARN });
        return secret?.SecretString ?? "";
      } catch (error) {
        logError("couldn't get secrets manager api key", error as Error);
      }
    }

Motivation

Multiple ZenDesk tickets opened by customers
https://datadoghq.atlassian.net/browse/SVLS-6813

Testing Guidelines

This is a really rare edge case so I haven't been able to reproduce it, but I did manually verify that none of these changes breaks getting the API key from SecretsManager. Traces and metrics still get sent to Datadog successfully

Additional Notes

Types of Changes

• Bug fix
• New feature
• Breaking change
• Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

• This PR's description is comprehensive
• This PR contains breaking changes that are documented in the description
• This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
• This PR impacts documentation, and it has been updated (or a ticket has been logged)
• This PR's changes are covered by the automated tests
• This PR collects user input/sensitive content into Datadog
• This PR passes the integration tests (ask a Datadog member to run the tests)