Run e2e tests on each PR. #584

purple4reina · 2025-04-10T21:14:58Z

What does this PR do?

Run the serverless-e2e-tests on PR, skipping anything that isn't a python test. First publishes the layer build to sandbox account region us-west-2. Then, after gathering the arn for the new layer, pass that to the e2e tests for testing.

Motivation

Better to get alerted right away to potential e2e bugs.

Testing Guidelines

Additional Notes

Because this job is triggering job in a different repo, it is quite tough to access the results of that downstream job. As designed right now, a e2e-status job will be created in addition to the e2e-test job. This status job runs only after the e2e-test job completes. The reason this status job is necessary is so that we can post the results back to github.

The problem with this is the e2e-status job can take hours to fully complete. That's because not only do all the other tests in this repo need to run (integration, unit, layer size, etc), it also needs to run the e2e tests, and it must also wait for the full teardown of aws resources. The latter is what can take so long because these teardown jobs are set to run after a 2 hour delay. So, we technically aren't able to get results posted back to github until all of that completes.

The work around, which was too hard to implement with the time I had, is to use the gitlab api to poll the status of the e2e-test job. This might involve creating a Project Token which is more work that I think it's worth.

An alternative solution is to set the e2e-status test as required by github. There are some major problems with this though. There are times in which we would expect that the e2e tests would fail. For example, if I'm implementing a new feature which will turn an e2e test from not working to working, then the e2e tests themselves will always fail in that case, which is what we want. To get around that, we could remove the xfail strict requirement on the e2e tests downstream. But this would mean we aren't able to keep our feature parity doc in lock step with reality.

At this point, I think the best thing to do is to mark the e2e-status job as required. We are always able to override these failures in the UI, which shouldn't mean we're moving any slower.

Types of Changes

Bug fix
New feature
Breaking change
Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

This PR's description is comprehensive
This PR contains breaking changes that are documented in the description
This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
This PR impacts documentation, and it has been updated (or a ticket has been logged)
This PR's changes are covered by the automated tests
This PR collects user input/sensitive content into Datadog
This PR passes the integration tests (ask a Datadog member to run the tests)

https://datadoghq.atlassian.net/browse/APMSVLS-20

purple4reina · 2025-06-20T20:44:14Z

ci/input_files/build.yaml.tpl

+  image: registry.ddbuild.io/images/mirror/alpine:latest
+  tags: ["arch:amd64"]
+  needs:
+    - e2e-test


This job will run once the e2e-test job completes.

purple4reina · 2025-06-20T20:46:00Z

ci/publish_layers.sh

@@ -199,6 +199,8 @@ fi
 while [ $latest_version -lt $VERSION ]; do
    latest_version=$(publish_layer $REGION $layer $aws_cli_python_version_key $layer_path)
    printf "[$REGION] Published version $latest_version for layer $layer in region $REGION\n"
+    latest_arn=$(aws lambda get-layer-version --layer-name $layer --version-number $latest_version --region $REGION --query 'LayerVersionArn' --output text)
+    printf "[$REGION] Published arn $latest_arn\n"


Be sur to always print the arn to stdout when publishing layers. The publish gitlab jobs will do a regex search on this stdout to determine the arn for a newly published layer.

duncanista · 2025-06-23T14:35:18Z

ci/input_files/build.yaml.tpl

@@ -1,8 +1,11 @@
+{{- $e2e_region := "us-west-2" -}}


Is this a variable declaration? If so, why not use an environment variable?

Yes, it is just a variable declaration. I hadn't thought of just using an environment variable. I was thinking that this way, the value is just built into the final yaml file, instead of being resolved later, at ci execution time. What are your thoughts on that?

Ah, okay, so one of the places this variable is used is in a needs block. According to gitlab docs, variables are not available in needs blocks.

duncanista · 2025-06-23T14:36:53Z

ci/input_files/build.yaml.tpl

+      # Extract the arn from the publish log to be used as envvar in e2e tests
+      layer_arn="$(grep 'Published arn' publish.log | grep -oE 'arn:aws:lambda:.*')"
+      if [ -z "$layer_arn" ]; then
+        echo "Error: Layer ARN not found in publish log"
+        exit 1
+      else
+        echo "Found layer arn, $layer_arn"
+      fi
+      echo "PYTHON_{{ $runtime.name | strings.Trim "python" }}_VERSION=$layer_arn" > {{ $dotenv }}
+      cat {{ $dotenv }}


It would be ideal to not have scripts in the template definition, whereas we just call bash scripts

duncanista · 2025-06-23T14:40:23Z

ci/input_files/build.yaml.tpl

+
+e2e-status:
+  stage: e2e
+  image: registry.ddbuild.io/images/mirror/alpine:latest


why do you need an alpine image here?

duncanista · 2025-06-23T14:45:09Z

ci/input_files/build.yaml.tpl

+    {{- end }}
+    {{- end }}
+    - |
+      if [ "$CI_JOB_STATUS" = "success" ]; then


According to the docs, this is for the current executing job.

I'd suggest looking at how we check another's job status using the GitLab API

https://github.com/DataDog/dd-trace-js/blob/b269a7265f46955ab447a2b5a705cc08c4a959fb/.gitlab/benchmarks.yml#L89-L98

purple4reina · 2025-06-30T19:42:53Z

ci/publish_layers.sh

+if [ -n "$DOTENV" ]; then
+    printf "[$REGION] Exporting layer version to $DOTENV file...\n"
+    echo "PYTHON_${PYTHON_VERSION/./}_VERSION=$latest_arn" >> "$DOTENV"
+    cat "$DOTENV"


Here is where we write the arn to the given dotenv file. This dotenv file will be saved as an artifact on the job.

purple4reina force-pushed the rey.abolofia/run-e2e branch 3 times, most recently from f302bd5 to 6889e0f Compare April 10, 2025 21:54

purple4reina force-pushed the rey.abolofia/run-e2e branch from 6ab7639 to 33568a6 Compare May 1, 2025 15:41

purple4reina added 2 commits June 13, 2025 13:39

Always publish to us-east-1.

f19f26c

WIP

abfb1b6

purple4reina force-pushed the rey.abolofia/run-e2e branch from 33568a6 to abfb1b6 Compare June 13, 2025 20:39

Publishes to us-west-2.

2b283fd

purple4reina force-pushed the rey.abolofia/run-e2e branch 22 times, most recently from 49efd46 to 252675f Compare June 16, 2025 21:22

purple4reina commented Jun 20, 2025

View reviewed changes

purple4reina added 7 commits June 20, 2025 14:18

Have e2e-status job start right away then poll.

d158eb0

Comment out tests for now.

afda470

Use apk.

b2b1ba5

Debugging.

92607f2

Try JOB-TOKEN.

a0c1a76

Print out URL.

6924947

Try bridges api.

dfdc510

purple4reina force-pushed the rey.abolofia/run-e2e branch from 1b774ac to dfdc510 Compare June 20, 2025 22:53

Try GITLAB_API_TOKEN.

20a2b56

purple4reina force-pushed the rey.abolofia/run-e2e branch from 4ce0906 to 20a2b56 Compare June 20, 2025 23:02

Use CI_JOB_STATUS to determine pass/fail.

75357ef

purple4reina force-pushed the rey.abolofia/run-e2e branch from cf6e21c to 75357ef Compare June 21, 2025 00:15

purple4reina marked this pull request as ready for review June 21, 2025 00:22

purple4reina requested review from a team as code owners June 21, 2025 00:22

Remove switch.

57056ba

purple4reina force-pushed the rey.abolofia/run-e2e branch from bfe7f03 to 57056ba Compare June 21, 2025 00:39

duncanista reviewed Jun 23, 2025

View reviewed changes

purple4reina added 4 commits June 30, 2025 11:24

Create dotenv file from publish script.

6b9af41

Use docker docker image.

e78f44e

This needs not needed.

cb0bdee

Push check status to file in e2e repo.

59bee4c

purple4reina commented Jun 30, 2025

View reviewed changes

Start status checking in test stage.

648de9a

purple4reina force-pushed the rey.abolofia/run-e2e branch from 892c106 to 648de9a Compare June 30, 2025 20:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Run e2e tests on each PR. #584

Run e2e tests on each PR. #584

Uh oh!

purple4reina commented Apr 10, 2025 •

edited

Loading

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

purple4reina Jun 20, 2025

Uh oh!

duncanista Jun 23, 2025

Uh oh!

purple4reina Jun 30, 2025

Uh oh!

duncanista Jun 23, 2025

Uh oh!

duncanista Jun 23, 2025

Uh oh!

duncanista Jun 23, 2025

Uh oh!

purple4reina Jun 30, 2025

Uh oh!

Uh oh!

Run e2e tests on each PR. #584

Are you sure you want to change the base?

Run e2e tests on each PR. #584

Uh oh!

Conversation

purple4reina commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Testing Guidelines

Additional Notes

Types of Changes

Check all that apply

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

purple4reina commented Apr 10, 2025 •

edited

Loading