Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[DO NOT MERGE] AB Testing POC #13798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft

[DO NOT MERGE] AB Testing POC #13798

wants to merge 15 commits into from

Conversation

cemms1
Copy link
Contributor

@cemms1 cemms1 commented Apr 16, 2025

No description provided.

Copy link

github-actions bot commented Apr 16, 2025

Size Change: 0 B

Total Size: 973 kB

ℹ️ View Unchanged
Filename Size
dotcom-rendering/dist/1008.client.web.********************.js 2.88 kB
dotcom-rendering/dist/1032.client.web.********************.js 3.31 kB
dotcom-rendering/dist/117.client.web.********************.js 2.87 kB
dotcom-rendering/dist/1562.client.web.********************.js 20.3 kB
dotcom-rendering/dist/1656.client.web.********************.js 49.6 kB
dotcom-rendering/dist/1823.client.web.********************.js 3.9 kB
dotcom-rendering/dist/2045.client.web.********************.js 2.63 kB
dotcom-rendering/dist/2161.client.web.********************.js 3.88 kB
dotcom-rendering/dist/223.client.web.********************.js 527 B
dotcom-rendering/dist/2270.client.web.********************.js 3.65 kB
dotcom-rendering/dist/2309.client.web.********************.js 3.27 kB
dotcom-rendering/dist/2345.client.web.********************.js 3.38 kB
dotcom-rendering/dist/2360.client.web.********************.js 20 kB
dotcom-rendering/dist/2502.client.web.********************.js 157 B
dotcom-rendering/dist/302.client.web.********************.js 8.84 kB
dotcom-rendering/dist/3226.client.web.********************.js 2.85 kB
dotcom-rendering/dist/3292.client.web.********************.js 2.11 kB
dotcom-rendering/dist/3308.client.web.********************.js 3.12 kB
dotcom-rendering/dist/3485.client.web.********************.js 4.67 kB
dotcom-rendering/dist/3598.client.web.********************.js 3.65 kB
dotcom-rendering/dist/3605.client.web.********************.js 15.1 kB
dotcom-rendering/dist/3619.client.web.********************.js 4.46 kB
dotcom-rendering/dist/3950.client.web.********************.js 3.36 kB
dotcom-rendering/dist/4133.client.web.********************.js 3.1 kB
dotcom-rendering/dist/4258.client.web.********************.js 4.08 kB
dotcom-rendering/dist/4306.client.web.********************.js 22.7 kB
dotcom-rendering/dist/4308.client.web.********************.js 3.57 kB
dotcom-rendering/dist/4324.client.web.********************.js 619 B
dotcom-rendering/dist/4457.client.web.********************.js 6.26 kB
dotcom-rendering/dist/4501.client.web.********************.js 4.29 kB
dotcom-rendering/dist/476.client.web.********************.js 5.42 kB
dotcom-rendering/dist/5144.client.web.********************.js 2.55 kB
dotcom-rendering/dist/5448.client.web.********************.js 4.37 kB
dotcom-rendering/dist/545.client.web.********************.js 531 B
dotcom-rendering/dist/5455.client.web.********************.js 7.15 kB
dotcom-rendering/dist/562.client.web.********************.js 11.5 kB
dotcom-rendering/dist/6038.client.web.********************.js 2.76 kB
dotcom-rendering/dist/6217.client.web.********************.js 3.75 kB
dotcom-rendering/dist/6480.client.web.********************.js 3.68 kB
dotcom-rendering/dist/6540.client.web.********************.js 2.62 kB
dotcom-rendering/dist/6541.client.web.********************.js 2.91 kB
dotcom-rendering/dist/6562.client.web.********************.js 439 B
dotcom-rendering/dist/6584.client.web.********************.js 2.68 kB
dotcom-rendering/dist/6722.client.web.********************.js 3.03 kB
dotcom-rendering/dist/6802.client.web.********************.js 2.51 kB
dotcom-rendering/dist/6873.client.web.********************.js 4.13 kB
dotcom-rendering/dist/695.client.web.********************.js 4.51 kB
dotcom-rendering/dist/7116.client.web.********************.js 23.2 kB
dotcom-rendering/dist/732.client.web.********************.js 3.07 kB
dotcom-rendering/dist/7398.client.web.********************.js 3.57 kB
dotcom-rendering/dist/7487.client.web.********************.js 4.25 kB
dotcom-rendering/dist/7927.client.web.********************.js 2.76 kB
dotcom-rendering/dist/8151.client.web.********************.js 4.17 kB
dotcom-rendering/dist/8212.client.web.********************.js 5.04 kB
dotcom-rendering/dist/8385.client.web.********************.js 3.88 kB
dotcom-rendering/dist/8387.client.web.********************.js 2.78 kB
dotcom-rendering/dist/8436.client.web.********************.js 3.63 kB
dotcom-rendering/dist/8602.client.web.********************.js 3.39 kB
dotcom-rendering/dist/8619.client.web.********************.js 4.44 kB
dotcom-rendering/dist/864.client.web.********************.js 3.17 kB
dotcom-rendering/dist/8942.client.web.********************.js 4.44 kB
dotcom-rendering/dist/8973.client.web.********************.js 16.5 kB
dotcom-rendering/dist/8985.client.web.********************.js 11.1 kB
dotcom-rendering/dist/9018.client.web.********************.js 3.73 kB
dotcom-rendering/dist/9135.client.web.********************.js 3.54 kB
dotcom-rendering/dist/9139.client.web.********************.js 3.13 kB
dotcom-rendering/dist/9376.client.web.********************.js 2.52 kB
dotcom-rendering/dist/960.client.web.********************.js 7.99 kB
dotcom-rendering/dist/967.client.web.********************.js 2.87 kB
dotcom-rendering/dist/9709.client.web.********************.js 2.73 kB
dotcom-rendering/dist/9822.client.web.********************.js 7.63 kB
dotcom-rendering/dist/9837.client.web.********************.js 3.29 kB
dotcom-rendering/dist/9870.client.web.********************.js 4.77 kB
dotcom-rendering/dist/Accessibility-importable.client.web.********************.js 8.43 kB
dotcom-rendering/dist/AdBlockAsk-importable.client.web.********************.js 2.99 kB
dotcom-rendering/dist/AdPortals-importable.client.web.********************.js 4.76 kB
dotcom-rendering/dist/AlreadyVisited-importable.client.web.********************.js 425 B
dotcom-rendering/dist/AppsEpic-importable.client.web.********************.js 3.61 kB
dotcom-rendering/dist/AppsFooter-importable.client.web.********************.js 2.69 kB
dotcom-rendering/dist/AppsLightboxImage-importable.client.web.********************.js 2.66 kB
dotcom-rendering/dist/AppsLightboxImageStore-importable.client.web.********************.js 2.6 kB
dotcom-rendering/dist/AudioAtomWrapper-importable.client.web.********************.js 2.76 kB
dotcom-rendering/dist/AudioPlayerWrapper-importable.client.web.********************.js 6.61 kB
dotcom-rendering/dist/AustralianTerritorySwitcher-importable.client.web.********************.js 4.61 kB
dotcom-rendering/dist/Branding-importable.client.web.********************.js 2.89 kB
dotcom-rendering/dist/braze-web-sdk-core.client.web.********************.js 67.7 kB
dotcom-rendering/dist/BrazeMessaging-importable.client.web.********************.js 1.68 kB
dotcom-rendering/dist/CalloutBlockComponent-importable.client.web.********************.js 6.73 kB
dotcom-rendering/dist/CalloutEmbedBlockComponent-importable.client.web.********************.js 5.77 kB
dotcom-rendering/dist/CardCommentCount-importable.client.web.********************.js 2.67 kB
dotcom-rendering/dist/Carousel-importable.client.web.********************.js 6.67 kB
dotcom-rendering/dist/CarouselForNewsletters-importable.client.web.********************.js 4.55 kB
dotcom-rendering/dist/ChartAtom-importable.client.web.********************.js 539 B
dotcom-rendering/dist/CommentCount-importable.client.web.********************.js 2.3 kB
dotcom-rendering/dist/CrosswordComponent-importable.client.web.********************.js 2.88 kB
dotcom-rendering/dist/DiscussionApps-importable.client.web.********************.js 1.07 kB
dotcom-rendering/dist/DiscussionMeta-importable.client.web.********************.js 2.41 kB
dotcom-rendering/dist/DiscussionWeb-importable.client.web.********************.js 3.45 kB
dotcom-rendering/dist/DocumentBlockComponent-importable.client.web.********************.js 2.85 kB
dotcom-rendering/dist/Dropdown-importable.client.web.********************.js 1.72 kB
dotcom-rendering/dist/EditionSwitcherBanner-importable.client.web.********************.js 4.43 kB
dotcom-rendering/dist/EmbedBlockComponent-importable.client.web.********************.js 3.98 kB
dotcom-rendering/dist/EnhancePinnedPost-importable.client.web.********************.js 2.02 kB
dotcom-rendering/dist/FetchOnwardsData-importable.client.web.********************.js 1.94 kB
dotcom-rendering/dist/FilterKeyEventsToggle-importable.client.web.********************.js 3.72 kB
dotcom-rendering/dist/FocusStyles-importable.client.web.********************.js 618 B
dotcom-rendering/dist/FollowWrapper-importable.client.web.********************.js 2.53 kB
dotcom-rendering/dist/FootballMatchesPageWrapper-importable.client.web.********************.js 7.2 kB
dotcom-rendering/dist/FootballTablesCompetitionSelect-importable.client.web.********************.js 3.26 kB
dotcom-rendering/dist/FooterLabel-importable.client.web.********************.js 364 B
dotcom-rendering/dist/FooterReaderRevenueLinks-importable.client.web.********************.js 3.43 kB
dotcom-rendering/dist/frameworks.client.web.********************.js 20.9 kB
dotcom-rendering/dist/FrontSubNav-importable.client.web.********************.js 7.5 kB
dotcom-rendering/dist/GetCricketScoreboard-importable.client.web.********************.js 6.29 kB
dotcom-rendering/dist/GetMatchNav-importable.client.web.********************.js 11.5 kB
dotcom-rendering/dist/GetMatchStats-importable.client.web.********************.js 8.1 kB
dotcom-rendering/dist/GetMatchTabs-importable.client.web.********************.js 2.58 kB
dotcom-rendering/dist/guardian-braze-components-banner.client.web.********************.js 16.1 kB
dotcom-rendering/dist/guardian-braze-components-end-of-article.client.web.********************.js 10.2 kB
dotcom-rendering/dist/GuideAtomWrapper-importable.client.web.********************.js 782 B
dotcom-rendering/dist/index.client.web.********************.js 46.4 kB
dotcom-rendering/dist/InstagramBlockComponent-importable.client.web.********************.js 2.9 kB
dotcom-rendering/dist/InteractiveAtomMessenger-importable.client.web.********************.js 854 B
dotcom-rendering/dist/InteractiveBlockComponent-importable.client.web.********************.js 8.83 kB
dotcom-rendering/dist/InteractiveContentsBlockComponent-importable.client.web.********************.js 3.75 kB
dotcom-rendering/dist/KeyEventsCarousel-importable.client.web.********************.js 5.69 kB
dotcom-rendering/dist/KnowledgeQuizAtom-importable.client.web.********************.js 3.19 kB
dotcom-rendering/dist/LatestLinks-importable.client.web.********************.js 7.96 kB
dotcom-rendering/dist/LightboxHash-importable.client.web.********************.js 436 B
dotcom-rendering/dist/LightboxLayout-importable.client.web.********************.js 6.56 kB
dotcom-rendering/dist/LiveBlogEpic-importable.client.web.********************.js 3.56 kB
dotcom-rendering/dist/LiveblogGutterAskWrapper-importable.client.web.********************.js 2.49 kB
dotcom-rendering/dist/LiveblogNotifications-importable.client.web.********************.js 4.84 kB
dotcom-rendering/dist/Liveness-importable.client.web.********************.js 4.71 kB
dotcom-rendering/dist/LoopVideo-importable.client.web.********************.js 3.54 kB
dotcom-rendering/dist/ManyNewsletterSignUp-importable.client.web.********************.js 7.65 kB
dotcom-rendering/dist/MapEmbedBlockComponent-importable.client.web.********************.js 6.04 kB
dotcom-rendering/dist/Metrics-importable.client.web.********************.js 2.7 kB
dotcom-rendering/dist/MostViewedFooter-importable.client.web.********************.js 3.85 kB
dotcom-rendering/dist/MostViewedFooterData-importable.client.web.********************.js 5.97 kB
dotcom-rendering/dist/MostViewedRightWithAd-importable.client.web.********************.js 5.24 kB
dotcom-rendering/dist/OnwardsUpper-importable.client.web.********************.js 5.32 kB
dotcom-rendering/dist/PersonalityQuizAtom-importable.client.web.********************.js 3.35 kB
dotcom-rendering/dist/ProfileAtom-importable.client.web.********************.js 545 B
dotcom-rendering/dist/ProfileAtomWrapper-importable.client.web.********************.js 805 B
dotcom-rendering/dist/PulsingDot-importable.client.web.********************.js 749 B
dotcom-rendering/dist/QandaAtom-importable.client.web.********************.js 542 B
dotcom-rendering/dist/ReaderRevenueDev-importable.client.web.********************.js 470 B
dotcom-rendering/dist/readerRevenueDevUtils.client.web.********************.js 1.7 kB
dotcom-rendering/dist/RelativeTime-importable.client.web.********************.js 2.56 kB
dotcom-rendering/dist/RichLinkComponent-importable.client.web.********************.js 6.14 kB
dotcom-rendering/dist/ScrollableFeature-importable.client.web.********************.js 6.63 kB
dotcom-rendering/dist/ScrollableHighlights-importable.client.web.********************.js 7.26 kB
dotcom-rendering/dist/ScrollableMedium-importable.client.web.********************.js 2.11 kB
dotcom-rendering/dist/ScrollableSmall-importable.client.web.********************.js 2.17 kB
dotcom-rendering/dist/SecureSignup-importable.client.web.********************.js 4.2 kB
dotcom-rendering/dist/SendTargetingParams-importable.client.web.********************.js 2.21 kB
dotcom-rendering/dist/sentry.client.web.********************.js 802 B
dotcom-rendering/dist/SetABTests-importable.client.web.********************.js 3.91 kB
dotcom-rendering/dist/SetAdTargeting-importable.client.web.********************.js 487 B
dotcom-rendering/dist/ShareButton-importable.client.web.********************.js 2.18 kB
dotcom-rendering/dist/shimport.client.web.********************.js 2.8 kB
dotcom-rendering/dist/ShowHideContainers-importable.client.web.********************.js 893 B
dotcom-rendering/dist/ShowMore-importable.client.web.********************.js 919 B
dotcom-rendering/dist/SignInGateMain.client.web.********************.js 4.35 kB
dotcom-rendering/dist/SignInGateMainCheckoutComplete.client.web.********************.js 5.46 kB
dotcom-rendering/dist/SignInGateSelector-importable.client.web.********************.js 6.72 kB
dotcom-rendering/dist/SlideshowCarousel-importable.client.web.********************.js 4.56 kB
dotcom-rendering/dist/SlotBodyEnd-importable.client.web.********************.js 4.91 kB
dotcom-rendering/dist/SpotifyBlockComponent-importable.client.web.********************.js 5.8 kB
dotcom-rendering/dist/StickyBottomBanner-importable.client.web.********************.js 4.04 kB
dotcom-rendering/dist/SubNav-importable.client.web.********************.js 2.44 kB
dotcom-rendering/dist/TableOfContents-importable.client.web.********************.js 3.52 kB
dotcom-rendering/dist/TimelineAtom-importable.client.web.********************.js 1.23 kB
dotcom-rendering/dist/Titlepiece-importable.client.web.********************.js 13.6 kB
dotcom-rendering/dist/TopBar-importable.client.web.********************.js 8.85 kB
dotcom-rendering/dist/TopBarSupport-importable.client.web.********************.js 2.49 kB
dotcom-rendering/dist/TweetBlockComponent-importable.client.web.********************.js 1.13 kB
dotcom-rendering/dist/UnsafeEmbedBlockComponent-importable.client.web.********************.js 2.92 kB
dotcom-rendering/dist/VideoFacebookBlockComponent-importable.client.web.********************.js 6.06 kB
dotcom-rendering/dist/VineBlockComponent-importable.client.web.********************.js 3.32 kB
dotcom-rendering/dist/YoutubeBlockComponent-importable.client.web.********************.js 843 B

compressed-size-action

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/** Description of the AB test */
description: string;
/** Email address of owner(s) of the test */
owners: string[];
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Create owner definition as "team" type complete with name and email. Would like to be able to identify which team was running the AB test in the data lake

@i-hardy i-hardy force-pushed the commercial/ab-testing-poc branch from 67b8c62 to 59ca921 Compare April 16, 2025 14:16
@i-hardy i-hardy force-pushed the commercial/ab-testing-poc branch from 59ca921 to 1674a26 Compare April 16, 2025 14:17
@Jakeii
Copy link
Member

Jakeii commented Apr 17, 2025

Notes from 17/04/2025:

  • Need to outline how test expiration will work
  • use test groups instead of controlGroup/testGroups
  • can we have a way for analysts to see the tests and state, via frontend or github pages
  • 0% tests how will they work?
  • Opting in/out of tests

@Jakeii
Copy link
Member

Jakeii commented Apr 24, 2025

Notes from 24/04/2025:

  • We don't think we want/need explicit variant sizing, would like to size the whole test based on the overall test size.
  • no downside to expanding to 1000 mvtids, and will allow to split for example a 10% test into 3 x 3.3% variants if ever needed
  • have the date be a string, less boilerplate

Comment on lines +9 to +32
name: 🦕 AB testing
runs-on: ubuntu-latest
defaults:
run:
working-directory: ab-testing/scripts
steps:
- uses: actions/checkout@v4

# https://github.com/denoland/setup-deno#latest-stable-for-a-major
- uses: denoland/setup-deno@v1
with:
deno-version: v2.2

- name: Test
run: deno test

- name: Validate
run: deno run validate

# - name: Build
# run: deno run build

# - name: Deploy
# run: deno run deploy

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 5 days ago

To fix the issue, we will add a permissions block at the workflow level (root) to explicitly define the least privileges required. Since the workflow only involves testing and validation, it likely only needs contents: read permissions. This ensures that the workflow adheres to the principle of least privilege and avoids granting unnecessary write permissions.


Suggested changeset 1
.github/workflows/ab-testing.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/ab-testing.yml b/.github/workflows/ab-testing.yml
--- a/.github/workflows/ab-testing.yml
+++ b/.github/workflows/ab-testing.yml
@@ -1,2 +1,4 @@
 name: 🦕 AB testing
+permissions:
+  contents: read
 on:
EOF
@@ -1,2 +1,4 @@
name: 🦕 AB testing
permissions:
contents: read
on:
Copilot is powered by AI and may make mistakes. Always verify output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants