-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Add source.dot.net stage 1 asset publication #42132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add source.dot.net stage 1 asset publication #42132
Conversation
Test instance at https://netsourceindex-validation.azurewebsites.net/ |
- ${{ if eq(variables.enableSourceIndex, 'true') }}: | ||
- template: /eng/common/templates-official/job/source-index-stage1.yml@self | ||
parameters: | ||
sourceIndexBuildCommand: powershell -NoLogo -NoProfile -ExecutionPolicy Bypass -Command "eng/common/build.ps1 -restore -build -ci -nativeToolsOnMachine -configuration $(buildConfiguration)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to specify the binlog parameter here, or does the script always produce it to artifacts/log/$(buildConfiguration)/Build.binlog
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for Eric's comment. Also, I don't like the idea of copying the build process manually into a parameter, that then runs PowerShell, that then runs the build. Instead, it should use our build template, sdk-build.yml, to do a build. That template can do a basic build with no tests (and will produce a binlog) as long as the right parameters are provided to the template.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need a way to specify the binlog path, because there is no consistent buildConfiguration
variable across all repos we can rely on. If we could unify what we call our configuration, then great! But that's a lot of repos.
As-is, the template has a default of artifacts/log/Debug/Build.binlog
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@MiYanni okay, I've done some experimental redesign on the netsourceindex Arcade integration (using SDK as my test repo), and I'd appreciate some design feedback before I start trying to get it into Arcade.
Flow wise, it's not possible for netsourceindex's template to ingest sdk-build.yml, because both define jobs:
(and jobs:
cannot be nested within a job
). So I've moved all the steps:
from netsourceindex into a separate template which the existing job template imports (for backward compatibility) - and the new steps template can be imported by any existing (or new) job defined by the target repo as they see fit.
AS AN EXAMPLE, NOT A RECOMMENDATION I added it here on a not-yet-pushed-to-github internal copy of branch this. A build looks something like https://dev.azure.com/dnceng/internal/_build/results?buildId=2524323&view=logs&j=fa4e7bdb-0412-58a3-8823-7a72e2d8c65d&t=5cdd6b27-41ac-530b-e1af-013abe4bc807
I think adding a new managed-only Windows build leg would be best, but the template design allows that kind of decision to be made by the repo owner to best fit their needs.
If the idiom seems sound to you, then I can look at sending a PR to Arcade with the required prerequisite changes for adoption.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@directhex That looks much better as it doesn't require reconstitution of the build process by the repo. It being a drop-in template like this works much better, IMO. For our repo, if this process only needs to run once on the Windows job, you should do it like this:
- Add a new variable named
uploadSourceIndex
under here and set it totrue
. - Add the same
uploadSourceIndex
variable under here and set it tofalse
. - Since it looks like you want the build outputs included in the source index, the template should be added under here under that "Publish Logs" task.
- ${{ if eq(parameters.uploadSourceIndex, true) }}:
- template: /eng/common/core-templates/steps/source-index-stage1-publish.yml
parameters:
binLogPath: artifacts\log\$(buildConfiguration)\Build.binlog
This way, it only runs once on the official Windows job and no where else.
- ${{ if eq(variables.enableSourceIndex, 'true') }}: | ||
- template: /eng/common/templates-official/job/source-index-stage1.yml@self | ||
parameters: | ||
sourceIndexBuildCommand: powershell -NoLogo -NoProfile -ExecutionPolicy Bypass -Command "eng/common/build.ps1 -restore -build -ci -nativeToolsOnMachine -configuration $(buildConfiguration)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for Eric's comment. Also, I don't like the idea of copying the build process manually into a parameter, that then runs PowerShell, that then runs the build. Instead, it should use our build template, sdk-build.yml, to do a build. That template can do a basic build with no tests (and will produce a binlog) as long as the right parameters are provided to the template.
Due to lack of recent activity, this PR has been labeled as 'Stale'. It will be closed if no further activity occurs within 7 more days. Any new comment will remove the label. |
This is a prerequisite for inclusion on source.dot.net.
It seems a first attempt was made to add sdk.git to source.dot.net in early 2021, but it was added incorrectly and not fixed at any point in the interim.
Generally speaking, "V2" repositories (data sent in by individual repo CI) are more robust for inclusion than the old V1 repos (repo data generated during source-indexer CI runs) - as such we aren't really interested in adding or fixing any V1 repos, and would like to move to V2.
This move also allows the repo owners who know their code best to fine-tune their build flags and environment far more easily than for V1 repos.
Please note that there is an issue under active investigation where V2 repository publishing can occasionally fail due to obscure permissions errors from Azure since we switched uploads from SAS to Managed Identity (currently about 6 failures per day total across all repositories)