Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Test failure System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.AiaFetchDelayed #72240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
VincentBu opened this issue Jul 15, 2022 · 11 comments
Open

Test failure System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.AiaFetchDelayed #72240

VincentBu opened this issue Jul 15, 2022 · 11 comments
Labels
arch-x64 area-System.Security os-linux Linux OS (any supported distro)
Milestone
Future

Comments

@VincentBu
Copy link
Contributor

VincentBu commented Jul 15, 2022 •
edited by karelz
Loading

Frequency:

Day Run OS
3/27-7/22
7/17 Official run Ubuntu 22.04
7/16 Official run Ubuntu 22.04
7/14 Official run Ubuntu 22.04 
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('', 'AiaFetchDelayed', true, '', true);

Run: runtime-libraries-coreclr outerloop 20220714.2

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.2204.Amd64.Open

- System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.AiaFetchDelayed

Error message:

System.AggregateException : One or more errors occurred. (PartialChain
Expected: True
Actual:   False) (PartialChain
Expected: True
Actual:   False) (PartialChain
Expected: True
Actual:   False) (PartialChain
Expected: True
Actual:   False) (PartialChain
Expected: True
Actual:   False)
---- PartialChain
Expected: True
Actual:   False
---- PartialChain
Expected: True
Actual:   False
---- PartialChain
Expected: True
Actual:   False
---- PartialChain
Expected: True
Actual:   False
---- PartialChain
Expected: True
Actual:   False


Stack trace
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.AiaFetchDelayed() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 268
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs:line 69
----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
----- Inner Stack Trace #2 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
----- Inner Stack Trace #3 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
----- Inner Stack Trace #4 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
----- Inner Stack Trace #5 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
@VincentBu VincentBu added area-System.Security os-linux Linux OS (any supported distro) arch-x64 blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels Jul 15, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jul 15, 2022
@ghost
Copy link

ghost commented Jul 15, 2022

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-libraries-coreclr outerloop 20220714.2

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.2204.Amd64.Open

- System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.AiaFetchDelayed

Error message:

System.AggregateException : One or more errors occurred. (PartialChain
Expected: True
Actual:   False) (PartialChain
Expected: True
Actual:   False) (PartialChain
Expected: True
Actual:   False) (PartialChain
Expected: True
Actual:   False) (PartialChain
Expected: True
Actual:   False)
---- PartialChain
Expected: True
Actual:   False
---- PartialChain
Expected: True
Actual:   False
---- PartialChain
Expected: True
Actual:   False
---- PartialChain
Expected: True
Actual:   False
---- PartialChain
Expected: True
Actual:   False


Stack trace
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.AiaFetchDelayed() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 268
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs:line 69
----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
----- Inner Stack Trace #2 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
----- Inner Stack Trace #3 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
----- Inner Stack Trace #4 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
----- Inner Stack Trace #5 (Xunit.Sdk.TrueException) -----
   at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.TimeoutTests.<>c.<AiaFetchDelayed>b__5_0() in /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/RevocationTests/TimeoutTests.cs:line 297
   at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName) in /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs:line 67
Author: VincentBu
Assignees: -
Labels:

area-System.Security, os-linux, arch-x64, blocking-outerloop
Milestone: -

@runfoapp runfoapp bot mentioned this issue Jul 15, 2022
Closed
@vcsjones
Copy link
Member

vcsjones commented Jul 15, 2022 •
edited
Loading

Hm, I can reproduce it on Ubuntu 22.04 consistently. Doesn't reproduce on 20.04.

@danmoseley
Copy link
Member

Does this likely require a bug fix for .NET 7? It's marked blocking but has no milestone. Not sure how to evaluate significance..

@karelz
Copy link
Member

karelz commented Jul 22, 2022

Looks like recent regression - 3 hits in last few days. It's worth a look. Perhaps something changed recently.

@vcsjones
Copy link
Member

Perhaps something changed recently.

I think we added Ubuntu 22.04 to the CI matrix recently? It fails on 22.04 but not 20.04.

@vcsjones
Copy link
Member

vcsjones commented Jul 22, 2022 •
edited
Loading

Does this likely require a bug fix for .NET 7?

There are other AIA tests passing that lead me to believe AIA is not entirely broken. But I'll dig in a bit to try and understand why this is failing.

Update: this should be fixed for .NET 7.

@vcsjones
Copy link
Member

vcsjones commented Jul 22, 2022 •
edited
Loading

@bartonjs During this test, when we get in to IsCompleteChain, the chain contains the error X509_V_ERR_AKID_SKID_MISMATCH. That results in IsCompleteChain believing that the chain is complete, so no AIA fetch is performed. The "timeout" aspect of this is a red herring. The test fails regardless of a timeout that is present or not.

Adding X509VerifyStatusCodeUniversal.X509_V_ERR_AKID_SKID_MISMATCH to case in IsCompleteChain to indicate an incomplete chain fixes the test, but I am not sure this is the "right" fix. My gut tells me "no". Would appreciate your input here.

This leads me to believe this is a regression for OpenSSL 3 AIA fetching on Linux, so marking for .NET 7.

@vcsjones vcsjones added this to the 7.0.0 milestone Jul 22, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Jul 22, 2022
@bartonjs
Copy link
Member

Weird. The only place in OSSL3 I see returning that error code is

https://github.com/openssl/openssl/blob/98b183d3c65e56b0f21f4e77b2cd3d2aa62374f2/crypto/x509/v3_purp.c#L947-L950

And that would make it seem like the test CA system was generating gibberish AKID values, which would make all of the dynamic chain tests fail.

@vcsjones
Copy link
Member

vcsjones commented Jul 30, 2022 •
edited
Loading

Native stack (line numbers come from openssl/openssl@89cd17a)

frame #0: 0x0000ffbe6ae066d0 libcrypto.so.3`X509_check_akid(issuer=0x0000ffbe5c008600, akid=0x0000ffbe5c0d37a0) at v3_purp.c:955:8
frame #1: 0x0000ffbe6ae06600 libcrypto.so.3`ossl_x509_likely_issued(issuer=0x0000ffbe5c008600, subject=0x0000ffbe5c0039e0) at v3_purp.c:928:11
frame #2: 0x0000ffbe6ae179a8 libcrypto.so.3`check_issued(ctx=0x0000ffbe5c2c7510, x=0x0000ffbe5c0039e0, issuer=0x0000ffbe5c008600) at x509_vfy.c:346:15
frame #3: 0x0000ffbe6ae178a8 libcrypto.so.3`find_issuer(ctx=0x0000ffbe5c2c7510, sk=0x0000ffbe5c2c4ba0, x=0x0000ffbe5c0039e0) at x509_vfy.c:330:13
frame #4: 0x0000ffbe6ae1df88 libcrypto.so.3`build_chain(ctx=0x0000ffbe5c2c7510) at x509_vfy.c:3231:24
frame #5: 0x0000ffbe6ae17340 libcrypto.so.3`verify_chain(ctx=0x0000ffbe5c2c7510) at x509_vfy.c:216:15
frame #6: 0x0000ffbe6ae177a8 libcrypto.so.3`X509_verify_cert(ctx=0x0000ffbe5c2c7510) at x509_vfy.c:295:59
frame #7: 0x0000ffbe843689bc libSystem.Security.Cryptography.Native.OpenSsl.so`CryptoNative_X509VerifyCert(ctx=0x0000ffbe5c2c7510) at pal_x509.c:286:12

@vcsjones
Copy link
Member

vcsjones commented Jul 30, 2022 •
edited
Loading

Something is... really odd or I am tired. Or both.

(lldb) po X509_NAME_oneline(X509_get_issuer_name(subject), 0, 0)
"/O=AiaFetchDelayed/CN=A Revocation Test CA 0"

(lldb) po X509_NAME_oneline(X509_get_subject_name(issuer), 0, 0)
"/O=AiaFetchDelayed/CN=A Revocation Test CA 0"

KJ comment: both have a -> length of 20 but I was too lazy to get lldb to only dump 20 bytes, so only look at the first 20.

(lldb) x subject->akid->keyid->data
0xffbe6c020800: 3d f9 65 9c 84 cd 01 c7 be a9 92 3d 3d e1 46 46  =.e........==.FF
0xffbe6c020810: 32 54 76 72 00 00 00 00 35 00 00 00 00 00 00 00  2Tvr....5.......
(lldb) x issuer->skid->data
0xffbe6c2160d0: 29 85 ec 50 6a a7 e8 54 7f 1a 10 bc 94 c1 8b 27  )..Pj..T.......'
0xffbe6c2160e0: 97 41 2c 0e 00 00 00 00 35 00 00 00 00 00 00 00  .A,.....5.......

The issuer's subject matches the subject's issuer, but the SKID/AKID don't agree. When I actually dump the certs from the test, they do match.

@jeffhandley jeffhandley modified the milestones: 7.0.0, 8.0.0 Aug 12, 2022
@adamsitnik adamsitnik modified the milestones: 8.0.0, 9.0.0 Aug 1, 2023
@jeffhandley jeffhandley modified the milestones: 9.0.0, Future Jul 28, 2024
@BruceForstall BruceForstall removed the blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs label Apr 8, 2025
@BruceForstall
Copy link
Member

If it's marked "Future", it can't be blocking outerloop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 area-System.Security os-linux Linux OS (any supported distro)
Projects
None yet
Milestone
Future
Development

No branches or pull requests
8 participants
@vcsjones @jeffhandley @karelz @adamsitnik @danmoseley @bartonjs @BruceForstall @VincentBu