Please use TFS reporter

Does ut_tfs_junit_reporter generate a junit5 based xml format report which can be parsed by Bamboo?

It is a JUnit without nesting.
Please try it for yourself

Hi @EGTBosch
Did you manage to check and verify that the reporter works for you?

We will test the reporter to get the report integrated with Bamboo. I have planned to do this in this week. I will let you know.

It looks like the following XML generated by the TFS Reporter is not accepted by Bamboo, because it complains that it cannot find any test cases in the XML file.

<testsuites> <testsuite tests="14" id="1" package="flow_control.calculate_release_rates.unit_integration_test.uit_adb_fc_crr.ProductFeatures" errors="0" failures="0" name="Product Features" time="10.150383" timestamp="2023-06-15T08:47:33" hostname="VI\SRV03660"> <properties/> <testcase classname="flow_control.calculate_release_rates.unit_integration_test.uit_adb_fc_crr.ProductFeatures" name="ADB_FC_BothTsmZeroCapacity # [00], Both TSM no capacity" time=".194012"/> <testcase classname="flow_control.calculate_release_rates.unit_integration_test.uit_adb_fc_crr.ProductFeatures" name="ADB_FC_OneLoopZeroRatesFit # [01], One loop zero rate, fitting rates" time=".037942"/> <testcase classname="flow_control.calculate_release_rates.unit_integration_test.uit_adb_fc_crr.ProductFeatures" name="ADB_FC_OneLoopZeroTargetRateTooHigh # [02], One loop zero rate, target rate too high" time=".060827"/> <testcase classname="flow_control.calculate_release_rates.unit_integration_test.uit_adb_fc_crr.ProductFeatures" name="ADB_FC_OneLoopZeroRequiredRateTooHigh # [03], One loop zero rate, requested rate too high" time=".041829"/> <system-out/> <system-err/> </testsuite> <testsuite tests="14" id="2" package="flow_control.calculate_release_rates.unit_integration_test.uit_adb_fc_crr" errors="0" failures="0" name="ADB_UIT_FlowControl_AUTO" time="14.132688" timestamp="2023-06-15T08:47:33" hostname="VI\SRV03660"> <properties/> <system-out/> <system-err/> </testsuite> </testsuites>

I am currenly out of country without access to my laptop.
Please provide as much information as possible to help investigate this.
What version of Bamboo are you using?
What is the exact error message you see in Bamboo?
Did you try to reach out to them and ask why this problem exists on their side?

It would be great to have steps to reproduce.

I do see a potential problem in your sample report.

it is the part:

<testsuites>
...
  <testsuite tests="14" id="2" package="flow_control.calculate_release_rates.unit_integration_test.uit_adb_fc_crr" errors="0" failures="0" name="ADB_UIT_FlowControl_AUTO" time="14.132688" timestamp="2023-06-15T08:47:33" hostname="VI\SRV03660">
    <properties/> 
    <system-out/>
    <system-err/>
  </testsuite>
</testsuites>

There is a suite with 14 tests reported in it, but it contains zero test cases.

I am using Bamboo version 8.2.4 build 80210 - 13 Jun 22.

The JUnit XML file is generated during the build by the TFC reporter of utplsql (version 3.1.10).

The utplsql unit test looks like:

`CREATE OR REPLACE PACKAGE uit_adb_fc_crr
IS
-- %suite(ADB_UIT_FlowControl_AUTO)
-- %suitepath(Flow_Control.Calculate_Release_Rates.Unit_Integration_Test)
-- %rollback(manual)

-- %beforeall
PROCEDURE ut_suite_setup;

-- %beforeeach
PROCEDURE ut_setup;

-- %aftereach
PROCEDURE ut_teardown;

-- %afterall
PROCEDURE ut_suite_teardown;

-- %context(Product Features)
-- %name(ProductFeatures)

-- %test(ADB_FC_BothTsmZeroCapacity # [00], Both TSM no capacity )
PROCEDURE uit_flowcontrol_00;
-- %test(ADB_FC_OneLoopZeroRatesFit # [01], One loop zero rate, fitting rates )
PROCEDURE uit_flowcontrol_01;
-- %test(ADB_FC_OneLoopZeroTargetRateTooHigh # [02], One loop zero rate, target rate too high )
PROCEDURE uit_flowcontrol_02;
-- %test(ADB_FC_OneLoopZeroRequiredRateTooHigh # [03], One loop zero rate, requested rate too high )
PROCEDURE uit_flowcontrol_03;
-- %test(ADB_FC_BasicFlow # [04a], Basic flow )
PROCEDURE uit_flowcontrol_04a;
-- %test(ADB_FC_BasicFlowHigherLrtNormal # [04b], Basic flow with higher LRT Normal )
PROCEDURE uit_flowcontrol_04b;
-- %test(ADB_FC_BasicFlowLowerLrtNormal # [04c], Basic flow with lower LRT Normal )
PROCEDURE uit_flowcontrol_04c;
-- %test(ADB_FC_BasicFlowOvershootByMix # [05], Basic with overshoot caused by mix )
PROCEDURE uit_flowcontrol_05;
-- %test(ADB_FC_OvershootBlueTargetRateTooHigh # [06], Overshoot on Blue, target rate too high )
PROCEDURE uit_flowcontrol_06;
-- %test(ADB_FC_OvershootBlueRequiredRateTooHigh # [07], Overshoot on Blue, required rate too high )
PROCEDURE uit_flowcontrol_07;
-- %test(ADB_FC_OvershootBothTsmTargetRateTooHigh # [08], Overshoot on Both TSM, target rate too high )
PROCEDURE uit_flowcontrol_08;
-- %test(ADB_FC_OvershootBothTsmRequiredRateTooHigh # [09], Overshoot on Both TSM, required rate on one loop too high )
PROCEDURE uit_flowcontrol_09;
-- %test(ADB_FC_OvershootBothTsmTargetAndRequiredRateTooHigh # [10], Overshoot on Both TSM, target and required rate too high )
PROCEDURE uit_flowcontrol_10;
-- %test(ADB_FC_CalculateAtMaximumInterval # [01], Calculate Release Rates at Maximum Interval )
PROCEDURE uit_calc_at_max_interval_01;
-- %endcontext

END uit_adb_fc_crr;
`

I will rename the issue to: TFS reporter does not report flat testsuite structure correctly

@EGTBosch
Will you be able to update the utPLSQL version easily?
The fix will be delivered in version 3.1.14 or 3.1.15.

@jgebal, Yes, I can update the version easily. I am curious about the fix.

What is the expectation when the fix will be delivered?

@EGTBosch can you try this branch ? feature/1257_missing_testcases_tfs_reporter
There is still some changes to be done around that code and more testing, but if the issues is indeed a testsuite with no testcases even when tests are showing positive number then should solve issue.

@lwasylow, will try to test it in the upcoming weeks. I will let you know the results.

@lwasylow I have tried the feature/1257_missing_testcases_tfs_reporter and the xml file is indeed generated without the nested testsuite elements which looks ok, but the junit xml file will still not be accepted by the Bamboo JUnit parser task (reason "Failing task since test cases were expected but none were found"). Check the attached file junit_parser_not_successful.txt, but rename into xml file.

A junit xml from another project will be accepted by Bamboo, but looks a bit different (see attached file
junit_parser_successfully.txt, but rename into xml) than the one which is not accepted.

The file generated by the TFS reporter is according to the JUnit xsd, but is not completely valid for Bamboo.

@EGTBosch hi
And what is a format that Bamboo accepts?
I'm not been using it at all so not familiar with it.
Can you supply specification?

@lwasylow please look at https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd for the XML schema

Hi @EGTBosch
Tfs reporter is based on that.
Are you saying that Xml produced by tfs reporter is not valid with xsd schema you linked?
You mentioned earlier it's valid with xsd but not bamboo.
In order for us to help you we need to understand differences.
If you would be so kind to explain what is that Bamboo needs that differ from xsd we could look if there is anything we can do.

Alternative you can write and plug your own reporter, I believe there is a doc section about that.

Cheers Lukasz

@lwasylow I send you earlier 2 xml files which are both valid against the xml schema used by Bamboo. The xml generated by the TFS reporter (junit_parser_not_successful.txt) is not accepted by Bamboo while the other one (junit_parser_successfully.txt) will be accepted. That is quite strange. You can compare them and see the differences.

@EGTBosch
So I have parsed the document that you sent as successful against a xsd windyroad and it's failing with a big list of errors.
So I'm afraid it's not valid junit document.
Whatever bamboo is expecting is not valid.
We cannot make a modification to the current reporter that will break a format.
Perhaps Bamboo is using different format.
Please see errors.

Cvc-complex-type.3.2.2: Attribute 'name' Is Not Allowed To Appear In Element 'testsuites'., Line '1', Column '70'.
Cvc-complex-type.3.2.2: Attribute 'time' Is Not Allowed To Appear In Element 'testsuites'., Line '1', Column '70'.
Cvc-complex-type.3.2.2: Attribute 'tests' Is Not Allowed To Appear In Element 'testsuites'., Line '1', Column '70'.
Cvc-complex-type.3.2.2: Attribute 'failures' Is Not Allowed To Appear In Element 'testsuites'., Line '1', Column '70'.
Cvc-complex-type.3.2.2: Attribute 'file' Is Not Allowed To Appear In Element 'testsuite'., Line '2', Column '130'.
Cvc-complex-type.4: Attribute 'package' Must Appear On Element 'testsuite'., Line '2', Column '130'.
Cvc-complex-type.4: Attribute 'id' Must Appear On Element 'testsuite'., Line '2', Column '130'.
Cvc-complex-type.4: Attribute 'hostname' Must Appear On Element 'testsuite'., Line '2', Column '130'.
Cvc-complex-type.4: Attribute 'errors' Must Appear On Element 'testsuite'., Line '2', Column '130'.
Cvc-complex-type.2.4.b: The Content Of Element 'testsuite' Is Not Complete. One Of '{properties}' Is Expected., Line '3', Column '15'.
Cvc-complex-type.4: Attribute 'package' Must Appear On Element 'testsuite'., Line '4', Column '113'.
Cvc-complex-type.4: Attribute 'id' Must Appear On Element 'testsuite'., Line '4', Column '113'.
Cvc-complex-type.4: Attribute 'hostname' Must Appear On Element 'testsuite'., Line '4', Column '113'.
Cvc-complex-type.4: Attribute 'errors' Must Appear On Element 'testsuite'., Line '4', Column '113'.
Cvc-complex-type.2.4.a: Invalid Content Was Found Starting With Element 'testcase'. One Of '{properties}' Is Expected., Line '5', Column '107'.

@lwasylow hmmm, indeed, you are right. It is for me a bit unclear which JUnit format Bamboo is using. In this article JUnit4 as well as JUnit5 is mentioned. The attached file with successful in the name is indeed invalid against the JUnit5 format, but valid against JUnit4. So I think the TFS reporter should generate a JUnit4 xml format (by using the xsd as mentioned in the article).

Unfortunately we can't modify the tfs reporter as was written with TFS/Azure Devops in mind.
Sounds like you need a specific Bamboo reporter.
However junit reporter is based on junit4 as defined here:https://gist.github.com/kuzuha/232902acab1344d6b578
And one mentioned in article you linked.

Taking your successful xml against it and removing a file attribute from test suite is passing ok validation.

However as you mentioned junit4 allows a nested suites which are not allowed in bamboo so that would means bamboo not using junit4, or some modified version.

Have you parsed successful xml against junit5 xsd to verify its one that needed?

@lwasylow, it still not clear what the conclusion is. It seems that Bamboo JUnit parser is not doing XSD validation. The current output of the TFS reporter does not help us. Not sure if we can fix this issue.

@EGTBosch I'm unfortunately restricted time wise due to work etc.and don't have enough time at the moment to play with Bamboo.
What I could suggest is that you generate generate junit5 xml manually and publish it via Bamboo. In Jenkins I would just put in repo or run bash cat to file.
If junit5 with couple test cases works we can create junit5 reporter.
Otherwise you could wait when got a bit more time to get bamboo up and running myself and try to publish.
I assume there is no official documentation about publish test results in bamboo?

We generated junit5 files already, removed some attributes, added some other attributes, but still the file is not accepted by Bamboo. Indeed, there is not much information available how Bamboo interprets these junit files. I can wait till you have time to play with Bamboo.