Scripts to audit skipped tests for Fission.

1. Download the "fission-tests-status" sheet as a csv from here.

2. Run get_skipped_tests.py with that csv as input. The 2nd argument will filter skipped tests based on their build.

   $ ./get_skipped_tests.py fission-tests-status.csv [opt|debug]

   This will print a list of skipped tests to stdout.

3. Use unskip.sh to unskip tests locally.

   $ cat skipped-tests | xargs -n1 -I{} ./unskip.sh {} /path/to/mozilla-central

4. You can use run-tests.sh to run the tests locally to confirm that they're actually passing. This step will take time and is mostly optional. It may avoid running unnecessary tests on try.

   $ ./run-tests.sh skipped-tests /path/to/mozilla-central

5. If you ran the tests locally, you should now have 2 new files: pass and fail. Tests in the fail file are still failing locally, so they shouldn't be unskipped. Use reskip.sh to update their annotations.

   $ cat fail | xargs -n1 -I{} ./reskip.sh {} /path/to/mozilla-central

1. Commit the changes.

   $ cd /path/to/mozilla-central
   $ hg commit -m "Unskip fission tests" -m "$(cat /path/to/skipped-tests)"

2. Push to try. There's many ways to do this. The following runs the unskipped tests 3 times on an artifact build. When the fuzzy chooser shows up, filter on "-fis-" and select all jobs.

   $ cat /path/to/skipped-tests | xargs mach try fuzzy --rebuild 3 --artifact

The try push will take some time. After it's done you can use the scripts in analyze/ to fetch and parse test logs.

$ cd analyze/
$ ./fetch_logs.py 'https://treeherder.mozilla.org/#/jobs?repo=try&revision=$REVISION' logs/
$ ./check_logs.py logs/*.log

This will print tests that failed or timed out. You can manually go through this list and run reskip.sh to re-add the skip annotation for tests that are still failing. You should now have a list of tests that are no longer failing and can be unskipped permanantely. You'll need to manually go through this list and remove the "skip-if = false" annotations. There's generally not that many tests to update at this point, so it shouldn't take long, but scripts to automate this would be nice.