|
| 1 | +Here are a collection of notes about things we want to accomplish with |
| 2 | +the testing infrastructure. |
| 3 | + |
| 4 | +matplotlib.testing.image_comparison decorator |
| 5 | +============================================= |
| 6 | + |
| 7 | +Collect all the failures in the image_comparison() decorator and raise |
| 8 | +one failure that describes all the failed images. Right now the loop |
| 9 | +that iterates over the images raises an exception on the first |
| 10 | +failure, which clearly breaks out of the loop. |
| 11 | + |
| 12 | +Put image comparison results into more hierarchical view to facilitate |
| 13 | +knowing which test module creates which image. (Also, revive John's |
| 14 | +move_good.py script to deal with new image locations.) |
| 15 | + |
| 16 | +Misc testing stuff |
| 17 | +================== |
| 18 | + |
| 19 | +Recreate John's move_good.py script once the better hierarchy of test |
| 20 | +result images is established. |
| 21 | + |
| 22 | +Buildbot infrastructure |
| 23 | +======================= |
| 24 | + |
| 25 | +Put "actual" images online from buildbots, even without failure. |
| 26 | + |
| 27 | +Get a Windows XP 32-bit build slave. |
| 28 | + |
| 29 | +Build nightly binaries for win32 and Mac OS X. |
| 30 | + |
| 31 | +Build nightly source tarballs. |
| 32 | + |
| 33 | +Build nightly docs and put online somewhere. |
| 34 | + |
| 35 | +A note about the hack that is the failed image gathering mechanism |
| 36 | +================================================================== |
| 37 | + |
| 38 | +The trouble with the current actual/baseline/diff result gathering |
| 39 | +mechanism is that it uses the filesystem as a means for communication |
| 40 | +withing the nose test running process in addition to communication |
| 41 | +with the buildbot process through hard-coded assumptions about paths |
| 42 | +and filenames. If the only concern was within nose, we could |
| 43 | +presumably re-work some the old MplNoseTester plugin to handle the new |
| 44 | +case, but given the buildbot consideration it gets more difficult to |
| 45 | +get these frameworks talking through supported API calls. Thus, |
| 46 | +although the hardcoded path and filename stuff is a hack, it will |
| 47 | +require some serious nose and buildbot learning to figure out how to |
| 48 | +do it the "right" way. So I'm all for sticking with the hack right |
| 49 | +now, and making a bit nicer by doing things like having a better |
| 50 | +directory hierarchy layout for the actual result images. |
0 commit comments