[ LP comment 1 by: Fernando Perez, on 2010-01-18 01:39:06.370752+00:00 ]

Some notes on the above:

• I talked with William Stein (sage) a few days ago: no major progress from them on using pexpect for windows, instead they are moving away from pexpect as much as possible by building library wrappers for everything they can (like maxima and gap).
• I don't see us abandoning nose anytime soon. Trial has a few critical problems:
• it would make testing depend on twisted for us. Nose is a tiny, self-contained download, but Twisted is a much larger dependency. Too much for something as critical as testing.
• It has no discovery mechanism for doctests.
• My earlier efforts on doing ipython doctests without nose were only partly successful. In the last few weeks that I've been working a lot on this, I realized we really need quite a bit of machinery to be able to validate full ipython sessions, just doing syntax conversion to get a unittest out of a docstring (the part I was able to succeed on) isn't sufficient for the whole test suite.

I don't think it's really possible to expect our test suite to pass without an ipython shell running somewhere. We are more than just plain python, and to test all of that we really need a true ipython somewhere that we can beat on.

Having said all this, I think things now aren't really that bad. Having cleaned up the test machinery a lot recently, it's working quite well, debugging is possible with --pdb or other mechanisms, etc. The only thing that is hard is debugging a big doctest with lots of output, but that's simply how doctests are, not something specific to ipython (and for that reason, a bad idea in general if you want your tests to be debuggable).

I'm all for making our testing machinery as simple/robust as humanly possible, because I fully realize the importance of the test support in the health of a project. But having spent literally upwards of a hundred hours on this problem alone, I'm not sure right now I'm capable of making it much simpler without writing something like nose again, from scratch.

One ray of light may come from python itself: I think for 2.7 they are improving unittest, so perhaps we could find better support there once it's out.

I'm open to revisiting this question when possible, but I'm not terribly optimistic on me making lots of improvements beyond what's right now in my trunk-dev branch (which actually works quite well, it must be said). But I'd love to see someone who comes with a fresh eye on this problem make progress on it!

If we get rid of Twisted using code in 0.11, that would simplify our testing significantly as we would not have to much with iptest.

@ellisonbg, ok if we close this one? I think we've made already a ton of improvements to iptest and it's reasonably usable now. This bug is sort of an open ended 'make testing better', so I think we have done that much. We can obviously do better and always keep improving, but at this point I'd rather have new bugs for specific changes/fixes rather than an open-ended one like this.

@fperez: I think the only thing that we should do before the 0.11
release is to go through and remove old testing code that is not being
used. I agree that things are much better now that twisted/trial are
out of the way. But I still find the testing subpackage to be a bit
of a tangle. But if you don't want to do this for 0.11 I totally
understand as it doesn't affect users.

On Fri, Apr 8, 2011 at 9:38 AM, fperez
[email protected]
wrote:

@ellisonbg, ok if we close this one?  I think we've made already a ton of improvements to iptest and it's reasonably usable now.  This bug is sort of an open ended 'make testing better', so I think we have done that much.  We can obviously do better and always keep improving, but at this point I'd rather have new bugs for specific changes/fixes rather than an open-ended one like this.

Reply to this email directly or view it on GitHub:
#13 (comment)

Brian E. Granger
Cal Poly State University, San Luis Obispo
[email protected] and [email protected]

On Fri, Apr 8, 2011 at 11:26 AM, ellisonbg
[email protected]
wrote:

@fperez: I think the only thing that we should do before the 0.11
release is to go through and remove old testing code that is not being
used.  I agree that things are much better now that twisted/trial are
out of the way.  But I still find the testing subpackage to be a bit
of a tangle.  But if you don't want to do this for 0.11 I totally
understand as it doesn't affect users.

Sounds good, we'll leave this open for one last cleanup pass, and will
then close it.

After that, we can open targeted issues for specific improvements to
the test system as we find problems or have ideas.

But this may not happen for 0.11...

f

I think #324 essentially states the same issue that this is being left open for - do we want to close one or the other?

As described in the comments, we aren't blocking 0.11 on this and #324 represents the only part we might get to in 0.11, so I'm re-designating as 0.12.

I'm inclined to close this, as I think at this point it's basically a vague 'make the test suite better'. I'm not saying the test suite is perfect, but obviously anything can always be improved. But we've cleaned up our test code quite a bit since this was first opened, and at this point I simply don't know what concrete action this issue suggests, so my vote is to simply close it. Over-generic issues without a concrete metric for success are in practice more or less noise in the tracker, as they can never be truly satisfied.

If anyone wants to reopen this, please do so with specific information on what needs to be done to consider it fixed. Because right now, our test suite is fairly robust, it can be easily debugged with the standard nose tools, etc. So any reopen should be really very specific (and perhaps be made as a new issue).