because of cross-platform issues

Might be part of it. Looks like [skip actions] [skip azp] [skip cirrus] skips the benchmark tests, perhaps we need to move that part of the doctesting?

The circleci testing uses python tools/refguide_check.py -v, the failing tests look to use spin check-docs -v (scipy_doctests, looks like). I suspect the tests use different environments.

I don't have any context for why the doctests are in the benchmark test runner rather than running on circleci. @ev-br do you have context on this? Or why the doc builder is still using refguide-check.py? Maybe it should be migrated to use the new doctest runner? Or did we intentionally leave both running on CI?

My understanding is that the students mostly worked on windows.

Thanks Nathan.

@charris , the students for now are using a linux remote machine on Nebari for anything that involves buliding NumPy and/or running tests. I personally checked each example using the following:

spin build && python -m pip install . && spin docs && python tools/refguide_check.py --doctests && spin lint

I know there has been a change to the doctesting process recently, which could be the reason for the fail.

I know for a fact that zero output from any of our AI examples is hallucinated. The output never comes from AI, as I learned quite quickly that the AI output is completely unreliable. We first strip all generated output, and then run the AI generated (quite reliable) code against the current dev build and insert that output appropriately where needed.

One side effect of the AI gen process I've followed is that occasionally we encountered instances where the output from valid working code does not match what the doctester expects. The issue is reproducible in various environments, and doesn't seem to be dependent on platform. This happened with one of the ma functions, see possee-org/genai-numpy#109 for an example. I plan to submit an official issue to Numpy on this topic next week, if the interns don't do so before their internship ends.

I don't think spin lint ever ran the doctests. The command for that is now spin check-docs, please integrate that into your workflow. I also think maybe the doc builder should run that command too?

Sorry for assuming what I was seeing must have been hallucinated.

Will do. The spin check-docs option became available after the original PRs were submitted. I'm checking the tests locally here right now, and I'll have @otieno-juma check them on his end as well after doing a rebase.

The AI generated "output" is quite hilarious sometimes. It's ability to count to 10 is dismal. But it does a great job of generating working code (just don't trust the output).

@bmwoodruff spin check-docs has some dependencies, looks like

pip install scipy-doctest hypothesis matplotlib scipy pytz pandas

You probably don't need all of those if you make sure to just test the relevant module.

I'm getting the same errors you guys are seeing using spin check-docs. After too much time spent tracking down why this happened, I don't think it's an environment issue, nor an issue with the AI code processing we are doing. I think the error crept in when a rebase on the example was done.

When i run our AI parser code (just did a bit ago), against the current dev build of Numpy, I get the following:

AI Generated code log:

text = """
        Converting a simple dtype:
        
        >>> dt = np.dtype('i4')
        >>> ctype = np.ctypeslib.as_ctypes_type(dt)
        >>> ctype
        <class 'ctypes.c_int32'>

        Converting a structured dtype:
        
        >>> dt = np.dtype([('x', 'i4'), ('y', 'f4')])
        >>> ctype = np.ctypeslib.as_ctypes_type(dt)
        >>> ctype
        <class 'ctypes.Structure'>
"""
print(clean_and_process_text(text))

Here's the cleaned up output that should have been in the PR.

        Converting a simple dtype:

        >>> dt = np.dtype('i4')
        >>> ctype = np.ctypeslib.as_ctypes_type(dt)
        >>> ctype
        <class 'ctypes.c_int'>

        Converting a structured dtype:

        >>> dt = np.dtype([('x', 'i4'), ('y', 'f4')])
        >>> ctype = np.ctypeslib.as_ctypes_type(dt)
        >>> ctype
        <class 'struct'>

The cleaned code output matches the images on the review I did of this PR prior to submission, as well as the initial submission of #26827. However, the actual code in the merged version version of the PR matches the garbage AI output (a force pushed rebase on main hides the history here). @otieno-juma , did something happen when you rebased on main?

My best guess is that when when @otieno-juma did a rebase, he may have reverted to something that contained the original code. I did not confirm this rebase before @charris saw it (not sure how to implement that into our workflow once the code hits numpy/numpy as PR). As the images match the cleaned output, but the merged code matches the AI output, then this seems like human error. I'm sorry that I missed that.

@otieno-juma , when you rebase the other examples, please remember to run all tests (use spin check-docs instead of the refguide_check.py command). Then ping me once you push your changes. I'll review them before @charris looks at them.

Sorry for assuming what I was seeing must have been hallucinated

@ngoldbaum, Sorry that you needed to assume this. You were correct here, as somehow in the final steps of the review process the wrong code got put back into the PR. We tried to prevent hallucinations from hitting the code base....

I don't have any context for why the doctests are in the benchmark test runner rather than running on circleci. @ev-br do you have context on this? Or why the doc builder is still using refguide-check.py? Maybe it should be migrated to use the new doctest runner? Or did we intentionally leave both running on CI?

@ngoldbaum two things:

1. refguide-check.py does not run any doctests anymore; it's literally only checking consistency of the reference guide --- that the routine lists in __init__.py docstrings agree with __all__ lists.
2. spin check-docs could just as well run on CircleCI indeed. The only reason it's somewhere else is to avoid rebuilding numpy: CircleCI does not use spin build.