Codestin Search App

SidharthBansal · 2020-06-29T19:16:57Z

PR Summary

Generated a matplotlib.tests wheel that can be uploaded to PyPI as a separate PyPI package ("distribution", in distutils parlance), to make it possible to install test data from PyPI.
This is useful e.g. for mplcairo, whose test suite relies on matplotlib's one.
Also, useful for Baseline image problem optimisation.
Created the test_data folder needed for additional test images apart from the baseline images
Created the logic for the generation of the baseline images on fresh install of matplotlib
In case, the baseline images exists exist and have changed then the logic for modification of the baseline images is created.
Creation of a plugin to generate images for both matplotlib and mpl_toolkits folders
Added related documentation.

PR Checklist

Has Pytest style unit tests
Code is Flake 8 compliant
New features are documented, with examples if plot related
Documentation is sphinx and numpydoc compliant
Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

tacaswell · 2020-07-01T14:42:29Z

+                        stdout=subprocess.DEVNULL),
+                        universal_newlines=True)
+        # Clone mpl repo to tmpvenv and run pytests from new mpl repo created
+        subprocess.run(["git", "clone", str(mplRepoPath), "tmp/testenv"],


What version of git do we get worktree? I suspect that would be both faster and small footprint.

tacaswell · 2020-07-01T14:45:21Z

+                        stderr=subprocess.STDOUT,
+                        stdout=subprocess.DEVNULL,
+                        universal_newlines=True)
+        subprocess.run(["python", "-mpip", "install", "-e", mplRepoPath],


should this be the new venv python?

tacaswell · 2020-07-01T14:46:40Z

+                        stderr=subprocess.STDOUT,
+                        stdout=subprocess.DEVNULL,
+                        universal_newlines=True)
+        subprocess.run(["python", "-mpytest"],


Should also install pytest and all of the extra dependencies (pandas, ...) to make sure we generate all of the images or should we only match what the user has installed in their development environment?

I think installing all extra dependencies is a better option as testing aims to check integrity amongst the components too, which can't be achieved if only some of the installs are done.
First time all the images will be generated, later on we can move on specific missing installs and modifications of the baseline images.

What do you think @anntzer ?

SidharthBansal · 2020-07-08T15:21:31Z

Series of commands executed in the terminal:

python3 -mpytest lib/matplotlib/tests -m "baseline_image_generation_test"
mkdir lib/matplotlib/tests/baseline_images
cp -r result_images/test_agg lib/matplotlib/tests/baseline_images/
python3 -mpytest lib/matplotlib/tests -m "baseline_image_generation_test"

The first pytest command generates the baseline images with the baseline images not found error. Then, in the second pytest command, tests passes.
Similar approach for mpl_toolkits_tests need to be worked upon.

jklymak · 2020-08-28T16:56:52Z

+-------------------------
+The first time this test is run, there will be no baseline image to compare against, so the test will fail. The baseline
+images would be generated either by installing ``matplotlib-baseline-images`` package or by running the baseline image
+generation command from the commandline. These methods are described below. Later on, developer can run ::


Should we be suggesting folks generate the images themselves? That will introduce all sorts of changes due to bad font installs. I think tests should be done against canonical, even locally.

The point of this work is so that we don't have to compare against the canonical version of the images which eliminates the issues we have with freetype / latex drifting under us. So long as you don't change your local images commit-to-commit they should also not change commit-to-commit on a different machine with a different freetype / latex install.

The plan is to with a release publish a set of "blessed" images with a fixed freetype.

OK, I thought the point was not to have all the images in the repo.

So the idea is that I generate the images off of master, and then make a new branch, make my changes and then run the tests and compare my changes to the ones I made on master.

A possible difficulty here are all the fixed-size tests we have that are not image tests. There are many tests we have that assert the position of an element is at 0.9765, but it may not be if we don't have the same font library.

In either case, the above description is not clear on the process. I'd say something like

In order to run local image tests, the developer should checkout the ``master`` branch and run the tests once to get a set of baseline images that are consistent with the code in ``master``. Then subsequent tests from their branches will test against these baseline images. Alternately, the developer could download the ``matplotlib-baseline-images`` package and compare against these images. However, the developer will have to be certain to have development environment that has the same free type and other dependencies as the testing environment on the the machine that made ``matplotlib-baseline-images``.

The idea is to eventually remove all of the images but we have options about where devs get the baselines from.

jklymak · 2020-08-28T17:00:05Z

+
+Baseline image generation at the fresh install of matplotlib through command line
+----------------------------------------------------------------------------------
+The baseline images can be generated in the ``baseline_images`` directory by using ::


As above - not sure we should encourage this. Its basically a useless test if local images pass but fail when uploaded.

jklymak · 2020-08-28T17:06:00Z

+---------------------------------------------
+Tests generating the baseline images may change over time. So, the baseline images needs to be regenerated by the
+developer after fetching the changes through ``git pull`` or ``git checkout -b featured_branch``. Missing baseline
+images can be generated in the ``baseline_images`` directory with the following command::


So I still don't understand what will happen at this point. I fix a bug, but it changes the image for test_boo.py::test_existing_feature. I decide the change is acceptable, and submit a PR. How does the reviewer know that the image changed? How do they compare the old versus new and assess if the new change is actually acceptable or not? How do the CI tests pass?

Great thoughts. I will write the documentation for those cases. Thanks

jklymak · 2020-08-28T17:11:06Z

+----------------------------------------------------------------------
+In the previous flow the committed baseline images were changed when the tests generating them changed. Now, there is
+no need to change the baseline images as they will be generated by running the tests by other developers. Instead,
+change the name of the baseline images. It can be done by changing the id appended to the name of the test to the next


In the previous flow changed baseline images were explicitly committed to the repo as part of the PR as a change to the file, and you could compare before and after. I'm still not at all clear what the new plan is from these descriptions. This needs a thorough before and after guide.

tacaswell · 2020-09-07T16:09:09Z

+
+Baseline image generation
+-------------------------
+The idea is to eventually remove all of the images but we have options about where devs get the baselines from.


This needs a paragraph explaining the current state of things to the reader. To us (who have been thinking about this all summer), this makes perfect sense, but to a new contributor who does not even know we have baseline images checked into the repo this is going to be consufing.

…ectory and matplotlib directory

jklymak · 2021-05-08T20:45:17Z

@anntzer this seems like a good start, but is there any reason to keep it open?

anntzer · 2021-05-08T21:00:49Z

Let's close this for now; we can always revisit...

anntzer added the status: waiting for other PR label Jun 29, 2020

SidharthBansal force-pushed the removal_of_baseline_images_from_baseline_package branch 2 times, most recently from 25eb8c5 to 678b778 Compare June 30, 2020 15:03

anntzer reviewed Jun 30, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

anntzer reviewed Jun 30, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

anntzer reviewed Jun 30, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

anntzer reviewed Jun 30, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

anntzer reviewed Jun 30, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

anntzer reviewed Jun 30, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

anntzer reviewed Jun 30, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

anntzer reviewed Jun 30, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

SidharthBansal force-pushed the removal_of_baseline_images_from_baseline_package branch 3 times, most recently from c5b1486 to a000051 Compare July 1, 2020 14:26

tacaswell reviewed Jul 1, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

tacaswell reviewed Jul 1, 2020

View reviewed changes

tacaswell mentioned this pull request Jul 1, 2020

Create a wheel/sdist with the baseline images #17557

Closed

6 tasks

SidharthBansal force-pushed the removal_of_baseline_images_from_baseline_package branch 5 times, most recently from 7a6aedd to c885729 Compare July 6, 2020 10:26

SidharthBansal force-pushed the removal_of_baseline_images_from_baseline_package branch 5 times, most recently from 631f056 to cd7dda9 Compare July 15, 2020 03:59

SidharthBansal force-pushed the removal_of_baseline_images_from_baseline_package branch from cdfbf4e to b97b849 Compare August 24, 2020 02:03

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread setupext.py

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread lib/matplotlib/testing/decorators.py Outdated

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread doc/devel/contributing.rst

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread doc/devel/contributing.rst Outdated

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread doc/devel/contributing.rst

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread doc/devel/contributing.rst Outdated

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread doc/devel/contributing.rst Outdated

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread doc/devel/contributing.rst

tacaswell reviewed Aug 28, 2020

View reviewed changes

Comment thread doc/devel/testing.rst Outdated

jklymak reviewed Aug 28, 2020

View reviewed changes

Comment thread doc/devel/contributing.rst Outdated

jklymak reviewed Aug 28, 2020

View reviewed changes

tacaswell reviewed Sep 7, 2020

View reviewed changes

SidharthBansal added 11 commits September 8, 2020 20:45

Moved baseline images to sub-wheels/lib

1559df3

Added tests/test_data folder and related changes

ec5e8cf

Added machinery for the mpl_baseline package

282c0a5

Added logic for creation of baseline images for both mpl_toolkits dir…

ba5baeb

…ectory and matplotlib directory

changes to the test_backend_pgf to work according to the flag

fcdb268

Added documentation

a9d5aed

Shorten the flag name

b32cae7

Making both package and generation flag work simultaneously

5355933

Added Pytest Plugin to merge two image generation commands into one

8243f7d

Added feature to copy to and extract from mpl_cache the baseline images

57bfa89

Feature for baseline image diff added

3ac82c7

Uh oh!

Conversation

SidharthBansal commented Jun 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SidharthBansal Jul 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SidharthBansal commented Jul 8, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jklymak Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jklymak commented May 8, 2021

Uh oh!

anntzer commented May 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SidharthBansal commented Jun 29, 2020 •

edited

Loading

SidharthBansal Jul 2, 2020 •

edited

Loading

jklymak Aug 31, 2020 •

edited

Loading