Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 1f69a60

Browse files
kushalkolarkushalkolar
authored
notebook screenshot tests (#277)
* started nb screenshot tests, add reset_vmin_vmax() on ImageCmapFeature * nb test screenshots to CI * add simple.ipynb test screenshots * imageio note in simple.ipynb * try to make failure detection work * faster scatter nb, less points * lines_cmap nb screenshot tests * simplify notebook_finished() * use real image for lines underlay * fix arg * fix test * update nb screenshots * type annotation, small change to try and get git lfs to pull in PR action
1 parent c7265d3 commit 1f69a60

22 files changed

+714
-251
lines changed

‎.github/workflows/ci.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,4 +97,6 @@ jobs:
9797
if: ${{ failure() }}
9898
with:
9999
name: screenshot-diffs
100-
path: examples/desktop/diffs
100+
path: |
101+
examples/desktop/diffs
102+
examples/notebooks/diffs

‎.github/workflows/screenshots.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,11 @@ jobs:
4444
run: |
4545
# regenerate screenshots
4646
REGENERATE_SCREENSHOTS=1 pytest -v examples
47+
REGENERATE_SCREENSHOTS=1 pytest --nbmake examples/notebooks/
4748
- uses: actions/upload-artifact@v3
4849
if: always()
4950
with:
5051
name: screenshots
51-
path: examples/desktop/screenshots/
52+
path: |
53+
examples/desktop/screenshots/
54+
examples/notebooks/screenshots/

‎examples/notebooks/lines_cmap.ipynb

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,19 @@
1313
"import fastplotlib as fpl"
1414
]
1515
},
16+
{
17+
"cell_type": "code",
18+
"execution_count": null,
19+
"id": "5d2ef4aa-0e4c-4694-ae2e-05da1153a413",
20+
"metadata": {
21+
"tags": []
22+
},
23+
"outputs": [],
24+
"source": [
25+
"# this is only for testing, you do not need this to use fastplotlib\n",
26+
"from nb_test_utils import plot_test, notebook_finished"
27+
]
28+
},
1629
{
1730
"cell_type": "code",
1831
"execution_count": null,
@@ -49,6 +62,18 @@
4962
"plot.show()"
5063
]
5164
},
65+
{
66+
"cell_type": "code",
67+
"execution_count": null,
68+
"id": "727282c3-aadf-420f-a88e-9dd4d4e91263",
69+
"metadata": {
70+
"tags": []
71+
},
72+
"outputs": [],
73+
"source": [
74+
"plot_test(\"lines-cmap-white\", plot)"
75+
]
76+
},
5277
{
5378
"cell_type": "markdown",
5479
"id": "889b1858-ed64-4d6b-96ad-3883fbe4d38e",
@@ -69,6 +94,19 @@
6994
"plot.graphics[0].cmap = \"jet\""
7095
]
7196
},
97+
{
98+
"cell_type": "code",
99+
"execution_count": null,
100+
"id": "3c9b0bc8-b176-425c-8036-63dc55ab7466",
101+
"metadata": {
102+
"tags": []
103+
},
104+
"outputs": [],
105+
"source": [
106+
"# for testing, ignore\n",
107+
"plot_test(\"lines-cmap-jet\", plot)"
108+
]
109+
},
72110
{
73111
"cell_type": "code",
74112
"execution_count": null,
@@ -81,6 +119,19 @@
81119
"plot.graphics[0].cmap.values = sine[:, 1]"
82120
]
83121
},
122+
{
123+
"cell_type": "code",
124+
"execution_count": null,
125+
"id": "6b19d2d4-90e7-40ed-afb9-13abe5474ace",
126+
"metadata": {
127+
"tags": []
128+
},
129+
"outputs": [],
130+
"source": [
131+
"# for testing, ignore\n",
132+
"plot_test(\"lines-cmap-jet-values\", plot)"
133+
]
134+
},
84135
{
85136
"cell_type": "code",
86137
"execution_count": null,
@@ -93,6 +144,19 @@
93144
"plot.graphics[0].cmap.values = cosine[:, 1]"
94145
]
95146
},
147+
{
148+
"cell_type": "code",
149+
"execution_count": null,
150+
"id": "0a6c4739-fa61-4532-865e-21107eab76f9",
151+
"metadata": {
152+
"tags": []
153+
},
154+
"outputs": [],
155+
"source": [
156+
"# for testing, ignore\n",
157+
"plot_test(\"lines-cmap-jet-values-cosine\", plot)"
158+
]
159+
},
96160
{
97161
"cell_type": "code",
98162
"execution_count": null,
@@ -105,6 +169,19 @@
105169
"plot.graphics[0].cmap = \"viridis\""
106170
]
107171
},
172+
{
173+
"cell_type": "code",
174+
"execution_count": null,
175+
"id": "45acfd2f-09f5-418c-bca5-3e574348b7d5",
176+
"metadata": {
177+
"tags": []
178+
},
179+
"outputs": [],
180+
"source": [
181+
"# for testing, ignore\n",
182+
"plot_test(\"lines-cmap-viridis\", plot)"
183+
]
184+
},
108185
{
109186
"cell_type": "code",
110187
"execution_count": null,
@@ -129,6 +206,19 @@
129206
"plot.graphics[0].cmap.values = cmap_values"
130207
]
131208
},
209+
{
210+
"cell_type": "code",
211+
"execution_count": null,
212+
"id": "7548407f-05ed-4c47-93cc-131c61f8e242",
213+
"metadata": {
214+
"tags": []
215+
},
216+
"outputs": [],
217+
"source": [
218+
"# for testing, ignore\n",
219+
"plot_test(\"lines-cmap-viridis-values\", plot)"
220+
]
221+
},
132222
{
133223
"cell_type": "code",
134224
"execution_count": null,
@@ -147,6 +237,30 @@
147237
"id": "c290c642-ba5f-4a46-9a17-c434cb39de26",
148238
"metadata": {},
149239
"outputs": [],
240+
"source": [
241+
"# for testing, ignore\n",
242+
"plot_test(\"lines-cmap-tab-10\", plot)"
243+
]
244+
},
245+
{
246+
"cell_type": "code",
247+
"execution_count": null,
248+
"id": "c4b9e735-72e9-4f0e-aa3e-43db57e65c99",
249+
"metadata": {
250+
"tags": []
251+
},
252+
"outputs": [],
253+
"source": [
254+
"# for testing, ignore\n",
255+
"notebook_finished()"
256+
]
257+
},
258+
{
259+
"cell_type": "code",
260+
"execution_count": null,
261+
"id": "f6735cc0-910c-4854-ac50-8ee553a6475e",
262+
"metadata": {},
263+
"outputs": [],
150264
"source": []
151265
}
152266
],

‎examples/notebooks/nb_test_utils.py

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
from typing import *
2+
import os
3+
from pathlib import Path
4+
5+
import imageio.v3 as iio
6+
import numpy as np
7+
8+
from fastplotlib import Plot, GridPlot
9+
10+
# make dirs for screenshots and diffs
11+
current_dir = Path(__file__).parent
12+
13+
SCREENSHOTS_DIR = current_dir.joinpath("screenshots")
14+
DIFFS_DIR = current_dir.joinpath("diffs")
15+
16+
os.makedirs(SCREENSHOTS_DIR, exist_ok=True)
17+
os.makedirs(DIFFS_DIR, exist_ok=True)
18+
19+
20+
# store all the failures to allow the nb to proceed to test other examples
21+
FAILURES = list()
22+
23+
24+
def plot_test(name, plot: Union[Plot, GridPlot]):
25+
snapshot = plot.canvas.snapshot()
26+
27+
if "REGENERATE_SCREENSHOTS" in os.environ.keys():
28+
if os.environ["REGENERATE_SCREENSHOTS"] == "1":
29+
regenerate_screenshot(name, snapshot.data)
30+
31+
try:
32+
assert_screenshot_equal(name, snapshot.data)
33+
except AssertionError:
34+
FAILURES.append(name)
35+
36+
37+
def regenerate_screenshot(name, data):
38+
iio.imwrite(SCREENSHOTS_DIR.joinpath(f"nb-{name}.png"), data)
39+
40+
41+
def assert_screenshot_equal(name, data):
42+
ground_truth = iio.imread(SCREENSHOTS_DIR.joinpath(f"nb-{name}.png"))
43+
44+
is_similar = np.allclose(data, ground_truth)
45+
46+
update_diffs(name, is_similar, data, ground_truth)
47+
48+
assert is_similar, (
49+
f"notebook snapshot for {name} has changed"
50+
)
51+
52+
53+
def update_diffs(name, is_similar, img, ground_truth):
54+
diffs_rgba = None
55+
56+
def get_diffs_rgba(slicer):
57+
# lazily get and cache the diff computation
58+
nonlocal diffs_rgba
59+
if diffs_rgba is None:
60+
# cast to float32 to avoid overflow
61+
# compute absolute per-pixel difference
62+
diffs_rgba = np.abs(ground_truth.astype("f4") - img)
63+
# magnify small values, making it easier to spot small errors
64+
diffs_rgba = ((diffs_rgba / 255) ** 0.25) * 255
65+
# cast back to uint8
66+
diffs_rgba = diffs_rgba.astype("u1")
67+
return diffs_rgba[..., slicer]
68+
69+
# split into an rgb and an alpha diff
70+
diffs = {
71+
DIFFS_DIR.joinpath(f"nb-diff-{name}-rgb.png"): slice(0, 3),
72+
DIFFS_DIR.joinpath(f"nb-diff-{name}-alpha.png"): 3,
73+
}
74+
75+
for path, slicer in diffs.items():
76+
if not is_similar:
77+
diff = get_diffs_rgba(slicer)
78+
iio.imwrite(path, diff)
79+
elif path.exists():
80+
path.unlink()
81+
82+
83+
def notebook_finished():
84+
if len(FAILURES) > 0:
85+
raise AssertionError(
86+
f"Failures for plots:\n{FAILURES}"
87+
)

‎examples/notebooks/scatter.ipynb

Lines changed: 44 additions & 72 deletions
Large diffs are not rendered by default.

‎examples/notebooks/screenshots/nb-astronaut.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-astronaut_RGB.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-camera.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-3d.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-cmap-jet-values-cosine.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-cmap-jet-values.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-cmap-jet.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-cmap-tab-10.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-cmap-viridis-values.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-cmap-viridis.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-cmap-white.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-colors.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-data.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines-underlay.png

Lines changed: 3 additions & 0 deletions
Loading

‎examples/notebooks/screenshots/nb-lines.png

Lines changed: 3 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)