numpy 1.25.0 fails with illegal instruction in aws batch #24028

emilyselwood · 2023-06-23T13:04:21Z

Steps to reproduce:

import numpy 1.25.0 in a task running in aws-batch fails with illegal instruction but runs locally.

python version: 3.10.11
working numpy version: 1.24.3
not working numpy version: 1.25.0

AWS batch is configured to use the following instance types:

m6g
r6g
c7g

Error message:

bash: line 1: 47 Illegal instruction (core dumped) ( python -u src-metrics-flows-test.py --quiet --metadata se

Unfortunately due to running in AWS batch we are not able to get the core dump currently.

Additional information:

Roll back to 1.24.3 and it works again without us changing any code. I would hazard an un-scientific guess that this could be related to the build work thats been going on.

seberg · 2023-06-23T13:56:57Z

Can you help us by saying what type of machines these are? I guess graviton? Also, getting the output of np.show_runtime() is likely necessary to narrow this down.

How exactly did you install/build numpy?

this could be related to the build work thats been going on.

The build work shouldn't affect the released versions of NumPy, although it could be a build issue. Another possibility is incorrect runtime CPU detection, that sometimes happens in virtual machines for example.

emilyselwood · 2023-06-27T12:45:43Z

A quick update, it seems to be the 7g series that cause the problem, but due to having both 6g and 7g nodes available it has become intermittent triggering this one.

I'll update with the results from a c7g node when I can get them

seberg · 2023-06-27T12:52:29Z

Thanks, in that case, it may be good to see the difference between the two also (if there is any).

james-takeone · 2023-07-13T05:35:41Z

I am seeing what I believe to be the same issue. I'm also using c7g instances on AWS Batch (EDIT: uses Graviton3). I upgraded from 1.24.3 to 1.25.1 today and all my AWS Batch jobs are failing with exit code 132, which based on some searching seems like is related to invalid instructions. Reverting to 1.24.3 resolved the issue. I instrumented my application to add the following to the entrypoint:

import numpy as np
np.show_runtime()

For context, I'm installing numpy via pip as part of the docker image build process (i.e., I don't build numpy from source). If it's helpful, the docker image is built on an M1 Macbook Pro running Docker Engine v24.0.2.

For 1.24.3 running on c7g instances on AWS Batch, I get the following output:

[{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
'not_found': []}},
{'architecture': 'neoversev1',
'filepath': '/var/task/numpy.libs/libopenblas64_p-r0-cecebdce.3.21.so',
'internal_api': 'openblas',
'num_threads': 8,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.21'}]

For 1.25.1 c7g instances on AWS Batch, the container exits with code 132 prior to printing anything.

For 1.24.3 running on my M1 Macbook Pro, I get the following output:

[{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'armv8',
  'filepath': '/var/task/numpy.libs/libopenblas64_p-r0-cecebdce.3.21.so',
  'internal_api': 'openblas',
  'num_threads': 8,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]

For 1.25.1 running on my M1 Macbook Pro, I get the following output:

[{'numpy_version': '1.25.1',
  'python': '3.10.12 (main, Jul  4 2023, 22:17:40) [GCC 10.2.1 20210110]',
  'uname': uname_result(system='Linux', node='4c76e3da03ce', release='5.15.49-linuxkit-pr', version='#1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023', machine='aarch64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'armv8',
  'filepath': '/var/task/numpy.libs/libopenblas64_p-r0-9fed50d4.3.23.so',
  'internal_api': 'openblas',
  'num_threads': 8,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23'}]

mattip · 2023-07-13T06:00:24Z

There are two unrelated systems to use advanced SIMD instructions in NumPy. One is builtin, and is reported in the simd_extensions section. The other is via OpenBLAS, which has its own runtime dispatch and is reflected in the architecture section.

Could you try limiting the simd_extension runtime dispatch by adding export NPY_DISABLE_CPU_FEATURES="ASIMDHP, ASIMDDP,ASIMDFHM" to the image? If you do this with the successful run of 1.24.3, you should see that change reflected in the simd_extensions entry of the np.show_runtime() output.

If that does not work, the problem might be with OpenBLAS (although I think this is less likely). Its dispatch can be controlled by adding export OPENBLAS_CORETYPE=ARMV8. If you do this with the successful run of 1.24.3, you should see that change reflected in the architecture entry of the np.show_runtime() output.

james-takeone · 2023-07-13T06:31:36Z

Quickly spun up a c7g ec2 instance to test outside of a docker container. Test consisted of creating a virtual env, and installing numpy, and running the python repl.

Here's the output for 1.24.3:

>>> import sys, numpy; print(numpy.__version__); print(sys.version)
1.24.3
3.9.16 (main, Feb 23 2023, 00:00:00) 
[GCC 11.3.1 20221121 (Red Hat 11.3.1-4)]
>>> numpy.show_runtime()
[{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'neoversev1',
  'filepath': '/home/ec2-user/numpy-test/myenv/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-cecebdce.3.21.so',
  'internal_api': 'openblas',
  'num_threads': 2,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]

Here's the output for 1.25.1:

>>> import sys, numpy; print(numpy.__version__); print(sys.version)
1.25.1
3.9.16 (main, Feb 23 2023, 00:00:00) 
[GCC 11.3.1 20221121 (Red Hat 11.3.1-4)]
>>> numpy.show_runtime()
[{'numpy_version': '1.25.1',
  'python': '3.9.16 (main, Feb 23 2023, 00:00:00) \n'
            '[GCC 11.3.1 20221121 (Red Hat 11.3.1-4)]',
  'uname': uname_result(system='Linux', node='ip-10-0-0-191.ec2.internal', release='6.1.34-59.116.amzn2023.aarch64', version='#1 SMP Thu Jun 29 18:11:49 UTC 2023', machine='aarch64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'neoversev1',
  'filepath': '/home/ec2-user/numpy-test/myenv/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-9fed50d4.3.23.so',
  'internal_api': 'openblas',
  'num_threads': 2,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23'}]

james-takeone · 2023-07-13T06:31:57Z

Thanks @mattip, I'll give that a shot tomorrow and report back.

mattip · 2023-07-13T07:21:06Z

test outside of a docker container

Hmm. Outside the container, you can use NumPy 1.25.1. Inside the container, you get an illegal instruction. Perhaps you are pulling in the macos wheel for numpy when building the container, instead of the linux aarch64 one?

james-takeone · 2023-07-13T16:19:18Z

Hmm. Outside the container, you can use NumPy 1.25.1. Inside the container, you get an illegal instruction. Perhaps you are pulling in the macos wheel for numpy when building the container, instead of the linux aarch64 one?

I'm not sure how to check that. However, I was able to run the same image directly on the c7g EC2 instance that fails on AWS Batch. Here is the output:

>>> import sys, numpy; print(numpy.__version__); print(sys.version)
1.25.1
3.10.12 (main, Jul  4 2023, 22:17:40) [GCC 10.2.1 20210110]
>>> numpy.show_runtime()
[{'numpy_version': '1.25.1',
  'python': '3.10.12 (main, Jul  4 2023, 22:17:40) [GCC 10.2.1 20210110]',
  'uname': uname_result(system='Linux', node='b4192cb08e0b', release='6.1.34-59.116.amzn2023.aarch64', version='#1 SMP Thu Jun 29 18:11:49 UTC 2023', machine='aarch64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'neoversev1',
  'filepath': '/var/task/numpy.libs/libopenblas64_p-r0-9fed50d4.3.23.so',
  'internal_api': 'openblas',
  'num_threads': 2,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23'}]

I'm a bit out of my depth now, so I'm going to reach out to AWS business support for some assistance. Thanks for your help @mattip!

martin-frbg · 2023-07-18T08:24:10Z

It occurs to me that we (OpenBLAS) are naively expecting SVE to be available if the detected cpu is known to support it (there's currently no fine-grained checking of hwcap, unlike other platforms) , maybe that is not always a given with containers ?

james-takeone · 2023-07-18T13:03:39Z

@martin-frbg Is there anything that I can do / add to our container to help debug this?

martin-frbg · 2023-07-18T13:11:46Z

If you could add a call to lscpu or something to print the contents of /proc/cpuinfo (if available) to see if "sve" shows up among the feature flags that would be great

james-takeone · 2023-07-18T23:22:16Z

@martin-frbg I was able to run the following directly on the host as well as instrument the container to run the following on entry.

TL;DR
It looks like your hunch was correct, SVE does **not ** appear in the AWS Batch processor feature list, although it does appear when running the commands directly on the host as well as in the image running on Docker.

Note that the same image is used when testing Docker via running on the host C7G instance and AWS Batch.

echo "****** Printing Diagnostic Info *****"
echo "****** start lscpu *****"
lscpu
echo "****** end lscpu *****"
echo "****** start cat /proc/cpuinfo *****"
cat /proc/cpuinfo
echo "****** end cat /proc/cpuinfo *****"

Here are the results:

Directly on the host C7G EC2 instance:

****** Printing Diagnostic Info *****
****** start lscpu *****
Architecture:          aarch64
  CPU op-mode(s):      32-bit, 64-bit
  Byte Order:          Little Endian
CPU(s):                2
  On-line CPU(s) list: 0,1
Vendor ID:             ARM
  Model:               1
  Thread(s) per core:  1
  Core(s) per socket:  2
  Socket(s):           1
  Stepping:            r1p1
  BogoMIPS:            2100.00
  Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc 
                       dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei
                       8mm svebf16 i8mm bf16 dgh rng
Caches (sum of all):   
  L1d:                 128 KiB (2 instances)
  L1i:                 128 KiB (2 instances)
  L2:                  2 MiB (2 instances)
  L3:                  32 MiB (1 instance)
NUMA:                  
  NUMA node(s):        1
  NUMA node0 CPU(s):   0,1
Vulnerabilities:       
  Itlb multihit:       Not affected
  L1tf:                Not affected
  Mds:                 Not affected
  Meltdown:            Not affected
  Mmio stale data:     Not affected
  Retbleed:            Not affected
  Spec store bypass:   Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:          Mitigation; __user pointer sanitization
  Spectre v2:          Mitigation; CSV2, BHB
  Srbds:               Not affected
  Tsx async abort:     Not affected
****** end lscpu *****
****** start cat /proc/cpuinfo *****
processor       : 0
BogoMIPS        : 2100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd40
CPU revision    : 1

processor       : 1
BogoMIPS        : 2100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd40
CPU revision    : 1

****** end cat /proc/cpuinfo *****

Docker running on the C7G EC2 Instance:

****** Printing Diagnostic Info *****
****** start lscpu *****
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           1
Stepping:                        r1p1
BogoMIPS:                        2100.00
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        2 MiB
L3 cache:                        32 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; CSV2, BHB
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
****** end lscpu *****
****** start cat /proc/cpuinfo *****
processor       : 0
BogoMIPS        : 2100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd40
CPU revision    : 1

processor       : 1
BogoMIPS        : 2100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd40
CPU revision    : 1

****** end cat /proc/cpuinfo *****

AWS Batch running on a C7G instance:

****** Printing Diagnostic Info *****
****** start lscpu *****
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           1
Stepping:                        r1p1
BogoMIPS:                        2100.00
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        2 MiB
L3 cache:                        32 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; CSV2, BHB
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs rng
****** end lscpu *****
****** start cat /proc/cpuinfo *****
processor	: 0
BogoMIPS	: 2100.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs rng
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x1
CPU part	: 0xd40
CPU revision	: 1
processor	: 1
BogoMIPS	: 2100.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs rng
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x1
CPU part	: 0xd40
CPU revision	: 1
****** end cat /proc/cpuinfo *****

martin-frbg · 2023-07-19T08:11:16Z

That would mean you could only run with OPENBLAS_CORETYPE=NEOVERSEN1 on AWS Batch, unless AWS support can
tell if/how to enable SVE with AWS Batch. (Without SVE enabled, you might as well use one of the Graviton2 instances which are possibly cheaper)

james-takeone · 2023-07-19T14:08:50Z

Thanks @martin-frbg, this is incredibly helpful. I'll try to sort this out with AWS support.

otterley · 2023-07-19T22:39:42Z

The sve flag in the list provided in /proc/cpuinfo is missing from Amazon Linux 2 with a 4.14 kernel. This kernel ships with the default AMI provided by AWS Batch on c7g instances. If you switch to an AMI with a 5.10 kernel for your Batch compute environment on c7g, then sve will appear in the list and NumPy 1.25.x will run properly.

An AMI ID with the 5.10 kernel compatible with Batch can be found by running the following command (where AWS_REGION is the region in which your Batch compute environment resides):

aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2/kernel-5.10/arm64/recommended/image_id --query 'Parameters[].Value' --output text --region $AWS_REGION

It should be noted that this issue isn't specific to AWS Batch - any Amazon Linux 2 AMI with the older 4.14 kernel in it will have this problem today.

AGSaidi · 2023-07-19T22:54:14Z

4.14 predated the kernel support for SVE, so the kernel doesn't enable access to SVE because it doesn't know how to save and restore the registers. The best solution here would be to check the HWCAP bits to confirm SVE is available before using it in OpenBLAS as just knowing the processor ID supports it isn't really sufficient given the kernel itself might not.

AGSaidi · 2023-07-19T22:57:37Z

~~@martin-frbg one way to address this would be to do the check for sve in detect() and if it fails fallback to the N1. I can work up a patch for this if you'd like.~~
Never mind, i see you already did in OpenMathLib/OpenBLAS@ca7199f

seberg · 2023-07-31T14:42:19Z

Hmmm, seems that OpenBLAS commit isn't yet in the version used by the wheels. Should we bump the commit pin slightly?

mattip · 2023-07-31T16:21:57Z

@martin-frbg thoughts about a stable commit to build against?

martin-frbg · 2023-07-31T16:35:26Z

Depends - I have not committed anything that I'd consider high risk since the fix for this from two weeks ago, but IIUC there should also be a sufficiently recent weekly test build for Scipy that (I think) you might be able to reuse for numpy (testing, at least) ?

mattip · 2023-07-31T16:39:56Z

We have the weekly builds here as well. Thanks, I will start the update process.

Edit: well, we have a PR for weekly builds :)

mattip · 2023-08-01T06:25:34Z

A newer OpenBLAS was built in MacPython/openblas-libs#112

Added scipy (for statistical tests) and ipython. We pin numpy to version 1.24.2, as versions newer than this cause pytho nto crash with SIGILL on AL2 4.14 c7g.metal instances. I expect this will be patched in the upcoming 1.25.3 release. See numpy/numpy#24028 Also drop boto3, which is unneccessary since d1015be. Signed-off-by: Patrick Roy <[email protected]> Co-authored-by: Pablo Barbáchano <[email protected]>

Added scipy (for statistical tests) and ipython. We pin numpy to version 1.24.2, as versions newer than this cause python to crash with SIGILL on AL2 4.14 c7g.metal instances. I expect this will be patched in the upcoming 1.25.3 release. See numpy/numpy#24028 Also drop boto3, which is unneccessary since d1015be. Signed-off-by: Patrick Roy <[email protected]> Co-authored-by: Pablo Barbáchano <[email protected]>

emilyselwood added the 32 - Installation Problems installing or compiling NumPy label Jun 23, 2023

martin-frbg mentioned this issue Jul 19, 2023

Fix runtime detection in ARMV8 DYNAMIC_ARCH to check SVE capability OpenMathLib/OpenBLAS#4150

Merged

kinghuang mentioned this issue Jul 20, 2023

Core dump using dbt-fal 1.5.4 with dbt-core 1.5.2 fal-ai/dbt-fal#890

Closed

mattip mentioned this issue Aug 1, 2023

BLD: update openblas to newer version #24311

Merged

seberg closed this as completed in #24311 Aug 1, 2023

charris mentioned this issue Aug 3, 2023

BLD: update openblas to newer version #24326

Merged

dleemiller mentioned this issue Sep 17, 2024

Fedora Linux: Illegal instruction (core dumped) dleemiller/WordLlama#10

Closed

Uh oh!

numpy 1.25.0 fails with illegal instruction in aws batch #24028

numpy 1.25.0 fails with illegal instruction in aws batch #24028

Comments

emilyselwood commented Jun 23, 2023

Steps to reproduce:

Error message:

Additional information:

seberg commented Jun 23, 2023

Uh oh!

emilyselwood commented Jun 27, 2023

Uh oh!

seberg commented Jun 27, 2023

Uh oh!

james-takeone commented Jul 13, 2023 • edited by rgommers Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Jul 13, 2023

Uh oh!

james-takeone commented Jul 13, 2023

Uh oh!

james-takeone commented Jul 13, 2023

Uh oh!

mattip commented Jul 13, 2023

Uh oh!

james-takeone commented Jul 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martin-frbg commented Jul 18, 2023

Uh oh!

james-takeone commented Jul 18, 2023

Uh oh!

martin-frbg commented Jul 18, 2023

Uh oh!

james-takeone commented Jul 18, 2023

Uh oh!

martin-frbg commented Jul 19, 2023

Uh oh!

james-takeone commented Jul 19, 2023

Uh oh!

otterley commented Jul 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AGSaidi commented Jul 19, 2023

Uh oh!

AGSaidi commented Jul 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Jul 31, 2023

Uh oh!

mattip commented Jul 31, 2023

Uh oh!

martin-frbg commented Jul 31, 2023

Uh oh!

mattip commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Aug 1, 2023

Uh oh!

james-takeone commented Jul 13, 2023 •

edited by rgommers

Loading

james-takeone commented Jul 13, 2023 •

edited

Loading

otterley commented Jul 19, 2023 •

edited

Loading

AGSaidi commented Jul 19, 2023 •

edited

Loading

mattip commented Jul 31, 2023 •

edited

Loading