Sycl test failures with intel oneapi 2024.2.1

The following tests are failing when testing with the Sycl backend on PVC arch in containerized testing:

```
The following tests FAILED:
   11 - blas_sycl (Failed)
   17 - sparse_sycl (Failed)
   18 - blocksparse_sycl (Failed)
```

More detailed output:

**blas_sycl**:
```
[ RUN      ] sycl_test.gemv_complex_double
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-6.3513,5.11578), h_y(i)=(-0.182747,-0.221999)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-5.73633,-8.32032), h_y(i)=(-2.78235,-6.85526)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-2.99421,6.36861), h_y(i)=(-3.32041,5.05302)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(4.34386,2.34132), h_y(i)=(9.68928,-0.660492)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-14.9297,-3.40846), h_y(i)=(-4.4837,-2.31755)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-1.23367,1.26546), h_y(i)=(-3.58353,-1.58239)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(1.18362,-5.81078), h_y(i)=(-0.666861,-4.31589)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-1.9987,4.08174), h_y(i)=(-0.2554,7.11416)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(3.94619,-0.926831), h_y(i)=(5.45093,1.06786)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-5.12418,-4.75785), h_y(i)=(-1.60102,0.164905)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-5.68494,7.78024), h_y(i)=(-2.15898,2.76716)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-3.56422,-2.89425), h_y(i)=(-4.84676,-0.28637)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:89: expected(i)=(-1.65791,-2.5608), h_y(i)=(-5.06562,-6.23372)
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:93: Failure
Expected equality of these values:
  numErrors
    Which is: 13
  0
Nonconst input, 13x13, alpha = (3,0), beta = (5,0), mode N: gemv incorrect
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:104: Failure
Expected equality of these values:
  numErrors
    Which is: 13
  0
Const vector input, 13x13, alpha = (3,0), beta = (5,0), mode N: gemv incorrect
/var/kokkos-kernels/blas/unit_test/Test_Blas2_gemv.hpp:114: Failure
Expected equality of these values:
  numErrors
    Which is: 13
  0
Const matrix/vector input, 13x13, alpha = (3,0), beta = (5,0), mode N: gemv incorrect
...

[ RUN      ] sycl_test.gemm_complex_double
Result: 7.453916e+01 2.949100e-11
/var/kokkos-kernels/blas/unit_test/Test_Blas3_gemm.hpp:231: Failure
Value of: (diff_C_average < 1.05 * diff_C_expected)
  Actual: false
Expected: true
Result: 7.509380e+01 2.949100e-11
/var/kokkos-kernels/blas/unit_test/Test_Blas3_gemm.hpp:231: Failure
Value of: (diff_C_average < 1.05 * diff_C_expected)
  Actual: false
Expected: true
Result: 2.506452e+01 8.170957e-12
/var/kokkos-kernels/blas/unit_test/Test_Blas3_gemm.hpp:295: Failure
Value of: (diff_C1_average < 1.05 * diff_C_expected)
  Actual: false
Expected: true
[  FAILED  ] sycl_test.gemm_complex_double (2696 ms)
```

**sparse_sycl**
```
[ RUN      ] sycl_test.sparse_spgemm_kokkos_complex_double_int_int_TestDevice
Values at index 90564, -399.648837 + 638.106131i and -90.396234 + 1182.969738i, differ too much (eps = 1.000000e-07, rel err = 3.230518e-01)
Values at index 90451, -2100.863880 + 4921.010308i and -2240.409924 + 7477.858185i, differ too much (eps = 1.000000e-07, rel err = 1.946234e-01)
Values at index 90283, 838.269796 + 2160.977407i and 1011.130006 + 2675.647265i, differ too much (eps = 1.000000e-07, rel err = 1.048479e-01)
Values at index 331846, 366.398783 + 1916.388550i and -417.653344 + 4025.614072i, differ too much (eps = 1.000000e-07, rel err = 3.751445e-01)
Values at index 331856, -1083.005506 + 2780.505743i and -2602.652846 + 3702.299711i, differ too much (eps = 1.000000e-07, rel err = 2.366808e-01)
Values at index 331858, 1291.656436 + 3642.077774i and 2035.513760 + 6425.605793i, differ too much (eps = 1.000000e-07, rel err = 2.716929e-01)
Values at index 331859, 2364.684012 + 2231.726175i and 1141.968906 + 3335.181053i, differ too much (eps = 1.000000e-07, rel err = 2.430373e-01)
```

**blocksparse_sycl**
```
[ RUN      ] sycl_test.sparse_block_spgemm_kokkos_complex_double_int_int_TestDevice
Values at index 2176, -1498.726137 + 2411.252211i and 634.426715 + -4131.915627i, differ too much (eps = 5.000000e-07, rel err = 9.804398e-01)
Values at index 2177, 132.861989 + -2325.549187i and 2897.311289 + 3123.432388i, differ too much (eps = 5.000000e-07, rel err = 9.272298e-01)
Values at index 2178, -2447.580954 + 1494.045837i and -6537.099215 + 1790.750803i, differ too much (eps = 5.000000e-07, rel err = 4.250970e-01)
Values at index 2179, 1800.488065 + -4077.669258i and 1835.076244 + -2538.182008i, differ too much (eps = 5.000000e-07, rel err = 2.028941e-01)
Values at index 2180, -2262.620489 + 1799.944738i and -7844.347816 + 2653.981930i, differ too much (eps = 5.000000e-07, rel err = 5.054145e-01)
...
[ RUN      ] sycl_test.sparse_bsr_spmv_kokkos_complex_double_int_int_TestDevice
/var/kokkos-kernels/sparse/unit_test/Test_Sparse_spmv_bsr.hpp:189 BsrMatrix SpMV failure!
alg:          SPMV_DEFAULT
mode:         T
A:            10x10
A blockdim:   2
alpha:        (1,0)
beta:         (0,0)
maxNnzPerRow: 18
First 100 errors:
y exp act err tol
- --- --- --- ---
0 (-80.767,292.792) (58.7568,192.374) 171.903 7.99361e-12
1 (-196.075,311.251)  (-56.5514,210.833)  171.903 7.99361e-12
6 (53.1863,146.264) (38.5696,92.176)  56.0283 7.99361e-12
7 (-46.212,170.418) (-10.3055,100.584)  78.5246 7.99361e-12
/var/kokkos-kernels/sparse/unit_test/Test_Sparse_spmv_bsr.hpp:213: Failure
Value of: errIdx.empty()
  Actual: false
Expected: true
/var/kokkos-kernels/sparse/unit_test/Test_Sparse_spmv_bsr.hpp:189 BsrMatrix SpMV failure!
alg:          SPMV_DEFAULT
mode:         T
A:            10x10
A blockdim:   2
alpha:        (1,0)
beta:         (1,0)
maxNnzPerRow: 18
First 100 errors:
y exp act err tol
- --- --- --- ---
0 (-77.5549,302.203)  (16.4674,97.811)  224.98  7.99675e-12
1 (-186.564,313.266)  (-8.42754,89.7261)  285.836 7.99675e-12
/var/kokkos-kernels/sparse/unit_test/Test_Sparse_spmv_bsr.hpp:213: Failure
Value of: errIdx.empty()
  Actual: false
Expected: true
...

[ RUN      ] sycl_test.sparse_bsr_spmmv_kokkos_complex_double_int_int_LayoutLeft_TestDevice
/var/kokkos-kernels/sparse/unit_test/Test_Sparse_spmv_bsr.hpp:516 BsrMatrix SpMMV failure!
alg:          SPMV_DEFAULT
mode:         T
A:            10x10
A blockdim:   2
alpha:        (1,0)
beta:         (0,0)
maxNnzPerRow: 20
First 100 errors:
i j exp act err tol
- - --- --- --- ---
0 0 (-25.4638,280.061)  (29.7172,87.2934) 200.51  8.88178e-12
1 0 (0.0983221,252.422) (55.2794,59.6547) 200.51  8.88178e-12
6 0 (-124.019,331.874)  (0,0) 354.289 8.88178e-12
7 0 (-73.2898,272.941)  (0,0) 282.61  8.88178e-12
14  0 (27.4019,54.1358) (0,0) 60.6758 8.88178e-12
15  0 (15.9583,89.6957) (0,0) 91.1042 8.88178e-12
16  0 (0.960895,97.3204)  (0,0) 97.3251 8.88178e-12
17  0 (-14.8097,45.7038)  (0,0) 48.0434 8.88178e-12
/var/kokkos-kernels/sparse/unit_test/Test_Sparse_spmv_bsr.hpp:542: Failure
Value of: errIdx.empty()
  Actual: false
Expected: true
/var/kokkos-kernels/sparse/unit_test/Test_Sparse_spmv_bsr.hpp:516 BsrMatrix SpMMV failure!
alg:          SPMV_DEFAULT
mode:         T
A:            10x10
A blockdim:   2
alpha:        (1,0)
beta:         (1,0)
maxNnzPerRow: 20
First 100 errors:
i j exp act err tol
- - --- --- --- ---
6 0 (-115.07,333.658) (-100.306,267.323)  67.9589 8.88492e-12
7 0 (-66.4648,281.731)  (-53.3612,235.461)  48.0901 8.88492e-12
14  0 (32.3214,62.4922) (4.91956,8.35635) 60.6758 8.88492e-12
15  0 (18.3279,97.8607) (2.36955,8.16502) 91.1042 8.88492e-12
16  0 (8.53592,102.981) (7.57503,5.66055) 97.3251 8.88492e-12
17  0 (-12.7579,46.9862)  (2.05173,1.28236) 48.0434 8.88492e-12

```


**Reproducer notes** (blake PV queue):
```bash
podman run -it --rm --device /dev/dri:/dev/dri --mount type=bind,src=${HOME}/kokkos,dst=/var/kokkos  --mount type=bind,src=${HOME}/kokkos-kernels,dst=/var/kokkos-kernels intel/oneapi-basekit:2024.2.1-0-devel-ubuntu22.04 bash

cd /var

cmake -S $PWD/kokkos \
            -B $PWD/kokkos-build \
            -DCMAKE_INSTALL_PREFIX=$PWD/kokkos-install \
            -DBUILD_SHARED_LIBS=ON \
            -DCMAKE_CXX_STANDARD=17 \
            -DCMAKE_BUILD_TYPE=Release \
            -DCMAKE_VERBOSE_MAKEFILE=ON \
            -DCMAKE_CXX_COMPILER=icpx \
            -DKokkos_ARCH_NATIVE=ON \
            -DKokkos_ENABLE_SYCL=ON \
            -DKokkos_ARCH_INTEL_PVC=ON \
            -DKokkos_ENABLE_DEPRECATED_CODE_4=OFF

time cmake --build $PWD/kokkos-build --parallel 32

time cmake --install $PWD/kokkos-build --prefix $PWD/kokkos-install

cmake -S $PWD/kokkos-kernels \
            -B $PWD/kokkos-kernels-build \
            -DCMAKE_INSTALL_PREFIX=$PWD/kokkos-kernels-install \
            -DBUILD_SHARED_LIBS=ON \
            -DCMAKE_CXX_FLAGS="-fsycl-device-code-split=per_kernel -fp-model=precise" \
            -DCMAKE_CXX_STANDARD=17 \
            -DCMAKE_BUILD_TYPE=Release \
            -DCMAKE_VERBOSE_MAKEFILE=OFF \
            -DCMAKE_CXX_COMPILER=icpx \
            -DKokkos_ROOT=$PWD/kokkos-install \
            -DKokkosKernels_INST_COMPLEX_DOUBLE:BOOL=ON \
            -DKokkosKernels_ENABLE_TESTS:BOOL=ON \
            -DKokkosKernels_ENABLE_EXAMPLES:BOOL=ON \
            -DKokkosKernels_ENABLE_PERFTESTS:BOOL=OFF \
            -DKokkosKernels_ENABLE_BENCHMARK:BOOL=OFF

export ZES_ENABLE_SYSMAN=1

time cmake --build $PWD/kokkos-kernels-build --parallel 32

export ONEAPI_DEVICE_SELECTOR=level_zero:gpu
ctest

```


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Sycl test failures with intel oneapi 2024.2.1 #2739

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sycl test failures with intel oneapi 2024.2.1 #2739

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions