Nightly test failures with Trilinos: various ifpack2 unit test failures #2622
Description
Nightly builds of Trilinos have the following ifpack2 unit test failures after recent updates to kokkos-kernels@develop:
22:46:17 The following tests FAILED:
22:46:17 788 - Ifpack2_RILUK_2streams_hb_belos_MPI_2 (Failed)
22:46:17 789 - Ifpack2_RILUK_4streams_hb_belos_MPI_2 (Failed)
22:46:17 790 - Ifpack2_RILUK_2streams_hb_belos_MPI_4 (Failed)
22:46:17 791 - Ifpack2_RILUK_4streams_hb_belos_MPI_4 (Failed)
22:46:17 792 - Ifpack2_RILUK_2streams_rcm_hb_belos_MPI_2 (Failed)
22:46:17 793 - Ifpack2_RILUK_4streams_rcm_hb_belos_MPI_2 (Failed)
22:46:17 794 - Ifpack2_RILUK_2streams_rcm_hb_belos_MPI_4 (Failed)
22:46:17 795 - Ifpack2_RILUK_4streams_rcm_hb_belos_MPI_4 (Failed)
22:46:17 826 - Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 (Failed)
Added details from Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4:
...
826: Running unit tests ...
826:
826: SUCCESS: test_BR_BTDC (A = I + R) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 1 rd 1.30549e-16
826: 0. Ifpack2BlockTriDi_double_int_longlong_Unit_UnitTest ... FAIL: test_BR_BTDC (A = D) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 1 rd 0.0336664
826: FAIL: test_BR_BTDC (A = D + R) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 1 rd 0.00970669
826: SUCCESS: test_BR_BTDC (A = D + R, norm) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 1 r 0.000332895
826: SUCCESS: test_BR_BTDC (A = D + R, damping factor) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 1 rd 5.55261e-18 eps 2.22045e-15
826: SUCCESS: test_BR_BTDC (A = D + R, damping factor) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 1 rd 5.55261e-18 eps 2.22045e-15
826: SUCCESS: test_BR_BTDC (A = I) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 1 rd 0
826: SUCCESS: test_BR_BTDC (A = I + R) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 3 rd 1.49476e-16
826: FAIL: test_BR_BTDC (A = D) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 3 rd 0.277114
826: FAIL: test_BR_BTDC (A = D + R) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 3 rd 0.151297
826: SUCCESS: test_BR_BTDC (A = D + R, norm) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 3 r 0.000338672
826: SUCCESS: test_BR_BTDC (A = D + R, damping factor) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 3 rd 3.65185e-18 eps 2.22045e-15
826: SUCCESS: test_BR_BTDC (A = D + R, damping factor) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 3 rd 3.65185e-18 eps 2.22045e-15
826: SUCCESS: test_BR_BTDC (A = I) test_BR_BTDC: bs 5 contig tridiag uniform_lines nvec 3 rd 0
826: SUCCESS: test_BR_BTDC (A = I + R) test_BR_BTDC: bs 5 contig tridiag point_wise uniform_lines nvec 1 rd 1.22241e-16
826: FAIL: test_BR_BTDC (A = D) test_BR_BTDC: bs 5 contig tridiag point_wise uniform_lines nvec 1 rd 0.0336664
826: FAIL: test_BR_BTDC (A = D + R) test_BR_BTDC: bs 5 contig tridiag point_wise uniform_lines nvec 1 rd 0.00970669
...
These test failures began after merge of these commits:
Changes
osx workflow: removing explicit parallelism levels (#2618) ([detail](https://jenkins-son.sandia.gov/job/KokkosEco_Trilinos_KokkosDev2_SemsCuda1180-nouvm/575/changes#detail))
sptrsv_cusparse: must call cusparseSpSV_analysis when matrix values have changed (#2568) ([detail](https://jenkins-son.sandia.gov/job/KokkosEco_Trilinos_KokkosDev2_SemsCuda1180-nouvm/575/changes#detail))
Update changelog and workflow yml files for 4.6.01 (#2620) ([detail](https://jenkins-son.sandia.gov/job/KokkosEco_Trilinos_KokkosDev2_SemsCuda1180-nouvm/575/changes#detail))
ConjTrans support for batched team gemm (#2580) ([detail](https://jenkins-son.sandia.gov/job/KokkosEco_Trilinos_KokkosDev2_SemsCuda1180-nouvm/575/changes#detail))
Reproducer notes (kokkos-dev-2):
# Repos
git clone -b develop https://github.com/trilinos/Trilinos.git
TRILINOS_DIR=$PWD/Trilinos
git clone -b develop https://github.com/kokkos/kokkos.git
KOKKOS_DIR=$PWD/kokkos
git clone -b develop https://github.com/kokkos/kokkos-kernels.git
KOKKOSKERNELS_DIR=$PWD/kokkos-kernels
cd $TRILINOS_DIR
ln -s ${PWD}/../kokkos kokkos
ln -s ${PWD}/../kokkos-kernels kokkos-kernels
cd ..
mkdir -p Build
cd Build
# Environment
module load sems-gcc/11.4.0 sems-cuda/11.8.0 sems-openmpi/4.1.6 sems-cmake sems-openblas/0.3.26 sems-git sems-ninja
module list
export OMPI_CXX=$KOKKOS_DIR/bin/nvcc_wrapper
# Configure Trilinos
cmake \
-GNinja \
-D CMAKE_INSTALL_PREFIX="${TRILINOS_INSTALL_DIR}" \
-D CMAKE_BUILD_TYPE:STRING=RELEASE \
-D BUILD_SHARED_LIBS:BOOL=OFF \
-DTPL_ENABLE_MPI=ON \
-DTPL_ENABLE_BLAS:STRING=ON \
-DBLAS_LIBRARY_DIRS:FILEPATH="$OPENBLAS_ROOT/lib" \
-DBLAS_LIBRARY_NAMES:STRING="openblas" \
-DTPL_ENABLE_LAPACK:STRING=ON \
-DLAPACK_INCLUDE_DIRS:FILEPATH="$OPENBLAS_ROOT/include" \
-DLAPACK_LIBRARY_DIRS:FILEPATH="$OPENBLAS_ROOT/lib" \
-DLAPACK_LIBRARY_NAMES:STRING="openblas" \
-DTPL_ENABLE_CUSPARSE:BOOL=ON \
-DTrilinos_ENABLE_ALL_PACKAGES=OFF \
-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
-DTrilinos_ENABLE_TESTS=ON \
-DTrilinos_MUST_FIND_ALL_TPL_LIBS=TRUE \
-DTrilinos_ENABLE_COMPLEX=ON \
-DTrilinos_ENABLE_OpenMP=OFF \
-DTrilinos_ENABLE_Kokkos=ON \
-D Kokkos_ENABLE_SERIAL=ON \
-D Kokkos_ENABLE_CUDA=ON \
-D Kokkos_ENABLE_TESTS=ON \
-D Kokkos_ARCH_VOLTA70=ON \
-D Kokkos_ENABLE_DEPRECATED_CODE_4=OFF \
-D Kokkos_ENABLE_IMPL_VIEW_LEGACY=ON \
-DTrilinos_ENABLE_KokkosKernels=ON \
-D KokkosKernels_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Tpetra=ON \
-D Tpetra_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Sacado=ON \
-D Sacado_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Stokhos=ON \
-D Stokhos_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Adelus=ON \
-D Adelus_ENABLE_TESTS=OFF \
-DTrilinos_ENABLE_Compadre=ON \
-D Compadre_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Amesos2=ON \
-D Amesos2_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Zoltan2=ON \
-D Zoltan2_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Ifpack2=ON \
-D Ifpack2_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Belos=ON \
-D Belos_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Phalanx=ON \
-D Phalanx_ENABLE_TESTS=ON \
-DTrilinos_ENABLE_Intrepid2=ON \
-D Intrepid2_ENABLE_TESTS=OFF \
\
-DTPL_ENABLE_Matio=OFF \
\
-DKokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \
-DKokkosKernels_SOURCE_DIR_OVERRIDE:STRING=kokkos-kernels \
\
-DStokhos_TpetraCrsMatrixUQPCEUnitTest_Cuda_MPI_4_DISABLE=ON \
\
-DTrilinos_ENABLE_INSTALLATION_TESTING=OFF \
$TRILINOS_DIR
make -j16