#28375 triggers errors for pandas < 2.0.0, despite just using scikit-learn internal functionalities.

As documented in https://scikit-learn.org/dev/install.html, we have pandas >= 1.1.3.