conda envs: Reproducibility issues #9257
Description
I open this issue because I now encountered several times that conda cannot resolve an environment.yml file that I created some time ago (see the example below).
I found that a bit disappointing, because having reproducible environments was one of my main motivations for switching to conda in the first place. Also, two popular pipeline engines snakemake and nextflow offer conda integration for reproducibility as some sort of 'docker light'.
- So is reproducibility not something I should reasonably expect from conda?
- What could be done to achieve reproducibility in the future?
I know this has been discussed in #2997, but it staled more than three years ago and I wanted to know what's the current take on that.
Example
Roughly half a year ago I created a conda environment for single-cell RNA-seq analysis and integrated it in a Snakemake pipeline. The environment resolved perfectly. Now, if I want to re-run the analysis, solving the environment fails.
Steps to reproduce:
conda version & channels
> conda --version
conda 4.7.11
For the sake of the test, I commented-out all channels from .condarc (they are included in the yml-file.)
Create the environment
Get the yaml env file and create the environment:
> wget -O single_cell_analysis.yml https://gist.githubusercontent.com/grst/4ef1cd621c30f1ebad1349a872bbe19c/raw/ace5947b0841d9d30ed14fba64e6ce9ccc06372a/single_cell_analysis
> conda env create -n single_cell_analysis -f single_cell_analysis.ymlResult
Collecting package metadata (repodata.json): done
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Package jpeg conflicts for:
r-base=3.5.1 -> jpeg[version='>=9b,<10a|>=9c,<10a']
Package libxml2 conflicts for:
r-base=3.5.1 -> libxml2[version='>=2.9.8,<2.10.0a0|>=2.9.9,<2.10.0a0']
Package libffi conflicts for:
python=3.6 -> libffi[version='3.2.*|>=3.2.1,<3.3.0a0|>=3.2.1,<4.0a0']
Package python conflicts for:
pandas=0.23.4 -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
scikit-learn=0.19.1 -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
networkx=2.1 -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
docopt=0.6.2 -> python[version='2.7.*|3.4.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
scipy=1.0.0 -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0']
intervaltree=2.1.0 -> python[version='2.7.*|3.5.*|3.6.*']
python-annoy=1.11.5 -> python[version='2.7.*|3.5.*|3.6.*']
statsmodels=0.8.0 -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0']
numpy=1.12.0 -> python[version='2.7.*|3.4.*|3.5.*|3.6.*']
scanpy=1.3.7 -> python[version='>=3.6,<3.7.0a0']
matplotlib[version='>2.0.2,<3'] -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
Package r-evaluate conflicts for:
r-rmarkdown=1.10 -> r-evaluate[version='>=0.8']
Package pandoc conflicts for:
r-rmarkdown=1.10 -> pandoc[version='>=1.12.3,<3.0']
Package freetype conflicts for:
r-base=3.5.1 -> freetype[version='>=2.9.1,<3.0a0']
matplotlib[version='>2.0.2,<3'] -> freetype[version='>=2.8,<2.9.0a0|>=2.9.1,<3.0a0']
Package python-dateutil conflicts for:
matplotlib[version='>2.0.2,<3'] -> python-dateutil
pandas=0.23.4 -> python-dateutil[version='>=2.5.*']
Package _r-mutex conflicts for:
r-base=3.5.1 -> _r-mutex=1[build=anacondar_1]
r-rmarkdown=1.10 -> _r-mutex=1[build='anacondar_1|mro_2']
r-reticulate=1.10 -> _r-mutex=1[build=anacondar_1]
Package r-base conflicts for:
r-rmarkdown=1.10 -> r-base[version='>=3.4.1,<3.4.2.0a0|>=3.5.1,<3.5.2.0a0']
r-reticulate=1.10 -> r-base[version='>=3.4.1,<3.4.2.0a0|>=3.5.1,<3.5.2.0a0']
r-bookdown=0.7 -> r-base[version='>=3.4.1,<3.4.2.0a0|>=3.5.1,<3.5.2.0a0']
Package blas conflicts for:
numpy=1.12.0 -> blas==1.1=openblas
scipy=1.0.0 -> blas[version='*|*|1.1',build='openblas|mkl|openblas']
scikit-learn=0.19.1 -> blas[version='*|*|1.0|1.1',build='openblas|mkl|openblas|mkl|openblas']
Package tk conflicts for:
r-base=3.5.1 -> tk[version='>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0']
matplotlib[version='>2.0.2,<3'] -> tk[version='8.6.*|8.6.*,>=8.6.7,<8.7.0a0']
python=3.6 -> tk[version='8.5.*|8.6.*|>=8.6.7,<8.7.0a0|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0']
Package numpy conflicts for:
statsmodels=0.8.0 -> numpy[version='>=1.11|>=1.9.3,<2.0a0']
scipy=1.0.0 -> numpy[version='>=1.9|>=1.9.3,<2.0a0']
matplotlib[version='>2.0.2,<3'] -> numpy
scikit-learn=0.19.1 -> numpy[version='>=1.11.3,<2.0a0|>=1.9|>=1.9.3,<2.0a0']
pandas=0.23.4 -> numpy[version='>=1.11.3,<2.0a0|>=1.9.3,<2.0a0']
Package glib conflicts for:
r-base=3.5.1 -> glib[version='>=2.56.2,<2.57.0a0|>=2.58.3,<3.0a0']
Package pip conflicts for:
python=3.6 -> pip
Package r-htmltools conflicts for:
r-rmarkdown=1.10 -> r-htmltools[version='>=0.3.5']
r-bookdown=0.7 -> r-htmltools[version='>=0.3.6']
Package r-tinytex conflicts for:
r-rmarkdown=1.10 -> r-tinytex[version='>=0.3']
r-bookdown=0.7 -> r-tinytex
Package libpng conflicts for:
matplotlib[version='>2.0.2,<3'] -> libpng[version='>=1.6.32,<1.7.0a0|>=1.6.34,<1.7.0a0']
r-base=3.5.1 -> libpng[version='>=1.6.34,<1.7.0a0|>=1.6.35,<1.7.0a0|>=1.6.37,<1.7.0a0']
Package r-yaml conflicts for:
r-bookdown=0.7 -> r-yaml[version='>=2.1.14']
r-rmarkdown=1.10 -> r-yaml[version='>=2.1.5']
Package statsmodels conflicts for:
scanpy=1.3.7 -> statsmodels
Package scikit-learn conflicts for:
scanpy=1.3.7 -> scikit-learn
Package mro-base conflicts for:
r-rmarkdown=1.10 -> mro-base[version='>=3.5.1,<3.5.2.0a0']
Package pango conflicts for:
r-base=3.5.1 -> pango[version='>=1.40.14,<1.41.0a0|>=1.42.4,<1.43.0a0|>=1.42.4,<2.0a0']
Package decorator conflicts for:
networkx=2.1 -> decorator[version='>=3.4.0|>=4.1.0']
Package openssl conflicts for:
python=3.6 -> openssl[version='1.0.*|1.0.*,>=1.0.2l,<1.0.3a|>=1.0.2m,<1.0.3a|>=1.0.2n,<1.0.3a|>=1.0.2o,<1.0.3a|>=1.0.2p,<1.0.3a|>=1.1.1a,<1.1.2a|>=1.1.1c,<1.1.2a']
Package libgcc-ng conflicts for:
scikit-learn=0.19.1 -> libgcc-ng[version='>=7.2.0']
r-reticulate=1.10 -> libgcc-ng[version='>=4.9|>=7.3.0']
python=3.6 -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
pandas=0.23.4 -> libgcc-ng[version='>=4.9|>=7.2.0|>=7.3.0']
matplotlib[version='>2.0.2,<3'] -> libgcc-ng[version='>=7.2.0']
scipy=1.0.0 -> libgcc-ng[version='>=7.2.0']
r-base=3.5.1 -> libgcc-ng[version='>=4.9|>=7.3.0']
statsmodels=0.8.0 -> libgcc-ng[version='>=7.2.0']
Package libcurl conflicts for:
r-base=3.5.1 -> libcurl[version='>=7.61.1,<8.0a0|>=7.62.0,<8.0a0|>=7.64.1,<8.0a0']
Package cycler conflicts for:
matplotlib[version='>2.0.2,<3'] -> cycler[version='>=0.10']
Package r-jsonlite conflicts for:
r-rmarkdown=1.10 -> r-jsonlite
r-reticulate=1.10 -> r-jsonlite
Package openblas conflicts for:
scikit-learn=0.19.1 -> openblas[version='0.2.19|0.2.19.*|0.2.20|0.2.20.*']
scipy=1.0.0 -> openblas[version='0.2.20|0.2.20.*']
numpy=1.12.0 -> openblas[version='0.2.19|0.2.19.*']
Package matplotlib-base conflicts for:
matplotlib[version='>2.0.2,<3'] -> matplotlib-base[version='2.2.3|2.2.4|2.2.4|2.2.4',build='py37hfd891ef_0|py27h60b886d_1|py36h60b886d_1|py37h60b886d_1|py27hfd891ef_0|py36hfd891ef_0']
Package h5py conflicts for:
scanpy=1.3.7 -> h5py
Package r-rcpp conflicts for:
r-reticulate=1.10 -> r-rcpp[version='>=0.12.7']
Package pcre conflicts for:
r-base=3.5.1 -> pcre[version='>=8.41,<9.0a0|>=8.42,<9.0a0']
Package r-rmarkdown conflicts for:
r-bookdown=0.7 -> r-rmarkdown[version='>=1.5']
Package r-knitr conflicts for:
r-rmarkdown=1.10 -> r-knitr[version='>=1.18']
r-bookdown=0.7 -> r-knitr[version='>=1.16']
Package natsort conflicts for:
scanpy=1.3.7 -> natsort
Package cairo conflicts for:
r-base=3.5.1 -> cairo[version='>=1.14.12,<1.15.0a0|>=1.14.12,<2.0a0|>=1.16.0,<1.17.0a0']
Package pytz conflicts for:
pandas=0.23.4 -> pytz
matplotlib[version='>2.0.2,<3'] -> pytz
Package xorg-libsm conflicts for:
r-rmarkdown=1.10 -> xorg-libsm=1.2
Package r-xfun conflicts for:
r-bookdown=0.7 -> r-xfun
Package make conflicts for:
r-base=3.5.1 -> make
Package libgfortran conflicts for:
scipy=1.0.0 -> libgfortran
r-base=3.5.1 -> libgfortran[version='>=3.0']
Package gcc_linux-64 conflicts for:
r-base=3.5.1 -> gcc_linux-64=7
Package r-mime conflicts for:
r-rmarkdown=1.10 -> r-mime
Package bzip2 conflicts for:
python=3.6 -> bzip2[version='>=1.0.6,<2.0a0']
r-base=3.5.1 -> bzip2[version='>=1.0.6,<2.0a0|>=1.0.8,<2.0a0']
Package louvain conflicts for:
scanpy=1.3.7 -> louvain
Package matplotlib conflicts for:
scanpy=1.3.7 -> matplotlib[version='>=2.2']
Package xz conflicts for:
r-base=3.5.1 -> xz[version='>=5.2.4,<5.3.0a0|>=5.2.4,<6.0a0']
python=3.6 -> xz[version='5.2.*|>=5.2.3,<5.3.0a0|>=5.2.3,<6.0a0|>=5.2.4,<5.3.0a0|>=5.2.4,<6.0a0']
Package gfortran_linux-64 conflicts for:
r-base=3.5.1 -> gfortran_linux-64=7
Package ncurses conflicts for:
r-base=3.5.1 -> ncurses[version='>=6.1,<7.0a0']
python=3.6 -> ncurses[version='5.9|5.9.*|6.0.*|>=6.0,<7.0a0|>=6.1,<6.2.0a0|>=6.1,<7.0a0']
Package xorg-libxext conflicts for:
r-rmarkdown=1.10 -> xorg-libxext=1.3
Package scipy conflicts for:
scanpy=1.3.7 -> scipy
statsmodels=0.8.0 -> scipy
scikit-learn=0.19.1 -> scipy
Package pytables conflicts for:
scanpy=1.3.7 -> pytables
Package pandas conflicts for:
statsmodels=0.8.0 -> pandas
scanpy=1.3.7 -> pandas[version='>=0.21']
Package sqlite conflicts for:
python=3.6 -> sqlite[version='3.13.*|3.20.*|>=3.20.1,<4.0a0|>=3.22.0,<4.0a0|>=3.23.1,<4.0a0|>=3.24.0,<4.0a0|>=3.25.2,<4.0a0|>=3.25.3,<4.0a0|>=3.26.0,<4.0a0|>=3.28.0,<4.0a0|>=3.29.0,<4.0a0']
Package libssh2 conflicts for:
r-base=3.5.1 -> libssh2[version='>=1.8.0,<1.9.0a0|>=1.8.0,<2.0a0|>=1.8.2,<1.9.0a0']
Package seaborn conflicts for:
scanpy=1.3.7 -> seaborn
Package libgfortran-ng conflicts for:
r-base=3.5.1 -> libgfortran-ng[version='>=7,<8.0a0']
scipy=1.0.0 -> libgfortran-ng[version='>=7.2.0,<8.0a0']
Package zlib conflicts for:
r-base=3.5.1 -> zlib[version='>=1.2.11,<1.3.0a0']
matplotlib[version='>2.0.2,<3'] -> zlib[version='>=1.2.11,<1.3.0a0']
python=3.6 -> zlib[version='1.2.*|1.2.11|1.2.8|>=1.2.11,<1.3.0a0']
Package libstdcxx-ng conflicts for:
pandas=0.23.4 -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0']
matplotlib[version='>2.0.2,<3'] -> libstdcxx-ng[version='>=7.2.0']
scikit-learn=0.19.1 -> libstdcxx-ng[version='>=7.2.0']
r-base=3.5.1 -> libstdcxx-ng[version='>=4.9|>=7.3.0']
python=3.6 -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0']
r-reticulate=1.10 -> libstdcxx-ng[version='>=4.9|>=7.3.0']
scipy=1.0.0 -> libstdcxx-ng[version='>=7.2.0']
Package python-igraph conflicts for:
scanpy=1.3.7 -> python-igraph
Package readline conflicts for:
r-base=3.5.1 -> readline[version='>=7.0,<8.0a0|>=8.0,<9.0a0']
python=3.6 -> readline[version='6.2.*|7.*|7.0|>=7.0,<8.0a0|>=8.0,<9.0a0']
Package * conflicts for:
scipy=1.0.0 -> *[track_features=blas_openblas]
numpy=1.12.0 -> *[track_features=blas_openblas]
scikit-learn=0.19.1 -> *[track_features=blas_openblas]
Package subprocess32 conflicts for:
matplotlib[version='>2.0.2,<3'] -> subprocess32
Package icu conflicts for:
matplotlib[version='>2.0.2,<3'] -> icu[version='>=58.2,<59.0a0']
r-base=3.5.1 -> icu[version='>=58.2,<59.0a0|>=64.2,<65.0a0']
Package setuptools conflicts for:
matplotlib[version='>2.0.2,<3'] -> setuptools
scanpy=1.3.7 -> setuptools
networkx=2.1 -> setuptools
Package bwidget conflicts for:
r-base=3.5.1 -> bwidget
Package anndata conflicts for:
scanpy=1.3.7 -> anndata[version='>=0.6.10']
Package libuuid conflicts for:
r-base=3.5.1 -> libuuid[version='>=1.0.3,<2.0a0|>=2.32.1,<3.0a0']
Package gxx_linux-64 conflicts for:
r-base=3.5.1 -> gxx_linux-64=7
Package libopenblas conflicts for:
scipy=1.0.0 -> libopenblas[version='>=0.2.20,<0.2.21.0a0']
scikit-learn=0.19.1 -> libopenblas[version='>=0.2.20,<0.2.21.0a0']
Package kiwisolver conflicts for:
matplotlib[version='>2.0.2,<3'] -> kiwisolver
Package pyqt conflicts for:
matplotlib[version='>2.0.2,<3'] -> pyqt[version='5.*,>=5.9.2,<5.10.0a0|5.6.*|5.9.*|>=5.6,<6.0a0|>=5.6.0,<5.7.0a0|>=5.9.2,<5.10.0a0']
Package r-base64enc conflicts for:
r-rmarkdown=1.10 -> r-base64enc
Package sortedcontainers conflicts for:
intervaltree=2.1.0 -> sortedcontainers
Package libtiff conflicts for:
r-base=3.5.1 -> libtiff[version='>=4.0.10,<5.0a0|>=4.0.9,<5.0a0']
Package krb5 conflicts for:
r-base=3.5.1 -> krb5[version='>=1.16.1,<1.17.0a0|>=1.16.2,<1.17.0a0|>=1.16.3,<1.17.0a0']
Package backports.functools_lru_cache conflicts for:
matplotlib[version='>2.0.2,<3'] -> backports.functools_lru_cache
Package r-matrix conflicts for:
r-reticulate=1.10 -> r-matrix
Package tktable conflicts for:
r-base=3.5.1 -> tktable
Package tornado conflicts for:
matplotlib[version='>2.0.2,<3'] -> tornado
Package gsl conflicts for:
r-base=3.5.1 -> gsl[version='>=2.2.1,<2.3.0a0|>=2.4,<2.5.0a0|>=2.5,<2.6.0a0']
Package r-rprojroot conflicts for:
r-rmarkdown=1.10 -> r-rprojroot
Package xorg-libxrender conflicts for:
r-rmarkdown=1.10 -> xorg-libxrender=0.9
Package functools32 conflicts for:
matplotlib[version='>2.0.2,<3'] -> functools32
Package curl conflicts for:
r-base=3.5.1 -> curl
Package r-stringr conflicts for:
r-rmarkdown=1.10 -> r-stringr[version='>=1.2.0']
Package patsy conflicts for:
statsmodels=0.8.0 -> patsy
Package numba conflicts for:
scanpy=1.3.7 -> numba
Package mkl conflicts for:
scikit-learn=0.19.1 -> mkl[version='>=2018.0.0,<2019.0a0|>=2018.0.3,<2019.0a0']
scipy=1.0.0 -> mkl[version='>=2018.0.0,<2019.0a0']
Package networkx conflicts for:
scanpy=1.3.7 -> networkx
Package pyparsing conflicts for:
matplotlib[version='>2.0.2,<3'] -> pyparsing