Work in progress towards NNPDF 4.1

This is a minor release is the starting line (and contains the base runcard) for 4.1, with feature scaling, charm asymmetry, updated and debugged versions of the data and (mostly) NNLO grids.

No big changes with respect to 4.1.0, mostly minor bugfixes, some relevant points to note:

• The framework is now compatible with JaX (which joins Tensorflow and pyTorch) #2342
• Updated hyperopt figure of merit #2317 and the possibility to run multiple architectures at once #2314
• NNPDF is now an affiliated project of the HEP Software Foundation https://hepsoftwarefoundation.org/projects/projects.html
• The NIKHEF cluster is now the main theory server

While not attached to a PDF release, this is a major release of the framework as many things have changed in the backend and legacy code has been removed. What follows is a (non-exhaustive) list of breaking changes and most important differences.

This tag allow us to branch off a stable 4.0.x branch where bugs might be back-ported and the 4.1.0 series which will not attempt to be completely backwards compatible with 4.0 (although compatibility at the level of runcards will be attempted).

Library changes

• Only pineappl >= 1 allowed
• Only eko >= 0.15 allowed (and ekos constructed with eko >= 0.14)
• parallel_models=true is now the default when running more than one replica at once
• Diagonal covmat #2301 now fully implemented and will eventually be made default
• ekos are now downloaded independently by vp-get eko <theoryid> to its own folder

Breaking changes

• All dataset names must follow the new naming conventions, old naming is no longer allowed.
• Older theories have been removed, except for those used for PDF releases.
• separate_multiplicative defaults to False
• resample_pseudodata defaults to False (negative pseudodata no longer resampled by default)
• Pseudodata is now generated using the t0 PDF by default

What's Changed

• Update docs, citations by @scarlehoff in #2296
• Positivity fktables at Q2 = 10 by @comane in #2285
• Merge all pytest workflows by @scarlehoff in #2298
• Add faketheoryid by @RoyStegeman in #2297
• Prepare a coverage report in the CI by @scarlehoff in #2299
• some things used in alphas studies by @RoyStegeman in #2302
• Define theory settings for 4.1 by @felixhekhorn in #2295
• Remove all old theories by @scarlehoff in #2304
• C-factors in the API extension by @achiefa in #2307
• Fix rapidity in SIHP-pp by @Radonirinaunimi in #2310
• Hotfix for macos 15.4 , rsync -> openrsync by @scarlehoff in #2311
• Miscellaneous improvements to docs and tests by @scarlehoff in #2305
• Modify how losses are computed in a multi-replicas hyperopt by @Radonirinaunimi in #2145
• Introduce frag scale to theory card by @t7phy in #2308
• True Positive and Negative rates in multiclosure weighted fits by @comane in #2258
• reactivate use_t0_sampling by @comane in #2300
• Modify the theory metadata for ATLAS_Z0J_8TEV_PT-M by @scarlehoff in #2316
• Use eko v0.15 by @scarlehoff in #2181
• Make parallel_models=true the default by @scarlehoff in #2313
• Data update 250521 by @scarlehoff in #2321
• Add an option to drop filter rules for a given dataset by @scarlehoff in #2309
• Modified label in theory covariance matrix by @enocera in #2324
• update scalevariationtheoryids for alphas fits by @RoyStegeman in #2327
• Remove accidental 's' from commondata by @RoyStegeman in #2329
• loop over replica_ folders rather than replica numbers starting from 0 by @RoyStegeman in #2328
• Update PineAPPL version to v1 by @Radonirinaunimi in #2217
• Diagonal covmat by @jacoterh in #2301
• Evolution of hessian sets and refactor of evolven3fit by @tgiani in #2325
• fix photon replica treatment if replica is equal to the number of replicas in the luxset by @RoyStegeman in #2333
• Use new theories and names for tests and examples by @scarlehoff in #2303
• don't resample L2 pseudodata by @RoyStegeman in #2275
• Separate the ekos into its own folder by @scarlehoff in #2331
• Remove a bunch of deprecated functions and options by @scarlehoff in #2334

Full Changelog: 4.0.10...4.1.0

This is the last release of the 4.0 family of fits, where many settings, seeds, dataset names etc are still compatible with 4.0 but a lot of the backend has been changed in preparation for 4.1.
The biggest change in that sense is the data and theory implementation.

While we have made every attempt to keep reproducibility intract, fits for the 4.0 "era" should be reproduced with previous tags such as 4.0.9

Below a selected list of changes with the relevant PRs when relevant.

Data and theory implementation #1709

• An nnpdf-data package is made available which contains only data and theories #2225
• All commondata implemented in NNPDF4.0 has been ported to the new format
• The new commondata format has been partially documented
• The new commondata reader has been merged to master and substituted the old one
• #2056
• The old commondata and the parsing of old commondata has been removed.
• The theory database is now in the form of .yaml files (#1997 #2028)
• #2099

Performance and logistics

• Refactoring to ease multireplica PDF, progress tracked in PR #1880
• Training/Validation masks are no longer set equal across multireplica GPU runs (#1788 )
• Refactored FK convolution for a slight speed up in CPU and a massive speed up in GPU (#1936)
• Rules are now immutable objects (#2076)

Build system and installation

• The usage of cmake is now optional and only used to install evolven3fit_legacy and buildmaster (#1773, #1690)
• evolven3fit has become the new standard evolution tool
• Code updated to be compatible with python3 / tensorflow >2.16 / keras 3 (#1975, #1970)
• Commondata and the theorydb is installed as part of the python package (#1861)
• ${XDG_CONFIG_HOME}/NNPDF/nnprofile.yaml is now the default location for the NNPDF profile (#1861)
• Added the flag redo-regression in github to autogenerate regression tests (#1944)
• Data can now be installed as a separate package (#2009)
• pre-commit has been added to the repository (#2026)
• Data can be now regenerated automatically in the CI (#2099)

New features

• Cuts can be now applied to positivity and integrability datasets (#2059, #2116)
• It is now possible to accept negative pseudodata in the replica generation (#2088)
• The fitting now is compatible with pytorch as well #2137

Breaking changes

• Replicas are now generated using directly the covariance matrix instead of the error breakdown #2107
• Paramfits has been removed #2190

evolven3fit

• The theory and operator cards are added to the log file (#2063)
• FastKernel Tables and EKO are separated in the server and donwloaded separately (#2081, #2087)

Bugfixes

• #2036
• #2034
• #2045
• #2048
• #2051
• #2066
• #2070
• #2119
• #2115
• #2114
• #2127

Automatic release notes

What's Changed

• [bot] Add conda environment for tag 4.0.9 by @github-actions in #1966
• Unpin pandas version + update tag description by @RoyStegeman in #1960
• Use eko.runner.managed.solve instead of eko.runner.solve by @niclaurenti in #1971
• Add theory 824 (N3LO+QED) by @niclaurenti in #1972
• Add a library of process dependent options by @scarlehoff in #1949
• Docs for new commondata format by @scarlehoff in #1708
• New CommonData Reader by @scarlehoff in #1678
• atlas 13tev ttb hadr implementation by @t7phy in #1973
• Fk refactor by @APJansen in #1936
• Fix multidense compatibility with tf 2.16 and keras 3 by @APJansen in #1975
• Remove cmake, make the conda recipe architecture independent by @scarlehoff in #1980
• Rename evolven3fit_new to evolven3fit + remove varflavors by @RoyStegeman in #1981
• Test only 3.9 and 3.11 ; build only in 3.11 by @scarlehoff in #1985
• add cuts for ttb and dis+j by @t7phy in #1983
• don't install SDK for mac in CI tests by @RoyStegeman in #1987
• Hyperopt loss by @APJansen in #1726
• Parallel hyperoptimization with MongoDB by @Cmurilochem in #1921
• change y to ystar for atlas 13 teV by @scarlehoff in #1992
• Make weight initialization reproducible by @APJansen in #1923
• hyperopt runcard by @APJansen in #1986
• fix cross correlation type to make them unique for each uncertainty by @t7phy in #1993
• Add a test for all commondata files by @scarlehoff in #1994
• Enable python 3.12 by @scarlehoff in #1989
• Updates to the NNPDF docs by @mariaubiali in #1998
• Enable apple silicon by @scarlehoff in #1999
• change 1 jet to jets by @t7phy in #2001
• Add some docs for process options by @scarlehoff in #2000
• Make the sqlite theorydb into a folder of yaml files by @scarlehoff in #1997
• Fix a bunch of dataset names by @scarlehoff in #2005
• don't compute predictions for all replicas if we only want replica0 by @RoyStegeman in #2004
• W-> l v pseudo rapidity (200804174) by @comane in #1927
• W -> l v, pseudo rap, bug fix by @comane in #2006
• Allow nf5 in produce eko2 by @niclaurenti in #2008
• NNPDF data package by @comane in #2009
• Fix positivity names from 5 GeV to sqrt(5) GeV and DY -> Z0 by @scarlehoff in #2011
• Remove apfelcomb flags by @scarlehoff in #2013
• Separate the theorydb utilities into the data package by @scarlehoff in #2018
• Fix kinematic vs kinematic_coverage check by @Radonirinaunimi in #2023
• Fix for PDF without error type by @scarlehoff in #2022
• add pch NLO and NNLO theories by @scarlehoff in #2012
• Polarised DIS and DY Commondata by @Radonirinaunimi in #1816
• add y labels, integrate utils in filters and remove comma in sys names v2 by @t7phy in #2021
• add .pre-commit-config.yaml by @RoyStegeman in #2026
• Remove buildmaster by @RoyStegeman in #2032
• Add unique log files for each mongo worker by @Cmurilochem in #2016
• fix std_error function in MCStats class by @RoyStegeman in #2036
• Fix to weight loading/saving for compatibility with Keras 3 by @scarlehoff in #2042
• Update to new commondata names - n3fit by @RoyStegeman in #2031
• Fixing bug in the MUL / ADD treatment when writing down closure data by @andreab1997 in #2045
• make pseudodata_table correctly deal with multiple replicas by @RoyStegeman in #2034
• Polish EIC data files by @comane in #2039
• Correct bug in commondata loading for closure tests by @andreab1997 in #2046
• Restore cut of CMS_TTBAR_8TEV_2L_DIF_MTTBAR-YT-NORM by @andreab1997 in #2050
• Use results to create th uncertainties with pdf errors. by @scarlehoff in #2051
• Regenerate test pol by @scarlehoff in #2053
• Use groups or reorder tests by @scarlehoff in #2054
• Improve sumrules module a bit by splitting operations by @Radonirinaunimi in #2057
• Update QED parts in theories 391 392 by @niclaurenti in #2038
• add nnpdf4.1 baseline theorycard by @RoyStegeman in #2048
• Polarised proton fits by @Radonirinaunimi in #1979
• capitalize EicC dataset names by @Radonirinaunimi in #2055
• add alphas variations NNLO and NLO by @RoyStegeman in #2029
• Add (and use) a default for IterEv by @scarlehoff in #2062
• add validphys.commondata to vp-setupfits providers list by @RoyStegeman in #2066
• Provide an unique truth for all theory cards by @andreab1997 in #2028
• Add normalization (and comment) to hash of theory by @scarlehoff in #2061
• Add an example using PineAPPL to compute predictions by @scarlehoff in #2049
• Add the square_singlet activation function by @scarlehoff in #2064
• check also for closure tes...

Release for 4.0 N3LO hep-ph/2402.18635
Last release fully backwards-compatible with 4.0 pipeline. 4.0 runcards will still work but: external tools, data and theory not used in the 4.0 family of fits will no longer be guaranteed to work going forward.
Last release compatible with the old commondata format

This release is compatible with python 3.9, 3.10, 3.11

To install from conda you can use:

conda install nnpdf=4.0.9.0 -c conda-forge -c https://packages.nnpdf.science/public

or pip:

 python -m pip install git+https://github.com/NNPDF/[email protected]

Highlights

• The usage of cmake is now optional and only used to install the legacy version of evolven3fit and buildmaster (#1773, #1690)
• Commondata and the theorydb is installed as part of the python package (#1861)
• ${XDG_CONFIG_HOME}/NNPDF/nnprofile.yaml is now the default location for the NNPDF profile (#1861)
• Added the flag redo-regression in github to autogenerate regression tests (#1944)
• Misc changes and improvements to th variation plots (#1910)
• Fits without charm sumrule can be now performed with the sum_rules: "ALLBUTCSR" option (#1926)

Full set of changes and PRs

• point to the right db by @scarlehoff in #1919
• Include the theory covariance matrix in the chi2 by dataset by @andreab1997 in #1918
• Be less strict and more granular when defining inconsistent version by @scarlehoff in #1920
• Merge prefactors into single layer by @APJansen in #1881
• Add more regression tests by @scarlehoff in #1924
• Add an lhapdf_compatibility module for LHAPDF by @scarlehoff in #1799
• Change from defaults to forge, update workflows by @scarlehoff in #1932
• Add flag double_precision by @APJansen in #1929
• Extend th covmat plots by @giacomomagni in #1910
• add nFONLL alphas varariations to theorydb by @RoyStegeman in #1937
• Automatic port old commondata to the new format by @scarlehoff in #1931
• New Common Datasets by @scarlehoff in #1813
• Automate regression updates by @APJansen in #1944
• Multi dense layer by @APJansen in #1905
• Add Q0 varied theories by @niclaurenti in #1948
• Ttb dataset fixes by @t7phy in #1947
• Added lru caches to speed up validphys multi-replica initalization by @goord in #1945
• Remove bugged raise Error when using multiple covmats by @RoyStegeman in #1951
• Drop the sqr and label the y by @t7phy in #1952
• Parallel replicas with varying tr-vl masks by @goord in #1788
• drop figure_by when not required by @t7phy in #1953
• drop sys breakdown in favor of covmat by @t7phy in #1957
• Complete baseline theories for polarised fits by @Radonirinaunimi in #1954
• FHMRUVV N3LO fits by @giacomomagni in #1911
• Remove charm sumrule by @tgiani in #1926
• Add info about 4.0.9 tag + update fitbot by @scarlehoff in #1959
• Add a stable reference report by @scarlehoff in #1963

Full Changelog: 4.0.8...4.0.9

Minor changes and bugfixing with respect to 4.0.7 in preparation for the final releases of the QED and MHOU PDFs.

• QED PDFs published as: "Photons in the proton: implications for the LHC" 2401.08749

• MHOU PDFs published as: "Determination of the theory uncertainties from missing higher orders on NNLO parton distributions with percent accuracy" 2401.10319

This tag can be used to reproduce the PDF sets released alongside the two papers above and which can be found in the NNPDF website:

• MHOU
• QED

What's Changed

• Theories added (799, 800, 803)
• [bot] Add conda environment for tag 4.0.7 by @github-actions in #1892
• also upload conda package if push corresponds to a tag creation by @RoyStegeman in #1897
• Fix dataset/prediction ordering problem in theory covmat creation by @RoyStegeman in #1899
• add description of 4.0.7 release to docs by @RoyStegeman in #1900
• Restore needed thcovmat functions by @andreab1997 in #1901

Bugfixes

• Fix alphaem running by @niclaurenti in #1906
• make vp-nextfitruncard use Q0 instead of Qmin for iterating preprocessing by @niclaurenti in #1898

For the whole list of changes, see the tag for 4.0.7

Full Changelog: 4.0.7...4.0.8

A pre-release for testing purposes that contains sets up all the functionality needed for the QED, MHOU and N3LO variants of NNPDF4.0.

What's Changed

• Enable conda package for 3.11 (#1890) @scarlehoff
• Add theory 390 for Leading Order (#1891) @scarlehoff
• Set to 0 not active flavours (#1884) @scarlehoff
• update fitbot reference (#1883) @RoyStegeman
• Add --legacy40 flag for evolven3fit_new to use the grid for nf=3 instead of nf=4 (#1879) @scarlehoff
• Add N3LO as varied theories (#1878) @giacomomagni
• Add two extra colors (#1876) @scarlehoff
• Allow photon PDF to reports (#1875) @niclaurenti
• Set deadline=None for test_kitable_with_cuts (#1873) @scarlehoff
• Set reportengine version ==0.30.28 (#1872) @scarlehoff
• Fix typo in validphys (#1870) @niclaurenti
• Add qcd exa to theory.db as theory 798 (#1857) @niclaurenti
• Raise a ValueError when the central value is outside of the percentiles (#1868) @scarlehoff
• calculate only predicitons for rep0 in thcovmat construction (#1863) @RoyStegeman
• Add a new plot_fancy_dataspecs and chi2 distributions with scale variations (#1830) @scarlehoff
• [bot] Add conda environment for tag 4.0.7 (#1851) @github-actions
• run black isort on recently edited files (#1848) @RoyStegeman
• N3LO fits with IHOU (#1698) @giacomomagni
• Raise an exception when use_pdferr is used together with plot_fancy (#1844) @scarlehoff
• some cosmetic fixes of things that bothered me (#1845) @RoyStegeman
• update pineappl dependency to 0.6.2 (#1842) @RoyStegeman
• remove qed input from ekobox.apply.apply_pdf (#1841) @RoyStegeman
• Generic hessian PDF covmat (#1831) @comane
• update postivity commondata (#1839) @RoyStegeman
• Remove NNLO specific cuts (#1835) @andreab1997
• Restart hyperopt (#1824) @Cmurilochem
• nFONLL theory indices up to and including 721 (#1810) @andreab1997
• Remove unused functions in photon module (#1833) @niclaurenti
• Update PLOTTING_ATLAS_Z_TOT_13TEV.yaml (#1827) @enocera
• add scale variations IDs for theory 600 (#1823) @t7phy
• Exa couplings2 (#1806) @niclaurenti
• Miscelaneous bugfixes (matplotlib & futuretests) (#1809) @scarlehoff
• Allow added cuts (#1745) @Zaharid
• Documentation for reportengine (#1804) @comane
• Test with less tests (#1805) @scarlehoff
• Add --lite option to vp-comparefits for quick comparisons (#1801) @scarlehoff
• Raise an exception if after many tries the replicas are still negative (#1790) @scarlehoff
• Use evolven3fit_new with the fitbot (#1789) @scarlehoff
• Fix regression tests for arclengths (#1791) @scarlehoff
• Add produce_eko_photon to evolven3fit_new (#1778) @niclaurenti
• Intermediate merge of Aron's stuff (#1775) @goord
• Refactoring model creation code (#1734) @APJansen
• Change fiatlux_dis_F{2,L} -> FIATLUX_DIS_F{2,L} (#1774) @niclaurenti
• Alpha variation but same thcovmat (#1768) @andreab1997
• Change also luxset with vp-nextfitruncard (#1767) @niclaurenti
• Add theory 527 and extend comments in theories 522, 523, 524, 525, 526 (#1766) @niclaurenti
• Separate DY_CC and DY_NC from DY in nnpdf31_process (#1760) @andreab1997
• Avoid using resources at the module level (#1763) @niclaurenti
• Add luminosity channels (#1762) @Zaharid
• QED tests (#1738) @niclaurenti
• Allow categorical variables in smpdf plots (#1715) @Zaharid
• Enable nf=3 with evolven3fit_new. (#1754) @scarlehoff
• Getting latest master changes (#1759) @goord
• Allow different matching scale for the photon (#1751) @niclaurenti
• Hessian PDF Covariance Matrix for theory predictions (#1743) @comane
• Tests for evolven3fit_new and lhapdf .info files. (#1746) @scarlehoff
• Fixes to prepare the code for pandas 2.0 (#1747) @scarlehoff
• Evolven3fit new w eko 0.13 (#1742) @giacomomagni
• Re-style codebase: validphys (#1737) @scarlehoff
• add .git-blame-ignore-revs (#1736) @RoyStegeman
• Add photon (#1643) @niclaurenti
• Make pineparser handle optimized fktables (#1716) @andreab1997
• Closure test L1 consistency in random noise generation (#1695) @comane
• nnpdf without pyplot imports (#1723) @comane
• Fixing logs of evolven3fit_new (#1731) @andreab1997
• Change the fitbot solver to libmamba (#1725) @scarlehoff
• futureproofing: make dropna keyword based (#1713) @RoyStegeman
• Positivity constraints for c and cbar (#1697) @enocera
• Remove replica selector functionality (#1703) @Zaharid
• Update recommended pylintrc (#1701) @Zaharid
• Added EXA theory (250) (#1689) @enocera
• k-factors for new theories (#1677) @giacomomagni
• Fix label of absolute uncertainty plots (#1692) @Zaharid
• Remove libNNPDF from python (#1680) @scarlehoff
• vp-comparefit issue, due to arclength errors (#1681) @comane
• Keep name of target dataset (avoid reutilizing fktables) (#1688) @scarlehoff
• Move xgrid back to n3fit (#1686) @scarlehoff
• Update to the plot (#1685) @scarlehoff
• Python closure sampling (with the only python commondata branch) (#1660) @scarlehoff
• Substitution of apfel with eko in evolven3fit (#1537) @andreab1997
• Use ndata instead of len (#1675) @scarlehoff
• Tensorflow in macos (#1673) @scarlehoff
• only build docs for 3.9 (#1671) @scarlehoff
• Build for Python 3.10 (#1628) @Zaharid
• Save pseudodata by default (#1669) @RoyStegeman
• Fix penalties and add tests (#1668) @scarlehoff
• Use python CommonData (#1650) @scarlehoff

New Features

• Multi dense logistics (#1818) @APJansen
• Order dataset differences by name of the commondata (#1867) @scarlehoff
• FTDY maxTau cuts in runcard (#1733) @andreab1997
• Add a pyproject.toml (#1710) @scarlehoff
• Mix bands and replicas (#1607) @scarlehoff
• read NN output size for hyperscan from runcard (#1696) @RoyStegeman
• Evolve n3fit with eko 0.12 (#1694) @giacomomagni

Bug fixes

• add explicit cut for FONLL-E (#1874) @giacomomagni
• bugfix with stopping (#1820) @APJansen
• correct sorting and indexing of level1 data (#1765) @comane
• don't print msr layer summary if msr not enforced (#1795) @RoyStegeman
• Fix custom_group processing (#1753) @Zaharid
• Fix Q2grid in evolven3fit_new (#1750) @giacomomagni
• Fix calling of fiatlux (#1749) @niclaurenti
• multiclosure internal dataset loader should use t0_covmat_from_systematics (#1719) @comane
• reintroduce AlphaS_* attributes in the PDF class and add a test of alpha_s_bundle_pdf (#1714) @RoyStegeman

Documentation

• remove mention that vp-setupfit doesn't always have to be the first step (#1771) @RoyStegeman
• Update styling docs (#1740) @RoyStegeman
• add use_scalevar_uncertainties to loading custom comvat docs (#1712) @RoyStegeman
• add version 4.0.6 to releases list in docs (#1670) @RoyStegeman

Workflow

• use mamba in docker construction (#1850) @RoyStegeman
• add skip_magic_trailing_comma to black config (#1828) @RoyStegeman
• Mamba ci (#1793) @APJansen

Refactoring

• Move replica axis to front everywhere (#1877) @APJansen
• Multi dense logistics (#1818) @APJansen
• Refactor stopping (#1792) @APJansen
• Refactor msr (#1781) @APJansen
• Refactor rotations (#1780) @APJansen
• Refactor preprocessing (#1777) @APJansen
• Refactor xintegrator (#1779) @APJansen
• Implement FkRotation as subclass of Rotation by rewriting using a rotation tensor (#1772) @APJansen

This is the last version that uses C++ objects in n3fit or validphys. It is also anticipated to be the last release that utilizes apfel for PDF evolution.

At the same time, it is the first version that includes a pineappl parser. pineappl is expected to become the main provider of theory predictions.

What's Changed

• add ccbar assym basis (#1658) @RoyStegeman
• Change numpy to python for int, float and bool. (#1653) @scarlehoff
• Update docs in vpinterface (#1640) @scarlehoff
• Train on 1-point datasets (#1636) @scarlehoff
• ccbar asymmetry (#1629) @tgiani
• Fix to #1637 (#1638) @scarlehoff
• Simplify MetaModel input handling (#1625) @Zaharid
• Adding filter rule to remove one linearly dependent bin from the CMS 2D top distribution (#1633) @enocera
• Remove additional t0 flags (#1626) @Zaharid
• Add flag for the derivative of thePDF (#1605) @scarlehoff
• Kinetic energy of the pdf (#1606) @scarlehoff
• Fix legend of the replica distribution plot (and experiments_chi2_stats) (#1612) @scarlehoff
• Implemented function to compute asymptotic exponents (#1581) @enocera
• Add t0 chi2 to comparefitsreport (#1613) @RoyStegeman
• Fix overfitmetric in comparefits (#1611) @scarlehoff
• Prevent offset in plot axis values (#1608) @RoyStegeman
• Update fitbot (#1604) @scarlehoff
• Top data fixes (#1602) @enocera
• update overfit metric fit (#1603) @scarlehoff
• update code citation with EPJC info (#1592) @RoyStegeman
• Evaluate only unique xgrids (#1587) @scarlehoff
• Raise an exception for wrongly loaded positivity or integrability datasets (#1591) @scarlehoff
• Add NLO and NNLO sv theories in db (#1589) @andreab1997
• Revert format (#1590) @Zaharid
• Enable positivity/integrability pineappl grids (#1585) @scarlehoff
• Validphys with pineappl rebased to master (#1578) @scarlehoff
• add plotting action for lumi1d per replica (#1580) @RoyStegeman
• Fit with theory covmat with n3fit (#1528) @andreab1997
• Decorrelation models (#1571) @enocera
• Bugfix for kfolds (#1572) @scarlehoff
• Use <fitname>.json by default and deprecate the .fitinfo file. (#1546) @scarlehoff
• [bot] Add conda environment for tag 4.0.5 (#1567) @github-actions

New Features

• Add overfit metric to vp (#1573) @RoyStegeman

Bug fixes

• Fixed normalised data-theory comparison errors (#1665) @J-M-Moore
• replace depricated np.asscalar with .item() (#1659) @RoyStegeman
• fixed bug following from introduction of parse_fakepdf in config.py (#1654) @comane
• fix default elements for ccbar_asymm basis (#1649) @tgiani
• bufix: use fakepdf to generate pseudodata instead of t0pdfset (#1647) @RoyStegeman
• Fixing the grouping and the thcovmat flag (#1588) @andreab1997
• Fix bugs in closuretest (#1570) @VLTSML26

Documentation

• Personnel update (#1593) @enocera
• Simplify docs of convolution module (#1569) @Zaharid
• Add note on 4.0.5 release (#1568) @Zaharid

Refactoring

• Separate xgrid and model generation within model trainer (#1644) @scarlehoff

This is the last release that supports the older C++ based nnfit code on which the NNPDF 3.1 sets were based. Support for nnfit will be disabled in future releases.

In addition many minor features and bugfixes were added to the code. Many features were ported from C++ to Python. The detailed changelog is below.

New Features

• Additional closure test histograms (#1321) @wilsonmr
• Add metadata groups to closure tables (#1324) @wilsonmr
• Delta chi2 table by group (#1329) @wilsonmr
• Return all members when using MC PDFs (#1522) @scarlehoff
• adjust hyper_scan step sizes (#1534) @RoyStegeman
• use error_members instead of rawdata (#1517) @scarlehoff
• Utilize stats everywhere where it can be used (#1515) @scarlehoff
• Use python convolutions also for positivity predictions (#1510) @scarlehoff
• Add tests for PDF-obs and obs-obs correlations (#1509) @scarlehoff
• Swap libNNPDF LHAPDFSet for a python PDF object (#1501) @scarlehoff
• Add a fallback to using all points for reports that are using old fits with bad filters. (#1506) @scarlehoff
• Add tests for sum rules (#1507) @scarlehoff
• Handle corrupt figures in report indexing (#1483) @Zaharid
• Access the PDF info explicitly (#1499) @scarlehoff
• Use python covmat in multiclosure tests (#1489) @scarlehoff
• Alphas theories: alphas=0.114,0.115,0.121,0.122 and PDF4LHC21 alphas=0.117,0.119 (#1492) @enocera
• Improve postfit error message (#1485) @Zaharid
• Change ThPredictions to python predictions (#1430) @scarlehoff
• CMS high-mass Drell-Yan at 13 TeV (#1192) @voisey
• Dispatching Python covariance matrices (#1477) @siranipour
• Add a single data point dataset to the tests (#1473) @scarlehoff
• Add a regresion thprediction test for a hessian PDF (#1469) @scarlehoff
• Disable spinner in non interactive mode (#1467) @Zaharid
• mc variations (#1464) @enocera
• Swap the libnnpdf profile for a python dictionary holding all relevant information (#1428) @scarlehoff

Bug fixes

• Fixed chi2 with thcovmat in report (#1560) @andreab1997
• Improve remote keyword functionality (#1561) @Zaharid
• Fix info file of mc2hessian (#1551) @scarlehoff
• updating server path (#1539) @scarrazza
• Fixed bugs of previous theorycovariance runcards and added new working runcards (#1524) @andreab1997
• Bugfix hyperopt integrability penalty (#1533) @RoyStegeman
• Fix callers of group_chi2_table (#1525) @Zaharid
• Make saved pseudodata match fitted pseudodata (#1514) @scarlehoff
• do what genrep says in makereplica (#1494) @scarlehoff
• Fix example runcard (#1488) @scarlehoff
• In process_function take nnpdf31_process from PLOTTING_*.yaml (#1472) @RoyStegeman
• Fix pseudodata indexing mismatch (#1466) @siranipour
• Fix undefined behaviour in buildmater number parsing (#1465) @enocera
• Changed reserve -> resize in FutureColliders.cc in buildmaster (#1461) @J-M-Moore
• rename meta/integ files (#1456) @RoyStegeman
• Fixing buildmaster compilation (#1453) @enocera

Documentation

• Add note on t0 usage (#1563) @Zaharid
• Improve documentation on regularized covmats (#1562) @Zaharid
• Documentation madgraph+amcfast (#1518) @enocera
• Add an overview of loss functions to the documentation (#1495) @RoyStegeman
• Add documentation on adding citations to Sphinx (#1497) @Zaharid
• [bot] Add conda environment for tag 4.0.4 (#1450) @github-actions
• mention rolling release in README (#1454) @RoyStegeman
• Add short description of 4.0.4 release (#1451) @Zaharid

Worflow

• libnnpdf/wrapper: don't try to install to system PYTHONPATH (#1566) @veprbl
• Enable python 3.9 and mamba build (#1463) @scarlehoff

Refactoring

• Access the PDF info explicitly (#1499) @scarlehoff
• Changing functions which are still using NNPDF pseudodata (#1424) @scarlehoff

This is an incremental bugfix release.

The main highlights are:

• Fix a bug in integrated luminosity plots #1442.
• Code for ancillary NNPDF 4.0 analysis (such as Hessian conversion #1344, Nf=4 re-evolution #1422, or alpha_s errors bundling #1340).
• User facing documentation, including for reproducibility of 4.0 the analysis (#1378)

Additional changes include:

• A bug in the evolution basis definition #1445, of no consequence to existing analyses was fixed (thanks to @msyan!).
• Enhancements to hyperoprimization allowing it to work on estimators depending on a full fit (#1316).
• Some work on moving features implemented in C++ to python (e.g. #1268).
• Tweaks and enhancements to various plotting functions, as well as new spider diagrams (#1194).
• Automation using Github Actions, to for example produce release Doceker images (#1390).

The full list of changes is included below.

What's Changed

New Features

• Improve varflavors script (#1422) @Zaharid
• Compute 1D luminosities with a rapidity cut (#1439) @enocera
• Add fit-based targets to hyperparameter optimization (#1316) @scarlehoff
• Speed up convolution (#1435) @scarlehoff
• Use internal cuts in tests with predictions (#1429) @scarlehoff
• Add integrability datasets for valence pdfs (#1412) @RoyStegeman
• adding --theory_id flag to evolven3fit (#1346) @scarrazza
• Add modified_tanh activation function (#1411) @RoyStegeman
• Add toggle to remove 68% legend labels (#1406) @Zaharid
• Improving reading and recreating fit pseudodata (#1328) @siranipour
• Adding spider plot for dataspecs groups (#1398) @siranipour
• Bump reportengine version (#1399) @Zaharid
• Saving new python pseudodata during fit (#1333) @siranipour
• Added separate theory for iterated K-factors (#1396) @enocera
• Hyperoptimization: take trials from a previous run (#1314) @scarlehoff
• Added scalar functions to plot d/u and dbar/ubar ratios (#1365) @enocera
• Adding action to bundle PDFs (#1340) @siranipour
• MC2Hessian in vp (#1348) @RoyStegeman
• General theory covmat (#991) @RosalynLP
• Create a hyperscan resource (#1313) @scarlehoff
• remove formatting of pdf scale in title - use default python format (#1327) @wilsonmr
• Making n3fit use python covmats (#1285) @siranipour
• impose_sumrules default setting (#1322) @RoyStegeman
• Modify vp-comparefits interface (#1312) @scarlehoff
• update matplotlib hatch style (#1317) @RoyStegeman
• At least 1d central values for t0 predictions (#1305) @siranipour
• Cleaner CommonData repr (#1310) @siranipour
• Plot higher mx data by default (#1308) @Zaharid
• Making n3fit use python pseudodata (#1268) @siranipour
• Allow the n3fit-vp interface to make use of more than 1 members (#1251) @scarlehoff
• Generate replicas when fitting models in parallel (#1261) @scarlehoff
• Create spider/radar diagram for comparing chi2s between fits (#1194) @RosalynLP

Bug fixes

• Fix bug in the definition of the 1D luminosity. (#1443) @enocera
• fixing T24 and V24 definition in vp (#1445) @tgiani
• Fix for smpdf (#1437) @scarlehoff
• Fixing training mask tables (#1423) @siranipour
• Collecting over cuts context for pseudodata (#1418) @siranipour
• Fix to check_cuts_considered (#1419) @scarlehoff
• Collect over groups to arrange groups_chi2 table (#1352) @siranipour
• Fix some problems in spider plots (#1389) @Zaharid
• Fix commodata loader bug (#1415) @RoyStegeman
• remove space at beggining of line in pdfas header (#1404) @RoyStegeman
• Fixed typo in PDF4LHC21 basis (#1393) @enocera
• fix default value in postfit help (#1391) @RoyStegeman
• Fix hyphotesis timeout (#1383) @scarlehoff
• Correcting kinematics override for DY datasets (#1357) @siranipour
• change header replica0 (#1342) @RoyStegeman
• Fix regressions by specifying dtype (#1336) @siranipour
• Recover overfit option for partitions (#1330) @scarlehoff
• Fixing memory leaks in buildmaster (#1300) @enocera
• Fixing buildmaster leaks round 2 (#1318) @scarrazza
• Correcting syntax in PLOTTING_POSDYU.yaml (#1307) @RosalynLP
• Fix potential bug in read_fit_pseudodata (#1249) @RosalynLP
• Make fitting theory covmat with indices aligned with experiment covmat (#1232) @RosalynLP
• Fix vp-deltachi2 by changing experiments to datasets inputs (#1292) @scarlehoff

Documentation

• Update resources for tests, examples and documentation to be NNPDF4.0 based (#1440) @scarlehoff
• Add warning on compatibility of Hessian sets (#1441) @Zaharid
• Fix make_replica example (#1425) @siranipour
• Fixing pseudodata tutorial (#1427) @siranipour
• Update git tutorial (#1431) @scarlehoff
• Add docs on how to do a future test (#1408) @scarlehoff
• Restructuring data kws docs (#1382) @siranipour
• update runcard names to agree with paper (#1402) @RoyStegeman
• Add compression tools to docs (#1395) @scarlehoff
• Temporarily remove nf=4 comment (#1401) @Zaharid
• Add sysroot to installation instruction (#1400) @Zaharid
• Refine wording on coding rules (#1387) @Zaharid
• Remove nnfit references (#1394) @scarlehoff
• Add documentation on compatibility policy (#1385) @Zaharid
• Write a publicly facing README (#1374) @Zaharid
• how to reproduce nnpdf40 fit - docs (#1378) @RoyStegeman
• Rephrasing docs frontpage /fixing typos (#1343) @RosalynLP
• Add dataset specification to runcard guide (#1379) @scarlehoff
• Create LICENSE (#1375) @scarrazza
• Docs code structure (#1368) @RoyStegeman
• restructured documentation (#1303) @mariaubiali
• Update runcards with latest choice of baseline data set (#1320) @enocera
• Add NNPDF4.0 conda environment (#1255) @Zaharid
• Remove authors from docs (#1304) @voisey
• Correct typos (#1301) @RosalynLP
• Removed reference to no longer existing data_layout file in doc (#1297) @enocera
• update LO runcards (#1296) @RoyStegeman

Workflow

• adding minimum setup for release drafter (#1447) @scarrazza
• updating tests for github actions (#1397) @scarrazza
• GitHub Packages for nnpdf (#1390) @scarrazza
• disable self-hosted runner for fit-bot (#1369) @scarrazza
• Add testing github action (#1363) @scarlehoff

Refactoring

• Revert correlated shift functionality (#1388) @Zaharid
• Delete validphys 1 (#1361) @Zaharid
• Remove unused class (#1350) @Zaharid
• Removing unused imports (#1338) @siranipour
• Delete old executables (#1283) @Zaharid

Full Changelog: 4.0.3...4.0.4

This version is used for producing all the publicly released fits for NNPDF4.0.