This PR makes several changes for Redrock I/O:

• Update redrock.results.read_zscan for reading rrdetails files
  • ~7x faster by refactoring how the zfit tables are parsed
  • add option nozfit=True to return just the chi2 vs. z (zscan) results, not including the zfit tables
• Add redrock.results.read_zfit to read+stack the Nth best fit zfit tables
• Both read_zscan and read_zfit
  • add option upper=True to force columns to UPPERCASE like most DESI files
  • update select_targetids to accept a single integer in addition to list/arrays of integers
• Fix bug where zbest table in rrdetails output was accidentally written then discarded
• Change rrdetails output precision to single precision float and smaller integers to save disk space
  • zerr, chi2, zz, zzchi2, coeff, deltachi2: float64 -> float32
  • zwarn, npixels: int64 -> int32
  • znum, ncoeff: int64 -> int8

The output precision change was motivated by deishub/desispec#2456 suggesting to use smaller redshift steps in zscan, in which case the rrdetails files were getting rather large. With this update, smaller steps in the galaxy scan results in files with the same size as before.

I did not change the precision of the outputs in the redrock*.fits table. I think we should do that too, but since those are used more and propagate into the final redshift tables for a production, I wanted to get a second opinion before making that update.

Example outputs in /pscratch/sd/s/sjbailey/desi/redrock/io, with originals in the loa subdirectory:

<perlmutter io> ls -lh *.* loa/*.*
-r--r----- 1 sjbailey desi 214M Mar 28 16:27 loa/coadd-2-1263-thru20240212.fits
-r--r----- 1 sjbailey desi 493K Mar 28 16:27 loa/redrock-2-1263-thru20240212.fits
-r--r----- 1 sjbailey desi 132M Mar 28 16:28 loa/rrdetails-2-1263-thru20240212.h5
-rw-r--r-- 1 sjbailey desi 493K Mar 28 16:30 redrock-2-1263-thru20240212-float32.fits
-rw-r--r-- 1 sjbailey desi  67M Mar 28 16:30 rrdetails-2-1263-thru20240212-float32.h5

fitsdiff confirms that redrock-2-1263-thru20240212-float32.fits and loa/redrock-2-1263-thru20240212.fits are still identical except for headers (though I suggest we downgrade the precision of some of the redrock columns).

rrdetails-2-1263-thru20240212-float32.h5 is 2x smaller than the original loa file.

a gotcha for consideration: currently redrock uses zchi2=9e+99 for the chi2 of a complete failed fit, which gets converted into inf for float32. That could be a pain, and we should consider using a smaller placeholder value for float32 chi2 of fit failures (e.g. 3e33).

import h5py

h1 = h5py.File('loa/rrdetails-2-1263-thru20240212.h5')
h2 = h5py.File('rrdetails-2-1263-thru20240212-float32.h5')

zchi2_orig = h1['zscan']['GALAXY']['zchi2'][:]
zchi2_new = h2['zscan']['GALAXY']['zchi2'][:]
print(zchi2_orig.dtype, zchi2_new.dtype)
print(np.all(zchi2_orig.astype(np.float32) == zchi2_new))
ii = np.isfinite(zchi2_new)
print(np.std(zchi2_new[ii] - zchi2_orig[ii]))

results

float64 float32
True
0.00036350274545413297