In this FITS file reading out the .data object of the 6th (0-indexed counting), the HDU named "Observation" takes over 3 minutes on my iMac 5K.
The table indicates a total size of 16 MB:

data.itemsize
16777661

but I don't believe that should be an issue, as indicated by our IDL reader, that has no noticeable delay in opening that file.
In case this is not an easy-to-fix performance bug with pyfits: Our group is creating these files, we can't change the structure of the files in terms of where is stored what, but maybe there are some flags that we can use to improve the structure internally on the binary level somehow?

Here's the code I used to show the issue:

from astropy.io import fits
import time
fname = 'mvn_iuv_l1a_periapse-orbit01063-muvdark_20150417T234058_v02_r01.fits.gz'
hdulist = fits.open(fname)
t0 = time.time()
data = hdulist[6].data
print(time.time() - t0)

193.38006591796875