Fix PmagPy_MagIC.ipynb under pandas 3.0 and current PmagPy/MagIC API - #866
Open
Swanson-Hysell wants to merge 10 commits into
Open
Swanson-Hysell wants to merge 10 commits into
Swanson-Hysell wants to merge 10 commits into
Conversation
Pandas no longer accepts axis as a positional argument to DataFrame.dropna(),
so spec_df.dropna('columns', ...) raised a TypeError. Updated to
dropna(axis='columns', ...) to match the current API.
The oersted parameter introduced in 1deb048 (2017) was never branched on, making the /10 Gauss-to-Tesla correction unconditional; the parameter itself was later removed in 533e8dc but the unconditional correction remained. Restore oersted=True as the default (matching the CIT format convention where AF step values are recorded in Gauss) and gate the conversion on the flag so callers with mT-formatted .sam files can pass oersted=False to get correct Tesla values. Replace the redundant -oe CLI flag with -mT.
…RE converters pandas 3.0's strict str dtype rejects .loc[mask, col] = <float> partial assignments into string-initialized columns. In iodp_jr6_lore, treat_temp and treat_dc_field are initialized as strings and then partially overwritten with float values for TD and ARM rows; in iodp_dscr_lore the same pattern fires on treat_dc_field for ARM rows. Cast these columns to object dtype immediately after their string init so the later partial assigns succeed.
save_plots had dir_path=None as default, then unconditionally did
os.path.join(dir_path, fname), which raises TypeError when no caller
forwards dir_path. Most callers in ipmag (aniso_magic, zeq_magic plot
saves, etc.) omit the argument, so they hit this whenever not running
on the MagIC server and no dpi kwarg is passed. Change the default to
"." so the docstring's promise ("directory in which to save plots") is
the actual fallback behavior.
Restore sites.txt and magic_contribution_16533.txt, which were deleted as collateral damage in commit 44293e8 (2021-06-20) when an in-progress extract_results.ipynb was added; the companion notebook was itself deleted 13 minutes later in 00f9151 but these data files were never put back. The PmagPy_MagIC.ipynb download_magic section has been quietly broken on that missing sites.txt ever since. sites.txt is restored from 9329ca3 (its last modification, 2020-09-04) and magic_contribution_16533.txt from 2ef7d21~1 (the last commit before its 2020-09-04 deletion).
…ples
- Cell 24 (convert_ages): cast frame to object before fillna('') so
pandas 3.0's strict str dtype accepts empty strings in numeric columns
- Cell 57 (private download example): replace stale credentials
(contribution 16960 / fa06255d-ec45-473f-9f33-fa2a9bb6d8d8) with a
currently-shared contribution (20672) since the original was no longer
accessible via the MagIC API
- Cell 60 (upload_magic): forward input_dir_path explicitly (the function
was split into separate input/output paths) and pass validate=False
- Cell 78 (markdown): note that the next cells are skipped in the rendered
documentation version of the notebook
- Cell 231 (cleanup): early-return in remove_magic_files when the directory
is missing, wrap remaining os.remove calls in try/except FileNotFoundError,
add 2g_asc_magic and 2g_bin_magic parents to the directory list, sweep
leftover magic_contribution_*.txt at the project root, and clean
data_files/download_magic/ while preserving the two example files
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates PmagPy to bring PmagPy_MagIC.ipynb and a handful of supporting conversion/plotting utilities back into a working state with pandas 3.0 and the current PmagPy/MagIC APIs, and restores missing example data used by the notebook.
Changes:
- Add an
-mTCLI flag /oerstedoption for CIT conversion so AF demag steps can be interpreted as either Gauss(Oe) or milliTesla. - Fix pandas 3.0 incompatibilities in
ipmag.specimens_extractand IODP LORE converters by avoiding deprecated positional args and relaxing string-typed columns before partial numeric assignment. - Make
pmagplotlib.save_plotsdefault to saving into the current working directory instead of erroring whendir_pathis omitted.
Reviewed changes
Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| programs/conversion_scripts/cit_magic.py | Adds -mT flag wiring to pass oersted=False into the CIT converter. |
| pmagpy/pmagplotlib.py | Changes save_plots default dir_path to "." and updates docstring accordingly. |
| pmagpy/ipmag.py | Updates dropna call to use keyword axis= for pandas 3.0 compatibility. |
| pmagpy/convert_2_magic.py | Implements oersted scaling for CIT AF steps; casts selected LORE measurement columns to object for pandas 3.0 partial assignment compatibility. |
| data_files/download_magic/sites.txt | Restores example sites.txt data needed by the notebook’s download_magic section. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Member
Author
|
This PR will close #835 |
… tests - Move oersted=True to the end of the cit() signature (and docstring) so positional callers written against the post-2020 signature aren't broken. - Add test_convert_cit.py to pmagpy/test (the CI test suite, pytest-style), asserting the expected Tesla values for the USGS bl9-1 sample under both Gauss (default) and mT interpretations. - Mirror equivalent assertions into pmagpy_tests/test_imports3.py (TestCitMagic), the older unittest-based suite, for completeness.
The new test_convert_cit.py covers basic conversion of bl9-1.sam to the five MagIC 3.0 tables (record counts, headers, sitename/locname propagation, specimen↔measurement consistency), the oersted flag's Gauss vs milliTesla interpretation of AF demag steps, and the actionable-error return tuple for invalid inputs. Mirrors and extends the unittest-style coverage in pmagpy_tests/test_imports3.py, which is the older suite that CI does not run.
Several pmagpy/test/test_ipmag_*.py files import pmagpy.ipmag, which conditionally pulls in cartopy via pmag.import_cartopy(). When cartopy isn't installed in the CI image, pytest still collects the module but downstream cartopy-using paths break test collection during import. Adding cartopy to the workflow's pip install line resolves the missing dependency so the suite collects cleanly.
pmagpy/convert_2_magic.py imports pytz at module top, so the new test_convert_cit.py fails collection on a CI image where pytz isn't already pulled in transitively. Adds pytz alongside cartopy in the workflow's pip install line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PmagPy_MagIC.ipynb had drifted out of working order — several cells failed under
pandas 3.0 and the current PmagPy/MagIC API, and the
download_magicsectionrelied on example data files that had been deleted in 2021. This PR walks the
notebook end-to-end and fixes each failure at the right level: notebook-side
where the cell call was stale, upstream where the failure exposed a bug in a
pmagpy function.
Changes by area
pmagpy fixes (each in its own commit)
convert.cit: restore theoerstedflag. The parameter introduced in 2017(
1deb048a) was never branched on, making the Gauss→Tesla/10correctionunconditional; it was later removed in
533e8dc5but the unconditionalcorrection remained. The flag is now wired up and defaults to
Truetomatch standard CIT-format convention (e.g. an AF step of 10 means 10 G = 1 mT).
A
-mTCLI flag is added for files that store AF steps in milliTesla.ipmag.specimens_extract: passaxisas a keyword argument toDataFrame.dropna()— pandas no longer accepts it positionally.pmagplotlib.save_plots: change the defaultdir_pathfromNoneto".".The function unconditionally does
os.path.join(dir_path, fname), so theNonedefault caused aTypeErrorwhenever callers (most ofipmag.py)omitted the argument.
convert.iodp_jr6_lore/convert.iodp_dscr_lore: casttreat_tempandtreat_dc_fieldtoobjectdtype after their string initialization.Pandas 3.0's strict
strdtype rejects.loc[mask, col] = <float>partialassignments into string-typed columns, which broke these two LORE converters
on ARM and TD rows.
Notebook updates (PmagPy_MagIC.ipynb)
convert_agescell: cast frame toobjectbeforefillna('')so pandas 3.0accepts empty strings in numeric columns.
(the original example credentials had been retired in MagIC).
upload_magiccell: forwardinput_dir_pathexplicitly (the function wassplit into separate input/output paths) and pass
validate=False.os.removecalls intry/except FileNotFoundError, add2g_asc_magicand2g_bin_magicparents to the directory list, sweep straymagic_contribution_*.txtat the project root, and cleandata_files/download_magic/while preserving the two example files.Example data
data_files/download_magic/sites.txtandmagic_contribution_16533.txt. Both were deleted as collateral damage incommit
44293e8c(2021-06-20) when an in-progressextract_results.ipynbwas added; the companion notebook was itself deleted 13 minutes later, but
these data files were never put back. The
download_magicsection hasbeen quietly broken on the missing
sites.txtsince.Test plan
interactive MagIC password prompt — all cells before that and after than complete without
errors
convert.citondata_files/convert_2_magic/cit_magic/USGS/bl9-1/bl9-1.sam:AF steps 100, 200, ..., 800 in the
.samfile producetreat_ac_field = 0.01, 0.02, ..., 0.08T inmeasurements.txt(Gauss interpretation)convert.cit(..., oersted=False)on the same file: produces 10× largervalues (mT interpretation)
convert.iodp_jr6_loreondata_files/iodp_magic/U999A: completeswithout
TypeErrorand produces sensibletreat_dc_field(5e-5 T) onARM rows and
treat_temp(273 + step) on TD rowsipmag.aniso_magic_nb(infile='specimens.txt', dir_path='data_files/convert_2_magic/k15_magic/'):saves
_aniso-data.pngand_aniso-conf.pnginto CWD withoutTypeError