Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix PmagPy_MagIC.ipynb under pandas 3.0 and current PmagPy/MagIC API - #866

Open
Swanson-Hysell wants to merge 10 commits into
masterfrom
pmagpy_magic_notebook_testing
Open

Swanson-Hysell wants to merge 10 commits into
masterfrom
pmagpy_magic_notebook_testing

Conversation

@Swanson-Hysell

Copy link
Copy Markdown
Member

PmagPy_MagIC.ipynb had drifted out of working order — several cells failed under
pandas 3.0 and the current PmagPy/MagIC API, and the download_magic section
relied on example data files that had been deleted in 2021. This PR walks the
notebook end-to-end and fixes each failure at the right level: notebook-side
where the cell call was stale, upstream where the failure exposed a bug in a
pmagpy function.

Changes by area

pmagpy fixes (each in its own commit)

  • convert.cit: restore the oersted flag. The parameter introduced in 2017
    (1deb048a) was never branched on, making the Gauss→Tesla /10 correction
    unconditional; it was later removed in 533e8dc5 but the unconditional
    correction remained. The flag is now wired up and defaults to True to
    match standard CIT-format convention (e.g. an AF step of 10 means 10 G = 1 mT).
    A -mT CLI flag is added for files that store AF steps in milliTesla.
  • ipmag.specimens_extract: pass axis as a keyword argument to
    DataFrame.dropna() — pandas no longer accepts it positionally.
  • pmagplotlib.save_plots: change the default dir_path from None to ".".
    The function unconditionally does os.path.join(dir_path, fname), so the
    None default caused a TypeError whenever callers (most of ipmag.py)
    omitted the argument.
  • convert.iodp_jr6_lore / convert.iodp_dscr_lore: cast treat_temp and
    treat_dc_field to object dtype after their string initialization.
    Pandas 3.0's strict str dtype rejects .loc[mask, col] = <float> partial
    assignments into string-typed columns, which broke these two LORE converters
    on ARM and TD rows.

Notebook updates (PmagPy_MagIC.ipynb)

  • convert_ages cell: cast frame to object before fillna('') so pandas 3.0
    accepts empty strings in numeric columns.
  • Private-contribution download cell: refresh contribution ID and share key
    (the original example credentials had been retired in MagIC).
  • upload_magic cell: forward input_dir_path explicitly (the function was
    split into separate input/output paths) and pass validate=False.
  • Cleanup cell: handle missing directories gracefully, wrap remaining
    os.remove calls in try/except FileNotFoundError, add 2g_asc_magic and
    2g_bin_magic parents to the directory list, sweep stray
    magic_contribution_*.txt at the project root, and clean
    data_files/download_magic/ while preserving the two example files.

Example data

  • Restore data_files/download_magic/sites.txt and
    magic_contribution_16533.txt. Both were deleted as collateral damage in
    commit 44293e8c (2021-06-20) when an in-progress extract_results.ipynb
    was added; the companion notebook was itself deleted 13 minutes later, but
    these data files were never put back. The download_magic section has
    been quietly broken on the missing sites.txt since.

Test plan

  • Run PmagPy_MagIC.ipynb cells under a kernel with current PmagPy and pandas 3.0 up to the
    interactive MagIC password prompt — all cells before that and after than complete without
    errors
  • convert.cit on data_files/convert_2_magic/cit_magic/USGS/bl9-1/bl9-1.sam:
    AF steps 100, 200, ..., 800 in the .sam file produce
    treat_ac_field = 0.01, 0.02, ..., 0.08 T in measurements.txt (Gauss interpretation)
  • convert.cit(..., oersted=False) on the same file: produces 10× larger
    values (mT interpretation)
  • convert.iodp_jr6_lore on data_files/iodp_magic/U999A: completes
    without TypeError and produces sensible treat_dc_field (5e-5 T) on
    ARM rows and treat_temp (273 + step) on TD rows
  • ipmag.aniso_magic_nb(infile='specimens.txt', dir_path='data_files/convert_2_magic/k15_magic/'):
    saves _aniso-data.png and _aniso-conf.png into CWD without
    TypeError

Pandas no longer accepts axis as a positional argument to DataFrame.dropna(),
so spec_df.dropna('columns', ...) raised a TypeError. Updated to
dropna(axis='columns', ...) to match the current API.
The oersted parameter introduced in 1deb048 (2017) was never branched
on, making the /10 Gauss-to-Tesla correction unconditional; the parameter
itself was later removed in 533e8dc but the unconditional correction
remained. Restore oersted=True as the default (matching the CIT format
convention where AF step values are recorded in Gauss) and gate the
conversion on the flag so callers with mT-formatted .sam files can pass
oersted=False to get correct Tesla values. Replace the redundant -oe CLI
flag with -mT.
…RE converters

pandas 3.0's strict str dtype rejects .loc[mask, col] = <float> partial
assignments into string-initialized columns. In iodp_jr6_lore, treat_temp
and treat_dc_field are initialized as strings and then partially overwritten
with float values for TD and ARM rows; in iodp_dscr_lore the same pattern
fires on treat_dc_field for ARM rows. Cast these columns to object dtype
immediately after their string init so the later partial assigns succeed.
save_plots had dir_path=None as default, then unconditionally did
os.path.join(dir_path, fname), which raises TypeError when no caller
forwards dir_path. Most callers in ipmag (aniso_magic, zeq_magic plot
saves, etc.) omit the argument, so they hit this whenever not running
on the MagIC server and no dpi kwarg is passed. Change the default to
"." so the docstring's promise ("directory in which to save plots") is
the actual fallback behavior.
Restore sites.txt and magic_contribution_16533.txt, which were deleted as
collateral damage in commit 44293e8 (2021-06-20) when an in-progress
extract_results.ipynb was added; the companion notebook was itself deleted
13 minutes later in 00f9151 but these data files were never put back.
The PmagPy_MagIC.ipynb download_magic section has been quietly broken on
that missing sites.txt ever since. sites.txt is restored from 9329ca3
(its last modification, 2020-09-04) and magic_contribution_16533.txt from
2ef7d21~1 (the last commit before its 2020-09-04 deletion).
…ples

- Cell 24 (convert_ages): cast frame to object before fillna('') so
  pandas 3.0's strict str dtype accepts empty strings in numeric columns
- Cell 57 (private download example): replace stale credentials
  (contribution 16960 / fa06255d-ec45-473f-9f33-fa2a9bb6d8d8) with a
  currently-shared contribution (20672) since the original was no longer
  accessible via the MagIC API
- Cell 60 (upload_magic): forward input_dir_path explicitly (the function
  was split into separate input/output paths) and pass validate=False
- Cell 78 (markdown): note that the next cells are skipped in the rendered
  documentation version of the notebook
- Cell 231 (cleanup): early-return in remove_magic_files when the directory
  is missing, wrap remaining os.remove calls in try/except FileNotFoundError,
  add 2g_asc_magic and 2g_bin_magic parents to the directory list, sweep
  leftover magic_contribution_*.txt at the project root, and clean
  data_files/download_magic/ while preserving the two example files

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates PmagPy to bring PmagPy_MagIC.ipynb and a handful of supporting conversion/plotting utilities back into a working state with pandas 3.0 and the current PmagPy/MagIC APIs, and restores missing example data used by the notebook.

Changes:

  • Add an -mT CLI flag / oersted option for CIT conversion so AF demag steps can be interpreted as either Gauss(Oe) or milliTesla.
  • Fix pandas 3.0 incompatibilities in ipmag.specimens_extract and IODP LORE converters by avoiding deprecated positional args and relaxing string-typed columns before partial numeric assignment.
  • Make pmagplotlib.save_plots default to saving into the current working directory instead of erroring when dir_path is omitted.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
programs/conversion_scripts/cit_magic.py Adds -mT flag wiring to pass oersted=False into the CIT converter.
pmagpy/pmagplotlib.py Changes save_plots default dir_path to "." and updates docstring accordingly.
pmagpy/ipmag.py Updates dropna call to use keyword axis= for pandas 3.0 compatibility.
pmagpy/convert_2_magic.py Implements oersted scaling for CIT AF steps; casts selected LORE measurement columns to object for pandas 3.0 partial assignment compatibility.
data_files/download_magic/sites.txt Restores example sites.txt data needed by the notebook’s download_magic section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pmagpy/convert_2_magic.py Outdated
Comment thread pmagpy/convert_2_magic.py
@Swanson-Hysell

Copy link
Copy Markdown
Member Author

This PR will close #835

… tests

- Move oersted=True to the end of the cit() signature (and docstring) so
  positional callers written against the post-2020 signature aren't broken.
- Add test_convert_cit.py to pmagpy/test (the CI test suite, pytest-style),
  asserting the expected Tesla values for the USGS bl9-1 sample under
  both Gauss (default) and mT interpretations.
- Mirror equivalent assertions into pmagpy_tests/test_imports3.py
  (TestCitMagic), the older unittest-based suite, for completeness.
The new test_convert_cit.py covers basic conversion of bl9-1.sam to the
five MagIC 3.0 tables (record counts, headers, sitename/locname
propagation, specimen↔measurement consistency), the oersted flag's Gauss
vs milliTesla interpretation of AF demag steps, and the actionable-error
return tuple for invalid inputs. Mirrors and extends the unittest-style
coverage in pmagpy_tests/test_imports3.py, which is the older suite that
CI does not run.
Several pmagpy/test/test_ipmag_*.py files import pmagpy.ipmag, which
conditionally pulls in cartopy via pmag.import_cartopy(). When cartopy
isn't installed in the CI image, pytest still collects the module but
downstream cartopy-using paths break test collection during import.
Adding cartopy to the workflow's pip install line resolves the missing
dependency so the suite collects cleanly.
pmagpy/convert_2_magic.py imports pytz at module top, so the new
test_convert_cit.py fails collection on a CI image where pytz isn't
already pulled in transitively. Adds pytz alongside cartopy in the
workflow's pip install line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

magic_contribution_16533.txt missing from data_files/download_magic

2 participants