Build tools for DART that do not need mkmf_ and path_names_ files #348

hkershaw-brown · 2022-05-06T17:22:31Z

Description:

There are science changes to DART in the pipeline that are potentially disruptive changes. However, a large part of the disruption is due to the build system in DART. For example, if you wanted to split filter_assim into two files, you would have to change ~730 path_names files.

The goal is have a build system that lets dart developers refactor code without the tax of altering individual mkmf_ and path_names files. And, ideally, as transparent as possible for DART users.

Slides about the build system are here
Original spec is here

Changes for users:

./quickbuild.sh rather than ./quickbuild.csh
To build a single program ./quickbuild.sh filter rather than
```
./preprocess
csh mkmf_filter -mpi
make
```
Note you can still iterate over changing code and compiling using make
```
./quickbuild.sh filter
*edit the code*
make
*edit the code*
make
```
.f90 files in the work directory take precedence over .f90 files with the same name elsewhere. So you can
have a local version of filter_assim_mod.f90 in work that will be used instead of
assimilation_code/modules/assimilation/assim_tools_mod.f90
quickdbuild.sh help gives the quickdbuild.sh usage.

For developers, added documentation in guide/quickbuild.rst about build functions and where these functions search for
source code. Html is here

Fixes issues

Fixes #216, fixes #335, fixes #333, fixes #157, fixes #129, fixes #327, fixes #114
developer_tests/time_manager now compiles.
Fixes #141 developer_tests/build_everything uses local obs_def/kind_mod.f90.

Renames obs_diag for streamflow to stream_flow_obs_diag. Note this does not fix issue #334.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

This pull request removes 1465 files ~30% of the repo

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.
- I have updated the DART tutorial pdfs.

Tests

Please describe any tests you ran to verify your changes.

Built DART executables (models, converters, developer tests) mac and Cheyenne

Note I've changed the code in models/wrf_hydro/hydro_dart_py/hydrodartpy/core/dartclasses.py to use quickdbuild.sh. However dartclasses.py expecting to ls the mkmf_ files to get a list of executables. I'm not sure where the list of executables is used so this pull request may break hydro_dart_py.

Checklist for merging

Updated changelog entry
Documentation updated
Version tag

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

preprocess builds

move streamflow_obs_diag to its own directory cice quickbuild.sh

quickbuild.csh wipes these. Do we want to make them?

nancycollins

i will confess to not reviewing all 1829 changed files individually. i did run tests and everything worked as expected. i think this is a positive step in lowering the barrier to making needed software changes and i approve this pull request.

not for this pull request, but in the future we should stop adding extensions to commands (e.g. quickbuild instead of quickbuild.csh or quickbuild.sh or quickbuild.py). then we can change the implementation language from csh to sh to python to a compiled program and it will all be transparent to the end user (and other scripts). commands should never have had .csh extensions. apparently it made it easier to have svn automatically set the permissions to add the executable flags. in my opinion that wasn't a good choice. it's too late for our existing scripts, but for future scripts it would be a better practice to drop the file extension altogether. (end rant)

hkershaw-brown · 2022-05-13T17:39:15Z

I love a good Nancy rant. Thanks for looking at this Nancy. I promise not to do any more 1829 file pull requests. (Probably this is a lie)

…

On Fri, May 13, 2022 at 11:36 AM nancy collins ***@***.***> wrote: ***@***.**** approved this pull request. i will confess to not reviewing all 1829 changed files individually. i did run tests and everything worked as expected. i think this is a positive step in lowering the barrier to making needed software changes and i approve this pull request. not for this pull request, but in the future we should stop adding extensions to commands (e.g. quickbuild instead of quickbuild.csh or quickbuild.sh or quickbuild.py). then we can change the implementation language from csh to sh to python to a compiled program and it will all be transparent to the end user (and other scripts). commands should never have had .csh extensions. apparently it made it easier to have svn automatically set the permissions to add the executable flags. in my opinion that wasn't a good choice. it's too late for our existing scripts, but for future scripts it would be a better practice to drop the file extension altogether. (end rant) — Reply to this email directly, view it on GitHub <#348 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEY6JH6FWLAAKS2IJGDXSB3VJ2HJTANCNFSM5VIYNTRQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

nancycollins · 2022-05-15T17:02:38Z

just a warning that with the new build system, i found that
models/wrf/experiments/Radar/IC/sounding_perturbation/pert_sounding_module.f90
does not compile with gfortran v11.2.0 on my mac. there are 3 places with an integer/real mismatch in subroutine arguments. this isn't our code - i'm guessing from the directory and name that it's used to perturb synthetic radar obs for testing/experiments. there are probably no makefiles that use this module but the new build system tries to compile everything and fails the wrf build because of this. i took a quick stab at trying to see if i could resolve the real(4) vs integer(4) mismatch but it wasn't simple to see what could be changed without breaking the code.

hkershaw-brown · 2022-05-16T14:16:45Z

just a warning that with the new build system, i found that models/wrf/experiments/Radar/IC/sounding_perturbation/pert_sounding_module.f90 does not compile with gfortran v11.2.0 on my mac. there are 3 places with an integer/real mismatch in subroutine arguments. this isn't our code - i'm guessing from the directory and name that it's used to perturb synthetic radar obs for testing/experiments. there are probably no makefiles that use this module but the new build system tries to compile everything and fails the wrf build because of this. i took a quick stab at trying to see if i could resolve the real(4) vs integer(4) mismatch but it wasn't simple to see what could be changed without breaking the code.

There is a script to compile this code:
DART/models/wrf/experiments/Radar/IC/sounding_perturbation/compile_pert_sounding.csh . It does not compile on main with gnu/11.2.0

hkershaw@cheyenne5:/glade/scratch/hkershaw/DART/Bugs/mismatch/DART/models/wrf/experiments/Radar/IC/sounding_perturbation(main)$ gfortran -c pert_sounding_module.f90 
pert_sounding_module.f90:175:27:

  175 |       amp(m) = mag*gaussdev(iseed1)
      |                           1
Error: Type mismatch in argument ‘idum’ at (1); passed REAL(4) to INTEGER(4)
pert_sounding_module.f90:179:34:

  179 |          phs(m) = 2.0*PI*(2.0*ran2(iseed2)-1.0)
      |                                  1
Error: Type mismatch in argument ‘idum’ at (1); passed REAL(4) to INTEGER(4)
pert_sounding_module.f90:192:25:

  192 |       q(k) = mag*gaussdev(iseed1)
      |                         1
Error: Type mismatch in argument ‘idum’ at (1); passed REAL(4) to INTEGER(4)
hkershaw@cheyenne5:/glade/scratch/hkershaw/DART/Bugs/mismatch/DART/models/wrf/experiments/Radar/IC/sounding_perturbation(main)$ gfortran --version
GNU Fortran (GCC) 11.2.0

I will move this to an issue, I think fix or remove this code from the repo, set as EXCLUDE in this pull request

pert_sounding does not compile with gfortran/11. see #351. May remove this code. bug-fix: remove iargc interface for advance_cymdh

see #302 for notes on different obs error files

clean after reading arguements.

Needed for switching between PGI and other compilers on Cheyenne

hkershaw-brown added 30 commits November 11, 2021 15:00

ignoring observation converter and model specific exectuables

375f71c

deleted mkmf_ files

b1323b8

removed path_names

34ad4ef

buildfunctions.sh from the destroy branch

7fb76ac

remove quickbuild.csh

b8a497a

removed workshop_setup.csh

b6b0e8b

quickbuild.sh from the destroy branch

74b65b6

preprocess path_names in the preprocess directory

6e5e7de

actually in the preprocess directory

391057c

mkmf.template defined in mkmf (or supplied by -t)

805dcf7

preprocess builds

part way through build functions without moving code. mpi vs no mpi

94ebf0f

DART env for mkmf fixsystem

2260636

buildfunctions.sh without moving code

032f718

successful build for lorenz_96 (no model specific programs)

46f5267

remove model specific programs from dartsrc

3f36e15

move streamflow_obs_diag to its own directory cice quickbuild.sh

model specific programs

b72e51e

split mpi and serial programs

f9cd002

model quickbuilds.sh

1765a67

ignore column_rand executable

837f9e8

more quickbuild.sh move brig solo files

7c01ae0

wrf programs in various directories

e846431

bgrid solo extra files

ff429ae

bgrid_solo exectutables

a7df430

wrf_hydro uses code from observation converters utilties

ac01888

bgrid solo extra source file

ed7cebb

mpas_atm moved programs that do not compile

961244f

giving the extra source file an extension. Picked .path_names

a24e785

noah uses wrf and wrf_hyfro code

2ad9c66

collect input.nml.defaults from directories.

bcf7a82

quickbuild.csh wipes these. Do we want to make them?

remove creating input.nml.defaults

ce601e2

hkershaw-brown added 5 commits May 12, 2022 07:56

NASA_Earthdata quickdbuild.sh

c6b711b

NSIDC see #340 for arguement mismatch in HDF5 utilties mod

71b700f

ocean_color quickbuild.sh

8019ffc

cleanup of input.nml*_default for converter builds

cf10048

typos and radiance_obs_seq_to_netcdf added to .gitignore

f712f97

hkershaw-brown marked this pull request as ready for review May 12, 2022 14:52

hkershaw-brown requested a review from nancycollins May 12, 2022 14:53

nancycollins approved these changes May 13, 2022

View reviewed changes

hkershaw-brown mentioned this pull request May 16, 2022

argument mismatch wrf pert_model_sounding #351

Closed

hkershaw-brown added 13 commits May 16, 2022 09:42

exclude experiments directory

21feb2b

pert_sounding does not compile with gfortran/11. see #351. May remove this code. bug-fix: remove iargc interface for advance_cymdh

bug-fix: SSEC converter needs MADIS meteor and ncep obs_error

851fa6d

see #302 for notes on different obs error files

bug-fix: dart_gitm_mod is not a program

bf99bc4

bug-fix: source functions imediately

1e21267

remove duplicate from quickdbuilds that had it.

72ec7ed

clean after reading arguements.

Merge branch 'main' into build-system

0ff2730

developer tests quickbuild.sh need preprocess

102f038

quickbuild.csh to quickbuild.sh from latest main update

6cbabe4

doc: mpi_utilities mpi vs nompi

36e5c65

clean preprocess build with quickbuild.sh clean

53e079c

Needed for switching between PGI and other compilers on Cheyenne

missed from last commit

eddc1c9

Merge branch 'main' into build-system

6dc23fc

bump version and changelog for release

eb431b3

hkershaw-brown merged commit cb7455e into main May 24, 2022

hkershaw-brown deleted the build-system branch May 24, 2022 18:29

hkershaw-brown mentioned this pull request Jul 13, 2022

DART geo template model out of date #300

Closed

hkershaw-brown mentioned this pull request Dec 22, 2022

Remove out of date build docs #443

Merged

15 tasks

hkershaw-brown mentioned this pull request May 31, 2023

obs_converter notes: obs_err mods #302

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Build tools for DART that do not need mkmf_ and path_names_ files #348

Build tools for DART that do not need mkmf_ and path_names_ files #348

Uh oh!

hkershaw-brown commented May 6, 2022 •

edited

Loading

Uh oh!

nancycollins left a comment

Uh oh!

hkershaw-brown commented May 13, 2022 via email •

edited

Loading

Uh oh!

nancycollins commented May 15, 2022

Uh oh!

hkershaw-brown commented May 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Build tools for DART that do not need mkmf_ and path_names_ files #348

Build tools for DART that do not need mkmf_ and path_names_ files #348

Uh oh!

Conversation

hkershaw-brown commented May 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Fixes issues

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Testing Datasets

Uh oh!

nancycollins left a comment

Choose a reason for hiding this comment

Uh oh!

hkershaw-brown commented May 13, 2022 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nancycollins commented May 15, 2022

Uh oh!

hkershaw-brown commented May 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hkershaw-brown commented May 6, 2022 •

edited

Loading

hkershaw-brown commented May 13, 2022 via email •

edited

Loading