Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@wilfonba
Copy link
Collaborator

@wilfonba wilfonba commented Jun 21, 2025

User description

Description

This PR adds the basic functionality of IGR to the code base. It includes 3rd and 5th order reconstruction, viscous and inviscid simulations, 2D and 3D simulations, Jacobi and Gauss-Seidel iterative solves, and an optional pressure limiter. It does not include more advanced features like unified memory or downsampling of I/O files.

Note (6/21/25, 5pm) - The current implementation matches the GB code performance on A100 GPUs. The performance of this implementation is still lacking on AMD hardware.

Note (7/6/25) - The update Frontier implementation is faster, and new traces have been added. The grindtime of the implementation in this code for a 300^3 case is 0.84. The grind time for the GB code with the same problem size was 0.64.

Note (7/8/25) - Frontier performance on this branch is now better than the BG code, it seems? 1.70 seconds on the GB code versus 1.33 seconds on this code? New hip traces have been added for both the GB code and this code for a 700^3 case.

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

Scope

  • This PR comprises a set of related changes with a common goal

How Has This Been Tested?

Checklist

  • I have added comments for the new code
  • I added Doxygen docstrings to the new code
  • I have made corresponding changes to the documentation (docs/)
  • I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
  • I have added example cases in examples/ that demonstrate my new feature performing as expected.
    They run to completion and demonstrate "interesting physics"
  • I ran ./mfc.sh format before committing my code
  • New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
  • This PR does not introduce any repeated code (it follows the DRY principle)
  • I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

  • Checked that the code compiles using NVHPC compilers
  • Checked that the code compiles using CRAY compilers
  • Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
  • Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
  • Enclosed the new feature via nvtx ranges so that they can be identified in profiles
  • Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
    Current implementation summary:
    newNV.csv
    GB Implementation summary (warm start iteration count is different)
    oldNV.csv
  • Ran a Rocprof Systems profile using ./mfc.sh run XXXX --gpu -t simulation --rsys --hip-trace, and have attached the output file and plain text results to this PR.
    Current Implementation Trace:
    newnewFrontier.csv
    GB Implementation Trace:
    oldoldFrontier.csv
  • Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature
    scaling

PR Type

Enhancement


Description

• Implements Information Geometric Regularization (IGR) as a new numerical method for fluid dynamics simulations
• Adds comprehensive IGR solver with 3rd and 5th order reconstruction, supporting both viscous and inviscid flows
• Integrates IGR throughout the entire codebase including simulation, pre-processing, and post-processing modules
• Provides IGR-specific boundary conditions, time stepping, and iterative solvers (Jacobi and Gauss-Seidel)
• Adds extensive test cases and example configurations for 2D and 3D IGR simulations
• Includes performance optimizations and GPU support for IGR computations
• Updates documentation with IGR parameter descriptions and methodology references
• Maintains backward compatibility with existing WENO-based solver functionality


Changes walkthrough 📝

Relevant files
Enhancement
31 files
m_rhs.fpp
Integration of Information Geometric Regularization solver

src/simulation/m_rhs.fpp

• Added use m_igr module import for Information Geometric
Regularization functionality
• Wrapped existing finite volume solver
code with conditional .not. igr checks
• Added IGR-specific code paths
including s_igr_riemann_solver, s_igr_iterative_solve, and
s_igr_sigma_x calls
• Modified memory allocation patterns to support
both IGR and traditional finite volume approaches

+687/-593
m_time_steppers.fpp
Time stepper modifications for IGR support                             

src/simulation/m_time_steppers.fpp

• Added conditional allocation of primitive variables based on igr
flag
• Modified RHS variable allocation to use extended bounds for IGR
(-1:m+1,-1:n+1,-1:p+1)
• Updated run-time information calls to use
conservative variables for IGR instead of primitive variables
• Added
IGR-specific enthalpy computation calls in time stepping routines

+137/-110
m_boundary_common.fpp
IGR boundary condition support implementation                       

src/common/m_boundary_common.fpp

• Added new jac_sf scalar field declaration for Jacobian handling

Implemented s_populate_F_igr_buffers subroutine for IGR boundary
condition handling
• Added support for periodic, reflective, and
default boundary conditions for IGR Jacobian field

+165/-0 
m_initial_condition.fpp
Conservative variable allocation bounds update                     

src/pre_process/m_initial_condition.fpp

• Modified conservative variable allocation to use extended buffer
bounds instead of domain bounds
• Changed from (0:m, 0:n, 0:p) to
(idwbuff(1)%beg:idwbuff(1)%end, idwbuff(2)%beg:idwbuff(2)%end,
idwbuff(3)%beg:idwbuff(3)%end)

+3/-1     
m_start_up.fpp
Integrate IGR module initialization and finalization         

src/simulation/m_start_up.fpp

• Added IGR module import and initialization/finalization logic

Modified module initialization order to conditionally use IGR or WENO
modules
• Updated grind time calculation with proper type casting

Added IGR-specific GPU variable updates

+33/-16 
m_mpi_common.fpp
Update MPI topology for IGR reconstruction order support 

src/common/m_mpi_common.fpp

• Added reconstruction order variable to handle both WENO and IGR
orders
• Updated processor topology calculations to use generic
reconstruction order
• Modified error messages to reference both WENO
and IGR orders

+17/-9   
m_checker.fpp
Add IGR input validation and compatibility checks               

src/simulation/m_checker.fpp

• Added comprehensive IGR input validation function
• Reorganized
input checking to conditionally validate IGR or WENO parameters

Added IGR compatibility checks with other features

+49/-15 
m_mpi_proxy.fpp
Add MPI support for IGR parameters                                             

src/simulation/m_mpi_proxy.fpp

• Added MPI broadcasting for IGR-related parameters
• Included new IGR
variables in MPI communication arrays

+8/-3     
m_checker_common.fpp
Add common IGR validation and improve cell counting           

src/common/m_checker_common.fpp

• Added IGR input validation function with order and grid size checks

• Reorganized common input checking to conditionally validate IGR

Updated total cells calculation with proper integer types

+29/-14 
m_sim_helpers.fpp
Implement IGR-specific variable conversion logic                 

src/simulation/m_sim_helpers.fpp

• Added IGR-specific variable extraction logic for density and volume
fractions
• Modified velocity and energy calculations for IGR
compatibility
• Updated primitive variable handling based on IGR vs
WENO modes

+39/-12 
m_variables_conversion.fpp
Update variable conversion for IGR support                             

src/common/m_variables_conversion.fpp

• Added IGR-specific alpha_rho and alpha extraction logic
• Modified
variable conversion to handle IGR's different variable organization

Updated conservative variable handling for IGR compatibility

+40/-10 
3dHardcodedIC.fpp
Add 3D IGR jet hardcoded initial condition                             

src/pre_process/include/3dHardcodedIC.fpp

• Added new hardcoded initial condition case 302 for 3D IGR jet
simulation
• Implemented jet flow field initialization with smooth
transitions

+33/-1   
m_start_up.f90
Add IGR support to post-processing startup                             

src/post_process/m_start_up.f90

• Added IGR parameter reading in input file processing
• Modified
alpha field calculation for IGR compatibility in post-processing

+16/-2   
m_global_parameters.fpp
Add IGR global parameters and system configuration             

src/pre_process/m_global_parameters.fpp

• Added IGR global parameters and initialization
• Modified system
size calculation for IGR's different variable organization
• Updated
coordinate bounds configuration to include IGR parameter

+14/-2   
m_global_parameters.fpp
Add IGR parameters to post-processing globals                       

src/post_process/m_global_parameters.fpp

• Added IGR parameters to post-processing global parameters
• Modified
system size calculation for IGR compatibility

+12/-1   
m_data_output.fpp
Disable primitive variable output for IGR mode                     

src/simulation/m_data_output.fpp

• Disabled primitive variable writing for IGR mode in 1D, 2D, and 3D
cases
• Added conditional checks to prevent incompatible output
formats with IGR

+3/-3     
m_data_output.fpp
Update data output for IGR boundary conditions                     

src/pre_process/m_data_output.fpp

• Modified boundary condition file writing to use conservative
variables for IGR
• Updated data file writing to use proper array
bounds

+11/-3   
m_helper_basic.fpp
Update coordinate bounds for IGR buffer requirements         

src/common/m_helper_basic.fpp

• Updated coordinate bounds configuration to handle IGR-specific
buffer sizes
• Added IGR parameter to buffer size calculation function

+13/-4   
m_start_up.fpp
Add IGR parameter support to pre-processing startup           

src/pre_process/m_start_up.fpp

• Added IGR parameter reading in input file processing
• Updated
global cell count calculation with proper integer types

+2/-2     
m_mpi_proxy.fpp
Add IGR MPI support to post-processing                                     

src/post_process/m_mpi_proxy.fpp

• Added IGR parameter to MPI broadcasting in post-processing

+3/-2     
m_constants.fpp
Add IGR default constants                                                               

src/common/m_constants.fpp

• Added default constants for IGR iteration parameters
• Defined
default values for IGR solver configuration

+3/-0     
m_patch_helper.fpp
Add patch helper module with smooth transition functions 

src/pre_process/m_patch_helper.fpp

• Added new module with smooth cutoff functions for patch
initialization
• Implemented mathematical functions for smooth
transitions in initial conditions

+47/-0   
m_mpi_proxy.fpp
Add IGR MPI support to pre-processing                                       

src/pre_process/m_mpi_proxy.fpp

• Added IGR parameter to MPI broadcasting in pre-processing

+2/-1     
m_patches.fpp
Import patch helper module                                                             

src/pre_process/m_patches.fpp

• Added patch helper module import for smooth transition functions

+2/-0     
case.py
Add 3D IGR jet example case                                                           

examples/3D_IGR_jet/case.py

• Added complete 3D jet simulation case configuration for IGR

Configured domain, fluid properties, and IGR-specific parameters

+128/-0 
case.py
Add 2D IGR triple point example case                                         

examples/2D_IGR_triple_point/case.py

• Added 2D triple point shock interaction case for IGR validation

Configured multi-fluid setup with IGR parameters

+102/-0 
case.py
Add IGR benchmark case                                                                     

benchmarks/igr/case.py

• Added IGR benchmark case for performance testing
• Configured
Taylor-Green vortex problem with viscous IGR

+113/-0 
case.py
Add 2D IGR two-fluid example case                                               

examples/2D_IGR_2fluid/case.py

• Added 2D two-fluid blast wave case for IGR testing
• Configured
multi-fluid setup with pressure jump

+105/-0 
case.py
Add 3D IGR Taylor-Green vortex example case                           

examples/3D_IGR_TaylorGreenVortex/case.py

• Added 3D Taylor-Green vortex case for IGR validation
• Configured
viscous flow with analytical initial conditions

+95/-0   
case.py
Add IGR support to case configuration system                         

toolchain/mfc/case.py

• Added IGR parameter handling in case configuration
• Updated WENO
order logic to accommodate IGR settings
• Added IGR-specific
preprocessor definitions

+15/-2   
case_dicts.py
Add IGR parameters to case dictionary system                         

toolchain/mfc/run/case_dicts.py

• Added IGR parameters to case dictionary definitions
• Updated case
optimization parameters to include IGR settings

+10/-1   
Configuration changes
2 files
m_global_parameters.fpp
Global parameter additions for IGR configuration                 

src/simulation/m_global_parameters.fpp

• Added IGR-related parameters: igr_iter_solver, igr_order, igr,
igr_pres_lim, and viscous
• Modified system size calculation to
account for IGR with different fluid configurations
• Added
initialization of IGR parameters with default values
• Updated
coordinate bounds configuration to include IGR parameter

+34/-5   
frontier.mako
Update Frontier template for IGR requirements                       

toolchain/templates/frontier.mako

• Added unlimited stack size setting for IGR computations
• Added
unbuffered output flag for interactive runs

+3/-1     
Bug fix
2 files
m_surface_tension.fpp
Fix GPU loop macro in surface tension module                         

src/simulation/m_surface_tension.fpp

• Fixed GPU loop macro usage for proper parallelization

+1/-1     
m_checker.fpp
Add precision validation for output format                             

src/post_process/m_checker.fpp

• Added precision validation to prevent double precision output with
single precision build

+1/-0     
Tests
26 files
cases.py
Add IGR test case generation and GPU filtering                     

toolchain/mfc/test/cases.py

• Added IGR test case generation with different orders and solvers

Integrated IGR testing into existing test framework
• Added
GPU-specific test filtering for Gauss-Seidel solver

+37/-5   
test.py
Add GPU test filtering for IGR solvers                                     

toolchain/mfc/test/test.py

• Added GPU-specific test filtering for Gauss-Seidel solver

+5/-0     
golden-metadata.txt
Add IGR test metadata file                                                             

tests/9F4E47F1/golden-metadata.txt

• Added test metadata file for IGR test case

+183/-0 
golden-metadata.txt
Add IGR test metadata file                                                             

tests/0045D9F8/golden-metadata.txt

• Added test metadata file for IGR test case

+183/-0 
golden-metadata.txt
Add IGR test metadata file                                                             

tests/238A7B1F/golden-metadata.txt

• Added test metadata file for IGR test case

+183/-0 
golden-metadata.txt
Add IGR test metadata file                                                             

tests/BF8FDC2A/golden-metadata.txt

• Added test metadata file for IGR test case

+183/-0 
golden-metadata.txt
Add IGR test metadata file                                                             

tests/023DC5B9/golden-metadata.txt

• Added test metadata file for IGR test case

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case 106C0BE6                         

tests/106C0BE6/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components (pre_process,
simulation, post_process, syscheck, documentation)
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case 18A71AB2                         

tests/18A71AB2/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case 2699E9DF                         

tests/2699E9DF/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case 4440D46B                         

tests/4440D46B/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case 9EE432D4                         

tests/9EE432D4/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case A6DB36C2                         

tests/A6DB36C2/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case AE3FC5CB                         

tests/AE3FC5CB/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case C7A6B609                         

tests/C7A6B609/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case E49EF7B6                         

tests/E49EF7B6/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case E8979E4A                         

tests/E8979E4A/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case EB58AF7F                         

tests/EB58AF7F/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case 37DDE283                         

tests/37DDE283/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case 6077374F                         

tests/6077374F/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case A57E30FE                         

tests/A57E30FE/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case AFBCBDFA                         

tests/AFBCBDFA/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case CEAF553A                         

tests/CEAF553A/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case D49DCFC0                         

tests/D49DCFC0/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case D7885E17                         

tests/D7885E17/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
golden-metadata.txt
Add test metadata for IGR test case F44A293A                         

tests/F44A293A/golden-metadata.txt

• Added new test metadata file with build configuration details

Contains CMake configuration for all MFC components
• Records system
information including Apple M2 CPU details and compiler versions

+183/-0 
Documentation
2 files
references.md
Add IGR methodology reference                                                       

docs/documentation/references.md

• Added reference to IGR methodology paper by Cao and Schäfer

+3/-1     
case.md
Document IGR configuration parameters and fix typo             

docs/documentation/case.md

• Added documentation for IGR (Information Geometric Regularization)
parameters
• Added 6 new IGR-related configuration options including
igr, igr_order, alf_factor, etc.
• Fixed typo in radius parameter (was
radiue)

+8/-1     
Additional files
36 files
CMakeLists.txt +9/-9     
m_igr.fpp +2621/-0
golden.txt +12/-0   
golden.txt +12/-0   
golden.txt +12/-0   
golden.txt +14/-0   
golden.txt +12/-0   
golden.txt +12/-0   
golden.txt +10/-0   
golden.txt +12/-0   
golden.txt +12/-0   
golden.txt +12/-0   
golden.txt +14/-0   
golden.txt +12/-0   
golden.txt +12/-0   
golden.txt +10/-0   
golden.txt +10/-0   
golden-metadata.txt +183/-0 
golden.txt +360/-0 
golden.txt +14/-0   
golden.txt +10/-0   
golden.txt +12/-0   
golden.txt +14/-0   
golden.txt +14/-0   
golden-metadata.txt +183/-0 
golden.txt +12/-0   
golden.txt +12/-0   
golden.txt +10/-0   
golden.txt +10/-0   
golden.txt +14/-0   
golden-metadata.txt +183/-0 
golden.txt +14/-0   
golden-metadata.txt +183/-0 
golden.txt +12/-0   
bench.yaml +8/-0     
case.py +0/-2     

Need help?
  • Type /help how to ... in the comments thread for any questions about Qodo Merge usage.
  • Check out the documentation for more information.
  • @sbryngelson sbryngelson self-requested a review July 14, 2025 06:31
    @sbryngelson sbryngelson self-requested a review July 15, 2025 00:45
    Copy link
    Collaborator

    @abbotts abbotts left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    I didn't do a close read so I can't speak to the math being right (I'll leave that up to others and the tests), but I did flag a couple portability issues that would be good to address before this gets merged.

    @sbryngelson sbryngelson self-requested a review July 17, 2025 00:24
    @sbryngelson sbryngelson requested review from abbotts and removed request for abbotts July 17, 2025 00:27
    @sbryngelson sbryngelson dismissed abbotts’s stale review July 17, 2025 15:18

    run benchmarking

    Copy link
    Member

    @sbryngelson sbryngelson left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    i added a comment on the helper file. also can you add to the readme IGR capability.

    @sbryngelson
    Copy link
    Member

    can merge after that

    @sbryngelson sbryngelson merged commit 658f188 into MFlowCode:master Jul 19, 2025
    39 of 41 checks passed
    @sbryngelson sbryngelson deleted the IGRMergeOne branch July 19, 2025 13:33
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Development

    Successfully merging this pull request may close these issues.

    4 participants