Refac data input - Post process #902

sbryngelson · 2025-06-22T04:12:05Z

User description

Ton of duplicate junk to refactor. Let's see if it works.

PR Type

Enhancement

Description

Refactored data input module by extracting common code into helper subroutines
Eliminated duplicate code for grid data reading across x, y, z directions
Consolidated MPI I/O parameter setup and IB data file reading
Simplified field array allocation with unified helper function

Changes walkthrough 📝

Relevant files

Enhancement

m_data_input.f90 `Refactor data input with helper subroutines` src/post_process/m_data_input.f90 Added 5 new helper subroutines to eliminate code duplication Replaced repetitive grid data reading with `s_read_grid_data_direction` Consolidated MPI I/O setup into `s_setup_mpi_io_params` Unified field array allocation with `s_allocate_field_arrays` Extracted IB data reading into `s_read_ib_data_files` Split parallel data reading into separate conservative data subroutine	+194/-245

Need help?
Type /help how to ... in the comments thread for any questions about Qodo Merge usage.
Check out the documentation for more information.

qodo-merge-pro · 2025-06-22T04:12:34Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Possible Issue The helper subroutine `s_read_grid_data_direction` uses hardcoded file unit 1 for all file operations, which could cause conflicts if multiple files are opened simultaneously or if unit 1 is already in use elsewhere in the codebase. open (1, FILE=trim(file_loc), FORM='unformatted', & STATUS='old', ACTION='read') read (1) cb_array(-1:size_dim) close (1) Code Quality The `s_setup_mpi_io_params` subroutine has many output parameters (8 total) which makes the interface complex and error-prone. Consider using a derived type or reducing the number of parameters. impure subroutine s_setup_mpi_io_params(data_size, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK) integer, intent(out) :: data_size integer(KIND=MPI_OFFSET_KIND), intent(out) :: m_MOK, n_MOK, p_MOK integer(KIND=MPI_OFFSET_KIND), intent(out) :: WP_MOK, MOK, str_MOK, NVARS_MOK ! Initialize MPI data I/O if (ib) then call s_initialize_mpi_data(q_cons_vf, ib_markers) else call s_initialize_mpi_data(q_cons_vf) end if ! Size of local arrays data_size = (m + 1)(n + 1)(p + 1) ! Resize some integers so MPI can read even the biggest file m_MOK = int(m_glb + 1, MPI_OFFSET_KIND) n_MOK = int(n_glb + 1, MPI_OFFSET_KIND) p_MOK = int(p_glb + 1, MPI_OFFSET_KIND) WP_MOK = int(8._wp, MPI_OFFSET_KIND) MOK = int(1._wp, MPI_OFFSET_KIND) str_MOK = int(name_len, MPI_OFFSET_KIND) NVARS_MOK = int(sys_size, MPI_OFFSET_KIND) Duplicate Logic The IB data reading logic is duplicated between serial and parallel versions within `s_read_ib_data_files`. The MPI and non-MPI code paths could be further consolidated to reduce duplication. #ifdef MFC_MPI call MPI_FILE_OPEN(MPI_COMM_WORLD, file_loc, MPI_MODE_RDONLY, mpi_info_int, ifile, ierr) data_size = (m + 1)(n + 1)(p + 1) disp = 0 call MPI_FILE_SET_VIEW(ifile, disp, MPI_INTEGER, MPI_IO_IB_DATA%view, & 'native', mpi_info_int, ierr) call MPI_FILE_READ(ifile, MPI_IO_IB_DATA%var%sf, data_size, & MPI_INTEGER, status, ierr) call MPI_FILE_CLOSE(ifile, ierr) #else open (2, FILE=trim(file_loc), & FORM='unformatted', & ACTION='read', & STATUS='old') read (2) ib_markers%sf(0:m, 0:n, 0:p) close (2) #endif else call s_mpi_abort('File '//trim(file_loc)//' is missing. Exiting.') end if

qodo-merge-pro · 2025-06-22T04:13:36Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Add MPI error checking Add error checking for MPI operations to handle potential failures gracefully. The current implementation doesn't verify if MPI calls succeed, which could lead to silent failures or undefined behavior. src/post_process/m_data_input.f90 [144-184] impure subroutine s_read_ib_data_files(file_loc_base) character(len=), intent(in) :: file_loc_base character(LEN=len_trim(file_loc_base) + 20) :: file_loc logical :: file_exist integer :: ifile, ierr, data_size integer, dimension(MPI_STATUS_SIZE) :: status integer(KIND=MPI_OFFSET_KIND) :: disp if (.not. ib) return write (file_loc, '(A)') trim(file_loc_base)//'/ib.dat' inquire (FILE=trim(file_loc), EXIST=file_exist) if (file_exist) then #ifdef MFC_MPI call MPI_FILE_OPEN(MPI_COMM_WORLD, file_loc, MPI_MODE_RDONLY, mpi_info_int, ifile, ierr) + if (ierr /= MPI_SUCCESS) call s_mpi_abort('Failed to open IB file: '//trim(file_loc)) data_size = (m + 1)(n + 1)*(p + 1) disp = 0 call MPI_FILE_SET_VIEW(ifile, disp, MPI_INTEGER, MPI_IO_IB_DATA%view, & 'native', mpi_info_int, ierr) + if (ierr /= MPI_SUCCESS) call s_mpi_abort('Failed to set MPI file view for IB data') + call MPI_FILE_READ(ifile, MPI_IO_IB_DATA%var%sf, data_size, & MPI_INTEGER, status, ierr) + if (ierr /= MPI_SUCCESS) call s_mpi_abort('Failed to read IB data from file') call MPI_FILE_CLOSE(ifile, ierr) #else open (2, FILE=trim(file_loc), & FORM='unformatted', & ACTION='read', & STATUS='old') read (2) ib_markers%sf(0:m, 0:n, 0:p) close (2) #endif else call s_mpi_abort('File '//trim(file_loc)//' is missing. Exiting.') end if end subroutine s_read_ib_data_files Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies that the return codes from MPI calls are not checked. Adding checks for the `ierr` variable after `MPI_FILE_OPEN`, `MPI_FILE_SET_VIEW`, and `MPI_FILE_READ` significantly improves the code's robustness by preventing silent failures.	Medium
General	Increase file path buffer size The hardcoded buffer size of 20 characters may be insufficient for longer file paths or extensions. Consider using a more robust approach to ensure adequate buffer space for file path construction. src/post_process/m_data_input.f90 [148] -character(LEN=len_trim(file_loc_base) + 20) :: file_loc +character(LEN=len_trim(file_loc_base) + 50) :: file_loc Suggestion importance[1-10]: 5 __ Why: The suggestion correctly points out that a fixed-size buffer for file paths can be risky. While the `improved_code`'s larger fixed size is a partial improvement, a more robust solution would dynamically determine the required length. The suggestion is valid for improving robustness against buffer overflows.	Low
Update

codecov · 2025-06-22T06:14:58Z

Codecov Report

Attention: Patch coverage is 53.62319% with 32 lines in your changes missing coverage. Please review.

Please upload report for BASE (master@c1e5159). Learn more about missing BASE report.
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
src/post_process/m_data_input.f90	53.62%	26 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master     #902   +/-   ##
=========================================
  Coverage          ?   45.98%           
=========================================
  Files             ?       68           
  Lines             ?    18629           
  Branches          ?     2239           
=========================================
  Hits              ?     8566           
  Misses            ?     8711           
  Partials          ?     1352

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sbryngelson · 2025-06-25T01:15:22Z

@wilfonba can you review this by just making sure it works - in particular for a short IBM case with parallel_io true and false. Just look at the output from this branch and see if it matches output from master.

wilfonba · 2025-06-26T19:04:29Z

@sbryngelson Sorry, I'm just now seeing this comment. Yeah, I'll put it on my list.

sbryngelson · 2025-06-26T19:05:38Z

@sbryngelson Sorry, I'm just now seeing this comment. Yeah, I'll put it on my list.

tyty

wilfonba · 2025-06-27T20:04:32Z

I fixed an IB bug in the PR code. Here's the results:

Serial I/O

test.mp4

Parallel I/O

test.mp4

Everything between the two is identical, except for the master branch doesn't seem to have the correct IB markers in the silo files with parallel_io = F, meaning that this PR fixes a bug in the master branch as well.

sbryngelson · 2025-06-27T20:17:37Z

Fun. Does this mean it's mergeable, or am I missing something subtle?

wilfonba · 2025-06-27T20:57:09Z

Fun. Does this mean it's mergeable, or am I missing something subtle?

It's mergeable

Copilot

Pull Request Overview

This PR refactors the data input module to eliminate duplicate code by extracting common file reading and MPI I/O setup logic into dedicated helper subroutines. Key changes include:

Addition of new helper subroutines such as s_read_grid_data_direction, s_setup_mpi_io_params, s_read_ib_data_files, and s_allocate_field_arrays.
Replacing repetitive grid data reading and MPI I/O operations with the newly created helper functions.
Consolidation of both serial and parallel data file reading logic, reducing duplication and improving maintainability.

Comments suppressed due to low confidence (1)

src/post_process/m_data_input.f90:433

Consider adding a detailed documentation header for the subroutine s_read_parallel_conservative_data, similar to the other helper subroutines, to clearly describe its parameters and expected behavior.

    impure subroutine s_read_parallel_conservative_data(t_step, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)

Copilot · 2025-06-28T22:53:15Z

src/post_process/m_data_input.f90

+            open (1, FILE=trim(file_loc), FORM='unformatted', &
+                  STATUS='old', ACTION='read')
+            read (1) cb_array(-1:size_dim)
+            close (1)


Consider using parameterized or named file unit numbers instead of the hard-coded unit 1 to prevent potential conflicts and improve code clarity.

Suggested change

open (1, FILE=trim(file_loc), FORM='unformatted', &

STATUS='old', ACTION='read')

read (1) cb_array(-1:size_dim)

close (1)

open (UNIT_CB, FILE=trim(file_loc), FORM='unformatted', &

STATUS='old', ACTION='read')

read (UNIT_CB) cb_array(-1:size_dim)

close (UNIT_CB)

Copilot · 2025-06-28T22:53:15Z

src/post_process/m_data_input.f90

+        file_loc = trim(t_step_ib_dir)//'/.'
+        call my_inquire(file_loc, dir_check)


[nitpick] The variable 'file_loc' is reused for different directory paths, which may reduce clarity; consider using a distinct variable name for IB directory handling to improve maintainability.

Suggested change

file_loc = trim(t_step_ib_dir)//'/.'

call my_inquire(file_loc, dir_check)

file_loc_ib = trim(t_step_ib_dir)//'/.'

call my_inquire(file_loc_ib, dir_check)

Malmahrouqi3 · 2025-07-12T14:38:55Z

Got these errors recently when using ./mfc.sh run with --no-mpi

/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:118:36:

  118 |         integer(KIND=MPI_OFFSET_KIND), intent(out) :: m_MOK, n_MOK, p_MOK
      |                                    1
Error: Symbol ‘mpi_offset_kind’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:119:36:

  119 |         integer(KIND=MPI_OFFSET_KIND), intent(out) :: WP_MOK, MOK, str_MOK, NVARS_MOK
      |                                    1
Error: Symbol ‘mpi_offset_kind’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:152:36:

  152 |         integer(KIND=MPI_OFFSET_KIND) :: disp
      |                                    1
Error: Symbol ‘mpi_offset_kind’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:436:36:

  436 |         integer(KIND=MPI_OFFSET_KIND), intent(inout) :: m_MOK, n_MOK, p_MOK
      |                                    1
Error: Symbol ‘mpi_offset_kind’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:437:36:

  437 |         integer(KIND=MPI_OFFSET_KIND), intent(inout) :: WP_MOK, MOK, str_MOK, NVARS_MOK
      |                                    1
Error: Symbol ‘mpi_offset_kind’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:433:69:

  433 |     impure subroutine s_read_parallel_conservative_data(t_step, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                     1
Error: Symbol ‘m_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:433:76:

  433 |  impure subroutine s_read_parallel_conservative_data(t_step, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘n_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:433:83:

  433 |  subroutine s_read_parallel_conservative_data(t_step, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘p_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:433:91:

  433 | ine s_read_parallel_conservative_data(t_step, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘wp_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:433:96:

  433 | _read_parallel_conservative_data(t_step, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:433:105:

  433 | allel_conservative_data(t_step, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘str_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:433:116:

  433 | lel_conservative_data(t_step, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                                  1

Error: Symbol ‘nvars_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:115:60:

  115 |     impure subroutine s_setup_mpi_io_params(data_size, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                            1
Error: Symbol ‘m_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:115:67:

  115 |     impure subroutine s_setup_mpi_io_params(data_size, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                   1
Error: Symbol ‘n_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:115:74:

  115 |    impure subroutine s_setup_mpi_io_params(data_size, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘p_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:115:82:

  115 | e subroutine s_setup_mpi_io_params(data_size, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘wp_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:115:87:

  115 | routine s_setup_mpi_io_params(data_size, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:115:96:

  115 | _setup_mpi_io_params(data_size, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                         1

Error: Symbol ‘str_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:115:107:

  115 | etup_mpi_io_params(data_size, m_MOK, n_MOK, p_MOK, WP_MOK, MOK, str_MOK, NVARS_MOK)
      |                                                                                  1

Error: Symbol ‘nvars_mok’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:151:42:

  151 |         integer, dimension(MPI_STATUS_SIZE) :: status
      |                                          1
Error: Symbol ‘mpi_status_size’ at (1) has no IMPLICIT type
/home/mohammed/Desktop/verrou/src/post_process/m_data_input.f90:132:46:

  132 |         m_MOK = int(m_glb + 1, MPI_OFFSET_KIND)
      |                                              1
Error: Symbol ‘mpi_offset_kind’ at (1) has no IMPLICIT type

sbryngelson · 2025-07-12T14:45:21Z

this must have been --no-mpi with --debug. can you figure out what the error is really tied to? code-wise.

Malmahrouqi3 · 2025-07-12T16:09:27Z

I guess the --no-mpi flag is glitching out. I just tried to clone and build with it on Phoenix and it showed the same errors. Those variables should not be present/used. A recent change in m_data_input.f90 could have caused this.

Malmahrouqi3 · 2025-07-12T16:29:13Z

Previous commits of m_data_input.f90 build just fine. This PR might be the issue.

sbryngelson · 2025-07-12T16:40:23Z

It likely is, then, indeed. I'd rather the PR be fixed rather than attempting to revert it, if possible. Unless the issue is so deep-rooted that fixing it makes the PR meaningless.

Co-authored-by: Ben Wilfong <48168887+wilfonba@users.noreply.github.com>

refac inputs

845a7a1

Copilot AI review requested due to automatic review settings June 22, 2025 04:12

sbryngelson requested a review from a team as a code owner June 22, 2025 04:12

This comment was marked as outdated.

Sign in to view

qodo-merge-pro bot added the Review effort 3/5 label Jun 22, 2025

sbryngelson requested a review from wilfonba June 25, 2025 01:15

wilfonba added 2 commits June 27, 2025 15:37

Merge branch 'master' into refac-data

4281144

bug fix

91a6518

sbryngelson enabled auto-merge (squash) June 28, 2025 22:40

Merge branch 'master' into refac-data

37ac613

sbryngelson removed request for a team and wilfonba June 28, 2025 22:44

sbryngelson self-assigned this Jun 28, 2025

sbryngelson requested a review from Copilot June 28, 2025 22:52

sbryngelson disabled auto-merge June 28, 2025 22:52

sbryngelson merged commit c0327c8 into MFlowCode:master Jun 28, 2025
27 of 28 checks passed

Copilot AI reviewed Jun 28, 2025

View reviewed changes

sbryngelson deleted the refac-data branch July 6, 2025 13:33

Malmahrouqi3 mentioned this pull request Jul 12, 2025

Bug fix in m_data_input.f90 #938

Merged

prathi-wind pushed a commit to prathi-wind/MFC-prathi that referenced this pull request Jul 13, 2025

Refac data input - Post process (MFlowCode#902)

6cd5cb3

Co-authored-by: Ben Wilfong <48168887+wilfonba@users.noreply.github.com>

		file_loc = trim(t_step_ib_dir)//'/.'
		call my_inquire(file_loc, dir_check)

Refac data input - Post process #902

Refac data input - Post process #902

Uh oh!

Conversation

sbryngelson commented Jun 22, 2025 • edited by qodo-merge-pro bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User description

PR Type

Description

Changes walkthrough 📝

Uh oh!

This comment was marked as outdated.

Uh oh!

qodo-merge-pro bot commented Jun 22, 2025

PR Reviewer Guide 🔍

Uh oh!

qodo-merge-pro bot commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Code Suggestions ✨

Uh oh!

codecov bot commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sbryngelson commented Jun 25, 2025

Uh oh!

wilfonba commented Jun 26, 2025

Uh oh!

sbryngelson commented Jun 26, 2025

Uh oh!

wilfonba commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Serial I/O

Parallel I/O

Uh oh!

sbryngelson commented Jun 27, 2025

Uh oh!

wilfonba commented Jun 27, 2025

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

Malmahrouqi3 commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbryngelson commented Jul 12, 2025

Uh oh!

Malmahrouqi3 commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Malmahrouqi3 commented Jul 12, 2025

Uh oh!

sbryngelson commented Jul 12, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

sbryngelson commented Jun 22, 2025 •

edited by qodo-merge-pro bot

Loading

qodo-merge-pro bot commented Jun 22, 2025 •

edited

Loading

codecov bot commented Jun 22, 2025 •

edited

Loading

wilfonba commented Jun 27, 2025 •

edited

Loading

Malmahrouqi3 commented Jul 12, 2025 •

edited

Loading

Malmahrouqi3 commented Jul 12, 2025 •

edited

Loading