containers/Containerfile.cuda (1)

2-2: Build is failing – base image requires authentication
nvcr.io/nvidia/cuda:12.9.1-base-ubuntu20.04 returns 401 Unauthorized in CI.
Options:

1. Switch to the public Docker-Hub mirror: nvidia/cuda:12.9.1-base-ubuntu20.04.
2. Add a docker/login-action step with an NGC API token (and set NVCR_TOKEN secret).
   Until one of these is done, the pipeline will remain red.

containers/Containerfile.cpu (2)

15-20: Combine build-time steps and set VIRTUAL_ENV for cleaner image layers

1. RUN uv sync, ENV PATH=…, and EXPOSE 80 can be squashed into a single RUN layer to reduce final image size.
2. Several Python tools rely on the VIRTUAL_ENV env-var. Declaring it avoids subtle path issues when invoking Dagster CLI inside the container.

-# Run uv sync
-RUN uv sync --extra dagster
-
-ENV PATH="/home/jim/.venv/bin:$PATH"
-# Expose the port that your Dagster instance will run on
-EXPOSE 80
+# Install deps & expose Dagster
+RUN uv sync --extra dagster \
+ && echo 'export VIRTUAL_ENV=/home/jim/.venv' >> /etc/profile.d/virtual_env.sh
+
+ENV VIRTUAL_ENV=/home/jim/.venv \
+    PATH="/home/jim/.venv/bin:$PATH"
+
+EXPOSE 80

---

9-12: Shallow-clone to speed up builds
Use --depth 1 --branch jim-dev to avoid fetching the full commit history and an extra checkout step.

-RUN git clone https://github.com/kazewong/jim.git
-
-WORKDIR /home/jim
-RUN git checkout jim-dev
+RUN git clone --depth 1 --branch jim-dev https://github.com/kazewong/jim.git /home/jim
+
+WORKDIR /home/jim

containers/Containerfile.cuda (2)

15-20: Mirror CPU-image improvements & set VIRTUAL_ENV
Apply the same layer consolidation and VIRTUAL_ENV export as suggested for the CPU image to keep both images consistent and slimmer.

---

1-1: Minor Hadolint warning – keyword casing
FROM ghcr.io/astral-sh/uv:python3.12-bookworm as uv-source → use AS (uppercase) to match FROM and silence DL4000.

-FROM ghcr.io/astral-sh/uv:python3.12-bookworm as uv-source
+FROM ghcr.io/astral-sh/uv:python3.12-bookworm AS uv-source

.github/workflows/CD.yml (2)

77-84: Consider building a CPU image as well
Renaming clarifies the CUDA build, but users on non-GPU nodes (e.g. local Kind clusters) will need a CPU-only image.
Add a second docker/build-push-action step (or a matrix) building containers/Containerfile.cpu and tagging it appropriately.

---

77-79: Step-id mismatch
id: push is now ambiguous because the step name mentions CUDA specifically. Renaming the id to push_cuda keeps logs self-describing and avoids future shadowing if a CPU step is added.

-      - name: Build and push Docker image for cuda
-        id: push
+      - name: Build and push CUDA Docker image
+        id: push_cuda

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

pipeline/dagster/RealDataCatalog/assets.py (2)

176-180: Critical inconsistency: Function still uses local filesystem after Minio migration.

The config_file function reads from the local "data/event_list.txt" file, but this data is now stored in Minio object storage. This will cause runtime failures since the local file no longer exists.

Update the function to read from Minio storage:

-    with open("data/event_list.txt", "r") as f:
-        lines = f.readlines()
-        event_dict = dict(line.strip().split() for line in lines)
+    # Download event list from Minio and read
+    import tempfile, os
+    with tempfile.NamedTemporaryFile(mode='w+', delete=False) as temp_file:
+        # Note: This function needs MinioResource parameter added to signature
+        minio.download_object("event_list.txt", temp_file.name)
+        with open(temp_file.name, 'r') as f:
+            lines = f.readlines()
+            event_dict = dict(line.strip().split() for line in lines)
+    os.unlink(temp_file.name)

Also, the function signature needs to be updated to include the MinioResource parameter:

-def config_file(context: AssetExecutionContext):
+def config_file(context: AssetExecutionContext, minio: MinioResource):

---

59-94: Suggest essential refactor: Migrate raw_data function to use Minio storage.

The raw_data function should be updated to store processed data in Minio object storage to maintain consistency with the storage strategy migration. Currently, it still saves data to local filesystem directories.

Consider updating the function to:

1. Store strain and PSD data in Minio with organized object naming (e.g., {event_name}/raw/{ifo}_data, {event_name}/raw/{ifo}_psd)
2. Use the existing Minio resource methods for object storage
3. Maintain the same error handling logic but adapt file operations to Minio operations

This will ensure the entire pipeline uses consistent object storage and works properly in containerized/K8s environments where local filesystem persistence is not guaranteed.

pipeline/dagster/RealDataCatalog/minio_resource.py (1)

27-30: Minor improvement: Add error handling and validation.

The put_object method works correctly but could benefit from parameter validation and error handling.

Consider adding validation:

 def put_object(self, object_name: str, data, size: int, content_type: str):
+    if not object_name or not object_name.strip():
+        raise ValueError("Object name cannot be empty")
+    if size < 0:
+        raise ValueError("Size cannot be negative")
     client = self.get_client()
-    
-    return client.put_object(self.bucket_name, object_name, data, size, content_type)
+    try:
+        return client.put_object(self.bucket_name, object_name, data, size, content_type)
+    except Exception as e:
+        raise RuntimeError(f"Failed to upload object {object_name}: {str(e)}") from e

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

pipeline/dagster/RealDataCatalog/assets.py (3)

201-218: Config file function has multiple filesystem dependencies.

This function has several issues:

1. Reads event_list.txt from local filesystem (line 203) instead of Minio
2. Checks for data/PSD files in local filesystem (lines 211-213) that are now in Minio

Update to use Minio:

-    with open("data/event_list.txt", "r") as f:
-        lines = f.readlines()
-        event_dict = dict(line.strip().split() for line in lines)
+    # Fetch from Minio
+    event_list_obj = minio.get_object("event_list.txt")
+    lines = event_list_obj.read().decode("utf-8").splitlines()
+    event_dict = dict(line.strip().split() for line in lines)

For checking file availability, you'll need to use Minio's object listing or head_object methods instead of os.path.exists.

---

311-668: All diagnostic plotting functions need Minio integration.

All the remaining diagnostic functions (loss_plot, production_chains_corner_plot, etc.) read from local results.npz files that may not exist if the pipeline is using Minio for storage.

Consider creating a helper function to handle Minio downloads for all diagnostic functions:

def download_results_from_minio(context: AssetExecutionContext, minio: MinioResource, event_name: str):
    """Download results.npz from Minio to a temporary location."""
    import tempfile
    temp_file = tempfile.NamedTemporaryFile(suffix='.npz', delete=False)
    try:
        minio.download_object(f"{event_name}/results.npz", temp_file.name)
        return temp_file.name
    except Exception as e:
        os.unlink(temp_file.name)
        raise FileNotFoundError(f"Results file not found in Minio: {e}")

This would simplify updating all diagnostic functions and ensure consistent error handling.

---

127-155: Update diagnostic assets to fetch data from Minio, not the local data/ directory

The plotting (raw_data_plot, psd_plot, etc.) and configuration (config_file) functions still assume files live under data/<event_name>/…, but upstream you’ve moved raw and PSD data into Minio. These functions will fail at runtime unless they first pull the required files from Minio.

Please update each diagnostic asset to:

• Inject the Minio resource via context.resources.minio.
• Download the needed file(s) (e.g. .npz, event_list.txt) into a temporary directory (or stream them) before loading.
• Clean up or close temporary files when done.

Affected locations:

• pipeline/dagster/RealDataCatalog/assets.py:
  • raw_data_plot (lines 127–155)
  • psd_plot (lines 156–191)
  • config_file (lines 201–263)
  • All other diagnostic plotting functions (lines 311–668)

Example diff for raw_data_plot:

 def raw_data_plot(context: AssetExecutionContext):
     """
     Plot the raw strain data for each IFO for the event.
     """
-    import matplotlib.pyplot as plt
+    import matplotlib.pyplot as plt
     import tempfile
     import os
     import numpy as np

     event_name = context.partition_key
-    event_dir = os.path.join("data", event_name, "raw")
+    # Download raw data from Minio
+    temp_dir = tempfile.mkdtemp()
+    minio = context.resources.minio
+
+    plots_dir = os.path.join("data", event_name, "plots")
     os.makedirs(plots_dir, exist_ok=True)

     ifos = ["H1", "L1", "V1"]
     plot_paths = []
     for ifo in ifos:
-        data_file = os.path.join(event_dir, f"{ifo}_data.npz")
+        local_npz = os.path.join(temp_dir, f"{ifo}_data.npz")
+        # pull from Minio bucket `<event_name>/raw/`
+        minio.download_object(
+            bucket_name=event_name,
+            object_name=f"raw/{ifo}_data.npz",
+            file_path=local_npz,
+        )
+        data_file = local_npz

         if os.path.exists(data_file):
             data = np.load(data_file)
             t = data["epoch"] + np.arange(data["td"].shape[0]) * data["dt"]
             td = data["td"]
             if t is not None and td is not None:
                 plt.figure()
                 plt.plot(t, td)
                 plt.xlabel("Time (s)")
                 plt.ylabel("Strain")
                 plt.title(f"{ifo} Strain for {event_name}")
                 plot_path = os.path.join(plots_dir, f"{ifo}_strain.png")
                 plt.savefig(plot_path)
                 plt.close()
                 plot_paths.append(plot_path)
     return plot_paths

Apply analogous changes to psd_plot, config_file, and all other diagnostics so that every file read is preceded by a Minio download.

pipeline/dagster/RealDataCatalog/assets.py (2)

59-65: Good fix! The function now correctly reads from Minio.

The implementation properly addresses the previous inconsistency by fetching the event list from Minio storage instead of the local filesystem.

---

164-191: PSD plot function also needs Minio integration.

Similar to raw_data_plot, this function expects local files but should download from Minio.

pipeline/dagster/RealDataCatalog/assets.py (1)

23-45: Well-implemented transition to Minio storage.

The function correctly uploads the event list to Minio instead of writing to local filesystem. The implementation properly handles string-to-bytes conversion and sets the appropriate content type.

Minor optimization: The buffer.seek(0) on line 38 is unnecessary since buffer.getvalue() returns the entire buffer contents regardless of the current position.

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

pipeline/dagster/RealDataCatalog/assets.py (2)

70-71: Consider using tempfile module for better cross-platform compatibility.

The hardcoded 'tmp' directory could cause issues on different platforms. Consider using Python's tempfile.mkdtemp() for better portability.

This issue was previously flagged and remains unaddressed. The current implementation still uses hardcoded paths which may not work consistently across different environments.

---

117-119: Improve exception handling and cleanup strategy.

The current approach has two issues:

1. Exception handling is too broad and may mask specific errors
2. Cleanup only removes .npz files but leaves directory structure

This issue was previously flagged and remains unaddressed. Consider implementing proper cleanup with shutil.rmtree() and more specific exception handling.

pipeline/dagster/RealDataCatalog/assets.py (3)

11-11: Fix typo in comment.

There's a typo in the comment: "configuration0" should be "configuration".

-# Create asset group for run and configuration0
+# Create asset group for run and configuration

---

317-352: Placeholder asset - implementation needed.

The run multi-asset is properly structured with correct dependencies and partitions, but currently unimplemented. This appears to be intentional for the current PR scope.

Would you like me to help implement the logic for loading results from the external run execution?

---

355-770: Consider refactoring to reduce code duplication across diagnostic assets.

The diagnostic assets follow very similar patterns for:

• Loading data from Minio
• Creating plots with matplotlib/corner
• Saving to temporary files
• Uploading to Minio
• Cleaning up temporary files

Consider creating helper functions to reduce duplication:

def create_and_upload_plot(event_name: str, minio: MinioResource, plot_func, plot_name: str, **kwargs):
    """Helper function to create plots and upload to Minio"""
    with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmpfile:
        fig = plot_func(**kwargs)
        fig.savefig(tmpfile.name)
        plt.close(fig)
        tmpfile.flush()
        tmpfile.seek(0)
        tmpfile_size = os.path.getsize(tmpfile.name)
        minio_plot_path = f"{event_name}/plots/{plot_name}.png"
        with open(tmpfile.name, "rb") as plotfile:
            minio.put_object(
                object_name=minio_plot_path,
                data=plotfile,
                size=tmpfile_size,
                content_type="image/png",
            )
    os.remove(tmpfile.name)
    return minio_plot_path

def load_results_from_minio(event_name: str, minio: MinioResource):
    """Helper function to load results.npz from Minio"""
    results_obj = minio.get_object(f"{event_name}/results.npz")
    return np.load(io.BytesIO(results_obj.read()), allow_pickle=True)

This would significantly reduce code duplication and improve maintainability.

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

pipeline/dagster/RealDataCatalog/assets.py (2)

70-71: Use Python's tempfile module for temporary directory management.

Instead of hardcoding 'tmp' directory, use Python's tempfile.mkdtemp() for better cross-platform compatibility and automatic cleanup.

-    # Use a temp directory, but keep event_name and "raw" part
-    event_dir = os.path.join('tmp', event_name, "raw")
+    import tempfile
+    temp_dir = tempfile.mkdtemp()
+    event_dir = os.path.join(temp_dir, event_name, "raw")

Remember to clean up the entire temporary directory at the end of the function or use a context manager.

---

115-119: Incomplete cleanup and broad exception handling.

Two issues to address:

1. The cleanup only removes .npz files but leaves the directory structure
2. The exception handling is too broad and doesn't properly clean up on error

For proper cleanup, consider:

import shutil
# At the end of the function (outside the try block)
if 'temp_dir' in locals():
    shutil.rmtree(temp_dir)

For better error handling:

-        except Exception as e:
-            print(f"Error fetching data for {ifo} during {event_name}: {e}")
-            continue
+        except gwosc.api.APIError as e:
+            context.log.error(f"GWOSC API error for {ifo} during {event_name}: {e}")
+            continue
+        except Exception as e:
+            context.log.error(f"Unexpected error for {ifo} during {event_name}: {e}")
+            # Clean up any partial files
+            for f in [data_file_path + '.npz', psd_file_path + '.npz']:
+                if os.path.exists(f):
+                    os.remove(f)
+            raise

pipeline/dagster/RealDataCatalog/assets.py (3)

11-11: Fix typo in comment.

There's a typo "configuration0" instead of "configuration".

-# Create asset group for run and configuration0
+# Create asset group for run and configuration

---

348-352: Implement the run asset to load results.

The run asset is currently a placeholder. This asset should load the results from execute_single_run.py and yield the specified assets.

Would you like me to help implement this asset to load results from the results.npz file and yield each asset appropriately?

---

412-427: Consider extracting parameter key lists to constants.

The parameter key lists are duplicated across multiple diagnostic assets. Consider extracting them to module-level constants for better maintainability.

# At module level
PARAMETER_KEYS = [
    "M_c", "q", "s1_mag", "s1_theta", "s1_phi",
    "s2_mag", "s2_theta", "s2_phi", "iota", "d_L",
    "phase_c", "psi", "ra", "dec"
]

Then use PARAMETER_KEYS in place of the hardcoded lists.

Also applies to: 464-479, 516-531, 567-582

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro