Operation Dark Water ⚓️ 🕵️‍♀️

📌 Overview

Operation Dark Water is a forensic geospatial data engineering pipeline designed to detect "Dark Vessels" ships that intentionally disable their AIS (Automatic Identification System) transponders to hide illegal activities like smuggling, illegal fishing, or unauthorized ship-to-ship transfers (rendezvous).

By analyzing large scale AIS datasets using Apache Spark and Sedona, this project identifies gaps in vessel tracking and flags high probability rendezvous events based on proximity and duration.

Key Features:

• Scalable Ingestion: Efficiently processes raw CSV AIS data into Delta Lake.
• Geospatial Forensics: Uses Apache Sedona for complex spatial joins and gap analysis.
• Hexagonal Indexing: Leverages Uber H3 for high-performance spatial partitioning and rendezvous detection.
• Delta Lake (Lakehouse): Ensures ACID transactions and time-travel capabilities for auditing vessel behavior.

---

🏗 Architecture

The pipeline follows a multi-hop (Medallion) architecture:

1. Bronze (Raw): Ingest raw AIS CSV data from NOAA into Delta tables.
2. Silver (Enriched): Perform gap detection using Apache Sedona to identify where and when a vessel "went dark."
3. Gold (Analytics): Identify potential encounters between dark vessels and other ships using H3 hexagonal indexing and proximity logic.
4. Visualize: Generate an interactive Kepler.gl map to visualize suspicious vessel tracks.

🛠 Tech Stack

• Compute: Apache Spark 3.5 (Scala 2.12)
• Spatial Engines: Apache Sedona, Uber H3
• Lakehouse: Delta Lake
• Storage: MinIO (S3-Compatible)
• Build Tool: sbt
• Infrastructure: Docker & Docker Compose

---

🚀 How to Run

Prerequisites

• Docker Desktop installed and running.
• Java 11+ and sbt (if you want to build the Scala jar locally).

1. Build the Project

Scala projects use sbt for building. We use the sbt-assembly plugin to create a "Fat Jar" containing all our code and dependencies.

sbt clean assembly

This creates the jar at: target/scala-2.12/OperationDarkWater-assembly-0.1.0.jar

2. Start Infrastructure

Start the Spark cluster and MinIO storage:

make up
# OR
docker-compose -f infrastructure/docker-compose.yml up -d

3. Run the Pipeline

Note: If you are on Windows (PowerShell), use backticks (`). On Linux/Mac (Bash), use backslashes (\).

A. Ingestion (Raw CSV -> Delta)

make ingest (for convenience) or run:

PowerShell:

docker exec darkwater-spark-master spark-submit `
  --class com.darkwater.ingest.AisIngestion `
  --master spark://spark-master:7077 `
  /opt/bitnami/spark/work-dir/target/scala-2.12/OperationDarkWater-assembly-0.1.0.jar

Bash:

docker exec darkwater-spark-master spark-submit \
  --class com.darkwater.ingest.AisIngestion \
  --master spark://spark-master:7077 \
  /opt/bitnami/spark/work-dir/target/scala-2.12/OperationDarkWater-assembly-0.1.0.jar

B. Analysis (Gap Detection)

make analyze (for convenience) or run:

PowerShell:

docker exec darkwater-spark-master spark-submit `
  --class com.darkwater.analysis.GapDetector `
  --master spark://spark-master:7077 `
  /opt/bitnami/spark/work-dir/target/scala-2.12/OperationDarkWater-assembly-0.1.0.jar

Bash:

docker exec darkwater-spark-master spark-submit \
  --class com.darkwater.analysis.GapDetector \
  --master spark://spark-master:7077 \
  /opt/bitnami/spark/work-dir/target/scala-2.12/OperationDarkWater-assembly-0.1.0.jar

C. Analytics (Rendezvous Finder)

Already covered in make analyze (for convenience) or run:

PowerShell:

docker exec darkwater-spark-master spark-submit `
  --class com.darkwater.analysis.RendezvousFinder `
  --master spark://spark-master:7077 `
  /opt/bitnami/spark/work-dir/target/scala-2.12/OperationDarkWater-assembly-0.1.0.jar

Bash:

docker exec darkwater-spark-master spark-submit \
  --class com.darkwater.analysis.RendezvousFinder \
  --master spark://spark-master:7077 \
  /opt/bitnami/spark/work-dir/target/scala-2.12/OperationDarkWater-assembly-0.1.0.jar

---

🛠 Simple Local Testing (Recommended for Dev)

If you don't want to deal with Spark Cluster submissions or if your containers are crashing, you can run Spark in Local Mode directly from your terminal. This is much faster for development.

1. Ensure MinIO is running

You still need the storage layer.

docker-compose up -d minio

2. Run directly via sbt

This runs the Scala code on your machine but connects to the MinIO in Docker.

# Run Ingestion
sbt "runMain com.darkwater.ingest.AisIngestion"

# Run Gap Detector
sbt "runMain com.darkwater.analysis.GapDetector"

Note: This uses your local CPU cores as a "mini Spark cluster".

---

✅ Verify Success

• Spark Master UI: Open http://localhost:8080. You should see the finished applications in the "Completed Applications" list.

• MinIO (Data Lake): Go to http://localhost:9000 (User/Pass: minioadmin).

  • Check the data bucket for ais_bronze, ais_silver, and rendezvous_gold folders.

• Visualization: It is recommended to use a virtual environment to avoid dependency conflicts.

  PowerShell (Windows):

  python -m venv venv
  .\venv\Scripts\Activate.ps1
  pip install -r src/main/python/requirements.txt
  python src/main/python/visualizing/generate_kepler_map.py

  Bash (Linux/Mac):

  python3 -m venv venv
  source venv/bin/activate
  pip install -r src/main/python/requirements.txt
  python src/main/python/visualizing/generate_kepler_map.py

  Open the generated kepler_map.html to see the results.

---

📊 Insights & Visuals

Pipeline Status (Spark UI)

Track the progress of ingestion and spatial analytics.

Data Lake Storage (MinIO)

Verify processed Delta tables in the Silver and Gold buckets.

Geospatial Visualization (Kepler.gl)

Interactive map showing Country Map with All Points Layers. Interactive map showing Dark Gaps and Potential Rendezvous.

Analytics Results (Potential Rendezvous)

Sample output of the hexagonal proximity join identifying suspicious ship-to-ship encounters.

---

💡 Developer Knowledge (sbt & Makefile)

Why sbt?

Think of sbt (Scala Build Tool) as the Scala equivalent of Maven or Gradle. It handles dependency resolution and compilation.

• build.sbt: The main configuration file.
• sbt assembly: Unlike a standard sbt package, this packages all library dependencies (Delta, Sedona, H3) into a single jar. This is critical for Spark because the executors need all libraries available locally.

Why the Makefile?

Submitting Spark jobs involves long docker exec commands with many flags. The Makefile abstracts this away:

• make up: Starts everything.
• make ingest: Triggers the first stage.
• make analyze: Triggers the spatial logic.

__init__.py in Visualization

The src/main/python/visualizing/__init__.py file is kept as a "marker" to ensure Python recognizes the folder as a package, which is especially important when mounting volumes in Docker environments.

---

🛠 Troubleshooting

1. sbt Error: "Not found org.apache.sedona"

If you get a ResolveException for Sedona 1.5.0, it's likely because that specific version isn't in Maven Central for Spark 3.5.

• Fix: Check build.sbt and ensure sedonaVersion is set to 1.5.1 or higher.

2. "Container ... is not running"

If docker exec fails with this error:

• Cause A: You haven't started the infrastructure. Run make up.
• Cause B: The container crashed due to insufficient memory. Spark requires at least 4GB of RAM allocated to Docker.
• Cause C: An error occurred during startup. Check logs: docker logs darkwater-spark-master.

3. "File not found: ...assembly-0.1.0.jar"

• Cause: You didn't build the project.
• Fix: Run sbt assembly. Verify the file exists at target/scala-2.12/OperationDarkWater-assembly-0.1.0.jar before running the spark-submit.

4. sbt Error: "Not a valid command: assembly"

• Cause: The sbt-assembly plugin is missing from the project configuration.
• Fix: I've created the project/plugins.sbt file with the following line:

  addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.1.1")

  Now you can run sbt assembly again.

5. sbt Error: "Connection timed out" or "Connection reset"

• Cause: Your internet connection had trouble downloading Spark/Sedona libraries from Maven Central.
• Fix: These are often temporary.
  1. Try running the command again. sbt will resume where it left off.
  2. If it persists, check your proxy/VPN settings or try a different network.
  3. You can try increasing the timeout by running: sbt -Dhttp.connectionTimeout=120000 clean assembly.

6. sbt Error: java.lang.OutOfMemoryError: Java heap space

• Cause: The sbt assembly process requires significant memory to merge multiple large Spark libraries into a single "Fat Jar".
• Fix: I've created a .sbtopts file in the project root with -J-Xmx4G to allocate 4GB of RAM to sbt. If you still encounter issues, you can run sbt with an explicit memory flag:

  sbt -J-Xmx8G clean assembly