GPU Nextflow Workflow on Google Cloud

This is a minimal Nextflow workflow to test GPU support on Google Cloud Platform (GCP).

Prerequisites

1. Google Cloud Account with appropriate permissions
2. Nextflow installed (version 22.10.0 or higher)
3. Google Cloud SDK installed and configured
4. Appropriate GCP permissions:
   • compute.instances.create
   • compute.instances.delete
   • compute.disks.create
   • compute.networks.get
   • storage.buckets.create (for intermediate files)

Setup Instructions

Configure Google Cloud Credentials

gcloud auth application-default login

Running the Workflow

Option 1: Run with Google Cloud (GPU enabled)

nextflow run main.nf -profile gcp_gpu

Option 2: Run locally for testing (no GPU)

nextflow run main.nf -profile local

Configuration Details

GPU Options Available

The nextflow.config file supports the following GPU types:

• nvidia-tesla-t4 - Good for inference and light ML workloads (recommended for cost)
• nvidia-tesla-v100 - Better for training workloads
• nvidia-tesla-p100 - High-performance computing
• nvidia-tesla-p4 - Cost-effective option

Changing GPU Type

Edit nextflow.config and modify the gpu_task process:

withLabel: gpu {
    gpuType = 'nvidia-tesla-v100'  // Change to your preferred GPU
    machineType = 'n1-highmem-8'   // Adjust machine type if needed
}

Machine Type Recommendations

• T4 GPU: Use n1-highmem-4 or n1-standard-4
• V100 GPU: Use n1-highmem-8 or larger
• P100/P4 GPU: Use n1-highmem-8 or larger

Monitoring

After running the workflow, check the generated reports:

• timeline.html - Visual timeline of task execution
• report.html - Detailed execution report
• trace.txt - Raw execution trace

Common Issues and Solutions

Issue: "Invalid value for '--accelerator': Must be between 1 and 8"

Solution: Adjust GPU count in process { withLabel: gpu }

Issue: "GPU type not available in zone"

Solution: Change zone in nextflow.config or try a different GPU type

Issue: "Quota exceeded"

Solution: Check your GCP quota limits or request quota increase

Next Steps

1. Modify gpu_task process in main.nf to run your actual GPU workload
2. Install required software in the container (see example below)
3. Test with your own data

Example: Adding CUDA/PyTorch to GPU Task

Modify the gpu_task process:

process gpu_task {
    label 'gpu'
    container 'nvidia/cuda:11.8.0-runtime-ubuntu22.04'
    
    // ... rest of process
}

Cost Estimation

T4 GPU pricing on GCP (varies by region):

• Compute instance: ~$0.35/hour
• T4 GPU: ~$0.35/hour
• Total: ~$0.70/hour for this minimal setup

Always monitor your resources to avoid unexpected charges!

Resources

• Nextflow Google Cloud Documentation
• Google Life Sciences Executor
• Google Cloud GPU Quotas