Add Queue Override Support for DAG Enqueue Operations

Problem

Queue names are hardcoded in DAG YAML files. When reprocessing data with different priorities (e.g., realtime vs historical), the same high-priority queue is always used, making it impossible to dynamically assign DAGs to different queues based on processing requirements.

Solution

Add optional parameter to override DAG-level queue definition during enqueue operations, allowing users to dynamically assign DAGs to different queues without modifying the DAG files.

Changes

1. CLI Command Enhancement

• Added --queue flag to enqueue CLI command
  • New flag: --queue (shorthand: -u) to override DAG-level queue definition
  • Updated help documentation to include the new flag
  • Example: dagu enqueue --queue="realtime" my_dag.yaml

2. API Endpoint Enhancement

• Added queue parameter to enqueue API endpoints
  • Updated OpenAPI schema (api/v2/api.yaml) to include optional queue field
  • Regenerated Go and TypeScript API types
  • Example request body:

    {
      "queue": "realtime",
      "params": "PRIORITY=high",
      "dagRunId": "optional-run-id"
    }

3. Queue Resolution Logic Update

• Enhanced queue resolution to use override if provided
  • Added Queue field to DAGRunStatus to persist queue override
  • Updated enqueue logic to store queue override in DAG run status
  • Modified queue API to prioritize stored queue override over DAG definition
  • Ensures queued items appear in the correct queue based on the override used

4. Data Model Changes

• Extended DAGRunStatus structure
  • Added Queue string field to store queue override
  • Added WithQueue(queue string) StatusOption for setting queue
  • Updated EnqueueOptions to include Queue field

5. Backend Implementation

• Updated core enqueue functionality
  • Modified internal/cmd/enqueue.go to handle queue override
  • Updated internal/dagrun/manager.go to pass queue override to CLI
  • Enhanced internal/frontend/api/v2/dags.go to process queue override
  • Improved internal/frontend/api/v2/queues.go to use stored queue override

Technical Details

Queue Resolution Priority

1. Stored queue override (from DAG run status) - highest priority
2. DAG definition queue (from YAML file) - fallback
3. DAG name (default) - final fallback

Backward Compatibility

• All existing DAGs continue to work without changes
• Queue override is optional - if not provided, uses DAG's default queue
• No breaking changes to existing APIs or CLI commands

Usage Examples

CLI Usage

# Override queue for high-priority processing
dagu enqueue --queue="realtime" data_processor.yaml

# Override queue for low-priority processing  
dagu enqueue --queue="historical" data_processor.yaml

# Use DAG's default queue
dagu enqueue data_processor.yaml

API Usage

# High-priority processing
curl -X POST "http://localhost:8080/api/v2/dags/data_processor/enqueue" \
  -H "Content-Type: application/json" \
  -d '{"queue": "realtime", "params": "PRIORITY=high"}'

# Low-priority processing
curl -X POST "http://localhost:8080/api/v2/dags/data_processor/enqueue" \
  -H "Content-Type: application/json" \
  -d '{"queue": "historical", "params": "PRIORITY=low"}'

Benefits

1. Flexible Queue Management: Users can override queue assignments without modifying DAG files
2. Priority-based Processing: Different data processing priorities can use different queues
3. Improved Monitoring: Queues page shows separate queues for different processing types
4. Dynamic Assignment: Queue assignment can be determined at runtime based on context
5. Backward Compatible: Existing workflows continue to work unchanged

Testing

• ✅ All existing tests continue to pass
• ✅ Added comprehensive test coverage for queue override functionality
• ✅ Manual testing confirms separate queues are displayed correctly
• ✅ API and CLI interfaces work as expected

Files Modified

Core Implementation

• internal/models/status.go - Added Queue field to DAGRunStatus
• internal/cmd/enqueue.go - Added queue override handling
• internal/cmd/start.go - Updated enqueueDAGRun call
• internal/cmd/flags.go - Added queue flag definition
• internal/dagrun/manager.go - Enhanced EnqueueOptions and EnqueueDAGRun

API Layer

• api/v2/api.yaml - Added queue parameter to enqueue endpoint
• internal/frontend/api/v2/dags.go - Updated API enqueue implementation
• internal/frontend/api/v2/queues.go - Enhanced queue resolution logic

Testing

• internal/cmd/enqueue_test.go - Added test case for queue override

Generated Files

• api/v2/api.gen.go - Regenerated API types
• ui/src/api/v2/schema.ts - Regenerated TypeScript types

Migration Notes

No migration is required. This is a purely additive feature that maintains full backward compatibility.

Future Enhancements

Potential future improvements could include:

• Queue override support for scheduled DAGs
• Queue priority levels within the same queue
• Queue-based resource allocation and limits
• Queue override validation against available queues