• .idea/.gitignore
• .idea/SurfSense.iml
• .idea/inspectionProfiles/profiles_settings.xml
• .idea/modules.xml
• .idea/vcs.xml
• surfsense_backend/alembic/versions/23_add_connector_schedules_table.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/scheduler_routes.py
• surfsense_backend/app/schemas/__init__.py
• surfsense_backend/app/schemas/connector_schedule.py
• surfsense_backend/app/services/connector_scheduler_service.py
• surfsense_web/app/dashboard/[search_space_id]/connectors/schedules/page.tsx

.idea/SurfSense.iml (1)

1-8: Remove IDE artifacts from version control.

This IntelliJ module file is environment-specific, churns often, and doesn’t contribute to the backend/frontend feature set. Please drop it (and the other .idea XML files) from the repo and rely on .gitignore to keep the directory untracked.

surfsense_backend/alembic/versions/23_add_connector_schedules_table.py (1)

61-78: Indexes: add composite (is_active, next_run_at); consider dropping redundant id index.

• Scheduler likely queries WHERE is_active = TRUE AND next_run_at <= now(); a composite index helps.
• The explicit id index duplicates the PK index; remove if not needed.

Apply this diff to add indexes:

     if "ix_connector_schedules_id" not in existing_indexes:
-        op.create_index("ix_connector_schedules_id", "connector_schedules", ["id"])
+        # Optional: id already indexed by PK; consider skipping this extra index.
+        pass
@@
     if "ix_connector_schedules_next_run_at" not in existing_indexes:
         op.create_index(
             "ix_connector_schedules_next_run_at", "connector_schedules", ["next_run_at"]
         )
+    if "ix_connector_schedules_search_space_id" not in existing_indexes:
+        op.create_index(
+            "ix_connector_schedules_search_space_id",
+            "connector_schedules",
+            ["search_space_id"],
+        )
+    if "ix_connector_schedules_is_active_next_run_at" not in existing_indexes:
+        op.create_index(
+            "ix_connector_schedules_is_active_next_run_at",
+            "connector_schedules",
+            ["is_active", "next_run_at"],
+        )

And include drops in downgrade:

-    op.drop_index("ix_connector_schedules_next_run_at", table_name="connector_schedules")
+    op.drop_index("ix_connector_schedules_is_active_next_run_at", table_name="connector_schedules")
+    op.drop_index("ix_connector_schedules_search_space_id", table_name="connector_schedules")
+    op.drop_index("ix_connector_schedules_next_run_at", table_name="connector_schedules")
-    op.drop_index("ix_connector_schedules_id", table_name="connector_schedules")
+    # If created earlier; otherwise safe to skip or guard with IF EXISTS pattern
+    # op.drop_index("ix_connector_schedules_id", table_name="connector_schedules")

surfsense_backend/app/app.py (2)

31-33: Async task is unnecessary; just await start_scheduler().

start_scheduler completes quickly; creating/canceling a task adds noise. Prefer awaiting directly and drop cancel block.

Apply this diff:

-    scheduler_task = asyncio.create_task(start_scheduler())
-    logger.info("Connector scheduler service started")
+    await start_scheduler()
+    logger.info("Connector scheduler service started")

Then remove the later cancel section (Lines 43-49).

---

26-29: Avoid create_all in production; prefer Alembic migrations.

Running create_db_and_tables() alongside Alembic can cause drift. Gate behind a dev flag or remove in prod environments.

surfsense_backend/app/utils/schedule_helpers.py (2)

37-41: Optional: clamp hourly_minute defensively.

Schemas validate range, but add a local guard to avoid surprises if called directly.

Apply this diff:

-        minute = hourly_minute if hourly_minute is not None else 0
+        minute = hourly_minute if hourly_minute is not None else 0
+        if minute < 0 or minute > 59:
+            raise ValueError("hourly_minute must be between 0 and 59")

---

51-64: Weekly scheduling logic: clarify default timezone semantics.

If daily/weekly times represent local-wall times, consider storing a timezone per search space or using config default TZ; current UTC assumption may surprise users across DST.

surfsense_backend/app/schemas/connector_schedule.py (1)

88-91: Consider adding optional time fields to Update schema

Allow updating daily/weekly/hourly options if persisted in DB; otherwise clients can’t change them post‑create.

 class ConnectorScheduleUpdate(BaseModel):
@@
-    schedule_type: ScheduleType | None = None
-    cron_expression: str | None = None
-    is_active: bool | None = None
+    schedule_type: ScheduleType | None = None
+    cron_expression: str | None = None
+    is_active: bool | None = None
+    daily_time: time | None = None
+    weekly_day: int | None = None
+    weekly_time: time | None = None
+    hourly_minute: int | None = None

Also applies to: 94-100

surfsense_backend/app/routes/scheduler_routes.py (1)

95-106: Bound the limit parameter to avoid heavy queries

Add simple clamping/validation (e.g., 1..100) to prevent abuse.

-async def get_upcoming_schedules(
-    limit: int = 10,
+async def get_upcoming_schedules(
+    limit: int = 10,
@@
-    try:
+    try:
+        limit = max(1, min(limit, 100))
@@
-async def get_recent_schedule_executions(
-    limit: int = 20,
+async def get_recent_schedule_executions(
+    limit: int = 20,
@@
-    try:
+    try:
+        limit = max(1, min(limit, 100))

Also applies to: 152-163

surfsense_backend/app/routes/connector_schedules_routes.py (1)

41-45: Frontend relies on filtering by search_space_id; ensure API supports it

The UI calls GET /connector-schedules/?search_space_id=... which your code supports. Consider also adding a similar filter to /search-source-connectors to simplify client-side filtering.

Also applies to: 118-121

surfsense_backend/app/services/connector_scheduler_service.py (2)

289-303: Recalculate next_run with full time options if persisted

Only schedule_type/cron are passed, ignoring daily/weekly/hourly options. If these fields exist in DB, include them.

-        next_run = calculate_next_run(
-            schedule.schedule_type, schedule.cron_expression
-        )
+        next_run = calculate_next_run(
+            schedule.schedule_type,
+            schedule.cron_expression,
+            getattr(schedule, "daily_time", None),
+            getattr(schedule, "weekly_day", None),
+            getattr(schedule, "weekly_time", None),
+            getattr(schedule, "hourly_minute", None),
+        )

---

366-370: start_scheduler awaits an infinite loop; ensure it’s spawned as a background task

If start_scheduler() is awaited directly in app startup, it will block the app. Spawn with asyncio.create_task(start_scheduler()) in lifespan/startup hooks.

Where is start_scheduler() invoked? If it’s awaited, switch to creating a task.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

• README.md
• docs/chinese-llm-setup.md
• surfsense_backend/alembic/env.py
• surfsense_backend/alembic/versions/23_associate_connectors_with_search_spaces.py
• surfsense_backend/alembic/versions/24_fix_null_chat_types.py
• surfsense_backend/alembic/versions/25_migrate_llm_configs_to_search_spaces.py
• surfsense_backend/alembic/versions/26_add_language_column_to_llm_configs.py
• surfsense_backend/alembic/versions/27_add_searxng_connector_enum.py
• surfsense_backend/alembic/versions/28_add_chinese_litellmprovider_enum.py
• surfsense_backend/app/agents/podcaster/configuration.py
• surfsense_backend/app/agents/podcaster/nodes.py
• surfsense_backend/app/agents/researcher/configuration.py
• surfsense_backend/app/agents/researcher/nodes.py
• surfsense_backend/app/agents/researcher/prompts.py
• surfsense_backend/app/agents/researcher/qna_agent/configuration.py
• surfsense_backend/app/agents/researcher/qna_agent/nodes.py
• surfsense_backend/app/agents/researcher/qna_agent/prompts.py
• surfsense_backend/app/agents/researcher/sub_section_writer/nodes.py
• surfsense_backend/app/agents/researcher/sub_section_writer/prompts.py
• surfsense_backend/app/connectors/google_calendar_connector.py
• surfsense_backend/app/connectors/google_gmail_connector.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/airtable_add_connector_route.py
• surfsense_backend/app/routes/chats_routes.py
• surfsense_backend/app/routes/documents_routes.py
• surfsense_backend/app/routes/google_calendar_add_connector_route.py
• surfsense_backend/app/routes/google_gmail_add_connector_route.py
• surfsense_backend/app/routes/llm_config_routes.py
• surfsense_backend/app/routes/luma_add_connector_route.py
• surfsense_backend/app/routes/search_source_connectors_routes.py
• surfsense_backend/app/schemas/llm_config.py
• surfsense_backend/app/schemas/search_source_connector.py
• surfsense_backend/app/services/connector_service.py
• surfsense_backend/app/services/llm_service.py
• surfsense_backend/app/services/query_service.py
• surfsense_backend/app/services/task_logging_service.py
• surfsense_backend/app/tasks/connector_indexers/airtable_indexer.py
• surfsense_backend/app/tasks/connector_indexers/clickup_indexer.py
• surfsense_backend/app/tasks/connector_indexers/confluence_indexer.py
• surfsense_backend/app/tasks/connector_indexers/discord_indexer.py
• surfsense_backend/app/tasks/connector_indexers/github_indexer.py
• surfsense_backend/app/tasks/connector_indexers/google_calendar_indexer.py
• surfsense_backend/app/tasks/connector_indexers/google_gmail_indexer.py
• surfsense_backend/app/tasks/connector_indexers/jira_indexer.py
• surfsense_backend/app/tasks/connector_indexers/linear_indexer.py
• surfsense_backend/app/tasks/connector_indexers/luma_indexer.py
• surfsense_backend/app/tasks/connector_indexers/notion_indexer.py
• surfsense_backend/app/tasks/document_processors/extension_processor.py
• surfsense_backend/app/tasks/document_processors/file_processors.py
• surfsense_backend/app/tasks/document_processors/markdown_processor.py

surfsense_backend/app/services/connector_scheduler_service.py (1)

127-145: Fix invalid datetime timezone usage.

Line 129 uses datetime.now(datetime.utc) which is incorrect. datetime.utc is a constant, not a timezone object, and will raise an AttributeError at runtime.

Apply this diff:

     async def _get_due_schedules(self, session: AsyncSession) -> List[ConnectorSchedule]:
         """Get all schedules that are due for execution."""
-        now = datetime.now(datetime.utc)
+        now = datetime.now(timezone.utc)

surfsense_backend/app/routes/connector_schedules_routes.py (2)

90-98: Consider skipping next_run calculation when creating inactive schedules.

Currently, next_run_at is calculated even when is_active is False. While the scheduler ignores inactive schedules, you could optimize by setting next_run_at = None for inactive schedules to avoid unnecessary computation.

Apply this diff:

-        # Calculate next run time
-        next_run_at = calculate_next_run(
+        # Calculate next run time (skip if inactive)
+        next_run_at = None if not schedule.is_active else calculate_next_run(
             schedule.schedule_type, 
             schedule.cron_expression,
             schedule.daily_time,
             schedule.weekly_day,
             schedule.weekly_time,
             schedule.hourly_minute
         )

---

327-363: Recalculate next_run_at when activating schedules via toggle.

When toggling a schedule from inactive to active, next_run_at may be stale (if it was set before deactivation). The scheduler will handle this by running it immediately, but it's cleaner to recalculate the next run time when activating.

Similarly, consider clearing next_run_at when deactivating for cleaner state.

Apply this diff:

         # Toggle the active status
         schedule.is_active = not schedule.is_active
+        
+        # Recalculate or clear next_run_at based on new state
+        if schedule.is_active:
+            # Recalculate next run time when activating
+            schedule.next_run_at = calculate_next_run(
+                schedule.schedule_type,
+                schedule.cron_expression,
+                schedule.daily_time,
+                schedule.weekly_day,
+                schedule.weekly_time,
+                schedule.hourly_minute,
+            )
+        else:
+            # Clear next run time when deactivating
+            schedule.next_run_at = None
+        
         await session.commit()
         await session.refresh(schedule)

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

surfsense_backend/app/routes/connector_schedules_routes.py (1)

353-359: Recalculate/clear next_run_at on toggle.

On deactivate: set next_run_at=None. On activate: compute next run and ensure tz-aware UTC.

Apply:

         # Toggle the active status
         schedule.is_active = not schedule.is_active
+        if schedule.is_active:
+            nr = calculate_next_run(
+                schedule.schedule_type,
+                schedule.cron_expression,
+                schedule.daily_time,
+                schedule.weekly_day,
+                schedule.weekly_time,
+                schedule.hourly_minute,
+            )
+            if nr.tzinfo is None:
+                nr = nr.replace(tzinfo=timezone.utc)
+            schedule.next_run_at = nr
+        else:
+            schedule.next_run_at = None
         await session.commit()

Also applies to: 355-357

surfsense_backend/app/routes/scheduler_routes.py (1)

63-68: Tighten log query filter.

• Add Log.source == "connector_scheduler" to avoid capturing unrelated messages.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

surfsense_backend/app/routes/connector_schedules_routes.py (1)

164-179: Consider eager-loading relationships to avoid N+1 queries.

If callers access the connector or search_space relationships from the returned schedules, each access would trigger a separate query. You can optimize this by eager-loading:

         query = (
             select(ConnectorSchedule)
             .join(SearchSourceConnector)
             .filter(SearchSourceConnector.user_id == user.id)
+            .options(
+                selectinload(ConnectorSchedule.connector),
+                selectinload(ConnectorSchedule.search_space)
+            )
         )

Don't forget to import selectinload:

from sqlalchemy.orm import selectinload

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

surfsense_backend/app/services/stt_service.py (2)

25-35: Consider error handling for model initialization failures.

The lazy loading pattern is appropriate, but model initialization could fail due to missing model files, insufficient memory, or other issues. Consider adding try-except around model creation to provide clearer error messages.

 def _get_model(self) -> WhisperModel:
     """Lazy load the Whisper model."""
     if self._model is None:
-        # Use CPU with optimizations for better performance
-        self._model = WhisperModel(
-            self.model_size,
-            device="cpu",
-            compute_type="int8",  # Quantization for faster CPU inference
-            num_workers=1,  # Single worker for stability
-        )
+        try:
+            # Use CPU with optimizations for better performance
+            self._model = WhisperModel(
+                self.model_size,
+                device="cpu",
+                compute_type="int8",  # Quantization for faster CPU inference
+                num_workers=1,  # Single worker for stability
+            )
+        except Exception as e:
+            raise RuntimeError(
+                f"Failed to initialize Whisper model '{self.model_size}': {e}"
+            ) from e
     return self._model

---

37-68: Add error handling for transcription failures.

The model.transcribe() call can fail for various reasons (corrupted audio, unsupported format, etc.). Consider adding try-except to provide more informative error messages rather than letting raw faster-whisper exceptions propagate.

 def transcribe_file(self, audio_path: str, language: str | None = None) -> dict:
     """Transcribe audio file to text.
     
     Args:
         audio_path: Path to audio file
         language: Optional language code (e.g., "en", "es")
     
     Returns:
         Dict with transcription text and metadata
     """
     model = self._get_model()
 
-    # Transcribe with optimized settings
-    segments, info = model.transcribe(
-        audio_path,
-        language=language,
-        beam_size=1,  # Faster inference
-        best_of=1,  # Single pass
-        temperature=0,  # Deterministic output
-        vad_filter=True,  # Voice activity detection
-        vad_parameters={"min_silence_duration_ms": 500},
-    )
+    try:
+        # Transcribe with optimized settings
+        segments, info = model.transcribe(
+            audio_path,
+            language=language,
+            beam_size=1,  # Faster inference
+            best_of=1,  # Single pass
+            temperature=0,  # Deterministic output
+            vad_filter=True,  # Voice activity detection
+            vad_parameters={"min_silence_duration_ms": 500},
+        )
+    except Exception as e:
+        raise RuntimeError(
+            f"Failed to transcribe audio file '{audio_path}': {e}"
+        ) from e
 
     # Combine all segments
     text = " ".join(segment.text.strip() for segment in segments)

surfsense_backend/app/db.py (1)

300-331: Add DB-level validation for ranges (defensive integrity).

Add CHECK constraints to enforce valid ranges:

• weekly_day: 0..6
• hourly_minute: 0..59

Apply this diff:

-from sqlalchemy import (
+from sqlalchemy import (
     ARRAY,
     JSON,
     TIMESTAMP,
     Boolean,
     Column,
     Enum as SQLAlchemyEnum,
     ForeignKey,
     Integer,
     String,
     Text,
     Time,
     UniqueConstraint,
     text,
+    CheckConstraint,
 )
@@
 class ConnectorSchedule(BaseModel, TimestampMixin):
     __tablename__ = "connector_schedules"
-    __table_args__ = (
-        UniqueConstraint(
-            "connector_id", "search_space_id", name="uq_connector_search_space"
-        ),
-    )
+    __table_args__ = (
+        UniqueConstraint(
+            "connector_id", "search_space_id", name="uq_connector_search_space"
+        ),
+        CheckConstraint(
+            "weekly_day IS NULL OR (weekly_day BETWEEN 0 AND 6)",
+            name="ck_connector_schedules_weekly_day_range",
+        ),
+        CheckConstraint(
+            "hourly_minute IS NULL OR (hourly_minute BETWEEN 0 AND 59)",
+            name="ck_connector_schedules_hourly_minute_range",
+        ),
+    )

Optional: if you’ll filter frequently by schedule_type, consider index=True on schedule_type.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro