Thanks to visit codestin.com
Credit goes to github.com

Skip to content

README.md

Latest commit

 

History

History
301 lines (230 loc) · 7.91 KB

README.md

File metadata and controls

301 lines (230 loc) · 7.91 KB
Raw

Atlas

AI-powered content analysis platform for technical education and research workflows

Atlas combines a YouTube analysis pipeline, academic papers RAG, comparison analysis, and assignment generation into a single product-oriented experience. The current application uses a FastAPI backend and a React frontend while preserving the original Python pipeline modules for search, transcript extraction, summarization, comparison, and educational content generation.

What Atlas Does

YouTube analysis pipeline

  • Search YouTube videos from a natural language query
  • Fetch subtitles and transcripts for selected videos
  • Generate structured AI summaries focused on technical learning outcomes
  • Compare multiple videos across depth, difficulty, teaching style, and audience fit
  • Generate educational assignments from the analyzed content

Academic papers RAG

  • Query indexed PDFs using natural language
  • Return AI-generated responses with citations and source excerpts
  • Use LanceDB-backed retrieval through the existing paper indexing stack

Product interface

  • Browse existing pipeline runs
  • Explore videos, transcripts, summaries, comparison outputs, and assignments in a modern React UI
  • Trigger fresh pipeline steps through the API when credentials are available

Tech Stack

Backend

  • FastAPI
  • Pydantic
  • Existing pipeline modules in src/
  • LlamaIndex + LanceDB for academic papers retrieval

Frontend

  • React
  • TypeScript
  • Vite
  • Tailwind CSS
  • TanStack Query
  • Framer Motion

Architecture

Atlas is split into a typed API layer and a React client, while the core domain logic remains in the original Python modules.

flowchart LR
  ReactFrontend[ReactFrontend] --> FastAPIBackend[FastAPIBackend]
  FastAPIBackend --> RunService[RunService]
  FastAPIBackend --> PipelineService[PipelineService]
  FastAPIBackend --> PapersService[PapersService]
  RunService --> PipelineArtifacts[pipeline_output_Artifacts]
  PipelineService --> YouTubePipeline[YouTubePipeline]
  PipelineService --> YouTubeComparator[YouTubeOutputComparator]
  PipelineService --> AssignmentGenerator[YouTubeAssignmentGenerator]
  PapersService --> AcademicPapersRAG[AcademicPapersRAG]
Loading

Backend responsibilities

  • Expose run metadata and artifact endpoints
  • Wrap the existing YouTube and papers workflows
  • Read pipeline artifacts from pipeline_output_* folders
  • Provide execution endpoints for live searches and downstream pipeline steps

Frontend responsibilities

  • Load the latest available run
  • Render structured search, transcript, summary, comparison, and assignment views
  • Query the academic papers system
  • Trigger new runs and artifact generation through the API

Core Modules

Existing domain modules

  • src/youtube_pipeline.py
  • src/compare_youtube_outputs.py
  • src/assignment_generator.py
  • src/papers_rag.py
  • src/fetch_youtube_transcript.py
  • src/summarize_youtube_transcript.py

Backend API layer

  • backend/main.py
  • backend/routers/runs.py
  • backend/routers/pipeline.py
  • backend/routers/papers.py
  • backend/services/run_service.py
  • backend/services/pipeline_service.py
  • backend/services/papers_service.py
  • backend/services/artifact_readers.py

Frontend application

  • frontend/src/App.tsx
  • frontend/src/features/pipeline/pipeline-dashboard.tsx
  • frontend/src/features/papers/papers-panel.tsx
  • frontend/src/lib/api.ts
  • frontend/src/lib/types.ts

Project Structure

atlas/
├── backend/                        # FastAPI application
│   ├── main.py
│   ├── routers/
│   ├── schemas/
│   └── services/
├── frontend/                       # React + Vite frontend
│   ├── src/
│   ├── package.json
│   └── vite.config.ts
├── src/                            # Existing Python domain logic
│   ├── youtube_pipeline.py
│   ├── compare_youtube_outputs.py
│   ├── assignment_generator.py
│   ├── papers_rag.py
│   └── configs/config.yaml
├── papers/agents/                  # Academic PDFs
├── pipeline_output_*/              # YouTube pipeline artifacts
├── storage/                        # Indexed papers storage
├── requirements.txt
└── README.md

Setup

Prerequisites

  • Conda
  • Node.js 18+
  • npm
  • OpenRouter API key
  • YouTube Data API key

1. Clone the repository

git clone https://github.com/ishandutta0098/atlas
cd atlas

2. Activate the conda environment

If your environment already exists:

conda activate atlas

If you need to create it first:

conda create -n atlas python=3.10 -y
conda activate atlas

3. Install Python dependencies

pip install -r requirements.txt

4. Install frontend dependencies

cd frontend
npm install
cd ..

5. Configure environment variables

Create or update .env in the repository root:

OPENROUTER_API_KEY=your_openrouter_key
YOUTUBE_API_KEY=your_youtube_key

Running Atlas

Atlas runs as two processes during development: the FastAPI backend and the React frontend.

1. Start the backend

From the repository root:

conda activate atlas
uvicorn backend.main:app --reload --host 127.0.0.1 --port 8000

Backend health endpoint:

http://127.0.0.1:8000/api/health

2. Start the frontend

In a second terminal:

cd frontend
npm run dev

Frontend development URL:

http://127.0.0.1:5173

The Vite dev server proxies /api requests to http://127.0.0.1:8000 via frontend/vite.config.ts.

Build Commands

Frontend

cd frontend
npm run lint
npm run build

Backend smoke check

conda activate atlas
python -c "from fastapi.testclient import TestClient; from backend.main import app; client = TestClient(app); print(client.get('/api/health').json())"

API Overview

Run and artifact endpoints

  • GET /api/runs
  • GET /api/runs/latest
  • GET /api/runs/{run_id}
  • GET /api/runs/{run_id}/videos
  • GET /api/runs/{run_id}/transcripts
  • GET /api/runs/{run_id}/summaries
  • GET /api/runs/{run_id}/comparison
  • GET /api/runs/{run_id}/assignments

Pipeline execution endpoints

  • POST /api/pipeline/search
  • POST /api/runs/{run_id}/transcripts
  • POST /api/runs/{run_id}/summaries
  • POST /api/runs/{run_id}/comparison
  • POST /api/runs/{run_id}/assignments

Papers endpoints

  • GET /api/papers/status
  • POST /api/papers/query

Application Flow

YouTube workflow

  1. Submit a query through the frontend
  2. Backend creates or reuses a pipeline run
  3. Video metadata is exposed from run artifacts
  4. Transcript, summary, comparison, and assignment artifacts are read or generated
  5. The frontend renders each stage as a separate structured view

Papers workflow

  1. Frontend sends a natural language query to POST /api/papers/query
  2. FastAPI delegates to AcademicPapersRAG
  3. The response includes answer text, citations, excerpts, and timing metadata

Configuration

Primary runtime configuration lives in src/configs/config.yaml.

Important settings include:

  • OpenRouter model selection
  • worker counts and concurrency behavior
  • transcript language defaults
  • YouTube API settings
  • prompt paths and output directory settings

Data and Storage

YouTube artifacts

Each pipeline run writes files under a pipeline_output_<timestamp> folder:

  • metadata/search_results_*.json
  • metadata/fetch_results_*.json
  • metadata/summary_results_*.json
  • transcripts/*.srt
  • summaries/*_summary.json
  • assignments/*_assignment.md

Academic papers storage

  • PDFs live in papers/agents/
  • paper index storage lives in storage/papers_index/
  • vector store data lives in storage/papers_vectordb/

Legacy Interface

The original Gradio application still exists in app.py as a legacy interface, but the primary development path is now the FastAPI backend plus the React frontend.

License

MIT License. See LICENSE for details.