This is an end-to-end AI RAG and sentiment system that processes crypto news.
- Text data is extracted from website articles.
- Extract metadata: publish_date, entities mentioned (coins, tokens, blockchains, bridges)
- Split text into chunks and calculate embeddings.
- Compute overall daily sentiment per entity
- Store data in Tortoise ORM models in Postgresql, also use pgvector for storing vectors from embeddings
- Setup FastAPI to create endpoints for inference and search
- Implement Leucene or other search system for BM25 search
- Implement hybrid search with RAG and BM25
- Create search endpoint
- Create QA endpoint with RAG and Reranking