• 🔧 Design
• 🔍 Installation
• 🛠️ Usage
  • 🕸️ Parsing
  • ✂️ Chunking
  • 🧬 Embedding
  • ☁️ Storing on AWS
  • 🔎 Retrieval
• 📝 Examples
• 🌟 Future Enhancements
• 🤝 Contribution
• 🐛 Bug Reporting

UnifiedParser: For parsing content from URLs,PDF or Text File.

ChunkUtility: To split the parsed content into chunks.

EmbeddingUtility: Utilizes the OpenAI API to embed the chunks into vectors.

VectorStoreAWS: A utility for AWS operations related to embedding storage.

Retrival: Provides functionality to retrieve knowledge and run QnA.

npm install rag-aiquest

Use the UnifiedParser to parse content from a URL.

const parser = new UnifiedParser();
const parsedValue = await parser.parse('YOUR_URL_HERE');

To split the parsed content into chunks:

const chunks = ChunkUtility.splitIntoChunks(parsedValue, chunkSize, overlapSize);

Embed chunks using OpenAI API.

const embedding = new EmbeddingUtility('YOUR_OPENAI_API_KEY');
const embedded = await embedding.createEmbedding(chunks);

To upload the embedded model to AWS:

const aws = new VectorStoreAWS(AWS_ACCESS_KEY_ID, AWS_ACCESS_SECRET, AWS_BUCKET_NAME);
await aws.uploadEmbededModeltoAWS(embedded, 'YOUR_FILE_NAME');

To retrieve and query the knowledge:

const knowledge = await aws.getKnowledgeData('YOUR_FILE_NAME');
const retrive = new Retrival('YOUR_OPENAI_API_KEY');
const search = await retrive.QnARetrival(knowledge, 'YOUR_QUERY');
console.log(search.choices[0].message);

As given in the provided code, you can easily integrate the utilities to parse, chunk, embed, store, and retrieve knowledge.

Compression: Improve storage efficiency by compressing embedded vectors.

Batch Processing: Enhance the library to handle batch processing of URLs.

Support for More Embeddings: Plan to add support for other embedding APIs.

If you wish to contribute to rag-aiquest, please refer to the CONTRIBUTING.md file.

Feel free to open an issue on GitHub if you find any bug.

• Feel free to Open an issue on GitHub to request any additional features you might need for your use case.
• Connect with me on LinkedIn. I'd love ❤️️ to hear where you are using this library.

Check here for release notes.

• Active OpenAI API Key
• AWS BUCKET SECRET_KEY,Bucket Name, AUTH KEY