All projects
Project

multimodal-hybrid-rag

A robust, TypeScript-based RAG system. This system is unique because it supports a Hybrid Architecture: use Google Gemini for high-performance Multimodal RAG (Images/Audio/PDF) or use Ollama for 100% local, privacy-focused text RAG.

View on GitHub Live demo
TypeScript 1 0
ai-toolsgemninimultimodalrag-chatbot

Multi-Modal Hybrid RAG System (Gemini & Ollama)

[!NOTE] This is a demo project designed to explain the core concepts of Retrieval-Augmented Generation (RAG) systems. It was built specifically to accompany the session: Building a Multimodal RAG System.

A robust, TypeScript-based RAG system. This system is unique because it supports a Hybrid Architecture: use Google Gemini for high-performance Multimodal RAG (Images/Audio/PDF) or use Ollama for 100% local, privacy-focused text RAG.

📋 Prerequisites

1. System Requirements

  • Node.js: v20.0.0 or higher.
  • TypeScript: Installed via devDependencies.

2. For Cloud Mode (Google Gemini)

  • API Key: A valid key from Google AI Studio.
  • Capabilities: Full support for Text, PDF, Images (.png, .jpg), and Audio (.mpeg, .mp3, .wav).
  • Important: Multimodal embedding (Images & Audio) is only supported when using the gemini-embedding-2-preview model.

3. For Local Mode (Ollama)

  • Ollama: Download and install from ollama.com.
  • Models: Pull the required models before starting: 
    ollama pull llama3.2:3b
# Or your preferred model like gemma:2b
ollama pull nomic-embed-text
  • Capabilities: Text and PDF extraction only. (Multimodal files are safely skipped in Local mode).

🚀 Setup & Usage

1. Install Dependencies

npm install

2. Configure Environment

Copy the example environment file and fill in your keys:

cp .env.example .env

[!IMPORTANT] Toggle USE_OLLAMA=true or false in your .env to switch between local and cloud modes.

3. Add Your Data

[!NOTE] The /data-sources folder is empty by default (ignored by Git).

You must manually place your own files (PDF, Markdown, JPG, PNG, MPEG) inside the /data-sources directory for the system to have a knowledge base to talk about.

4. Ingest and Index

This step converts your custom files into mathematical vectors and stores them in your own local FAISS index.

npm run ingest

[!WARNING] Model Switch Requirement: Every embedding model (Google Gemini vs. Ollama) has a different "Vector Dimension" (e.g., 3072 vs. 768). If you switch models in your .env file, you MUST re-run npm run ingest to rebuild your database. Failure to do this will cause a "Dimensionality Mismatch" crash.

Note: The /index folder contents are ignored by Git. You must run this command to generate your local search database.

5. Start Chatting

Launch the interactive CLI:

npm run chat

🏗️ Project Architecture

  • data-sources/: Your raw knowledge base (PDF, MD, JPG, PNG, MPEG).
  • index/: Contains faiss.index (vector math) and metadata.json (text/binary mapping).
  • ingest.ts: The pipeline that chunks text and generates multimodal embeddings.
  • chat.ts: The interactive interface with dynamic Top-K retrieval and time-awareness.
  • WORKFLOW.md: Detailed visual diagrams of the internal logic.
  • .env: Your private configuration and API keys.