Installation

Complete installation guide for the Obstetrics RAG Benchmark

This guide provides detailed installation instructions for setting up the Obstetrics RAG Benchmark on your local machine.

Prerequisites

Before installing, ensure your system meets these requirements:

System Requirements

• Operating System: Linux, macOS, or Windows with WSL2
• Python: Version 3.8 or higher
• Git: For repository cloning
• Internet Connection: Required for API calls and package installation

API Requirements

You’ll need an OpenAI API key to run evaluations. The benchmark uses:

• text-embedding-3-small for creating document embeddings
• GPT-4o (default) or other OpenAI models for answer generation

Get your API key from OpenAI Platform.

Installation steps

1. Verify Python installation
2. Check that Python 3.8+ is installed:
3. python --version
   # or
   python3 --version
   

4. Expected output: Python 3.8.x or higher
5. If Python is not installed or the version is below 3.8, download and install it from python.org.

6. Clone the repository
7. Clone the benchmark repository to your local machine:
8. git clone https://github.com/NicolasHoyosDevs/RAG-Benchmark.git
   cd RAG-Benchmark
   

9. Create a virtual environment (recommended)
10. It’s recommended to use a virtual environment to isolate dependencies:
11. Linux/macOS

    python -m venv venv
    source venv/bin/activate
    

    Windows

    python -m venv venv
    venv\Scripts\activate
    
12. You should see (venv) appear in your terminal prompt.
13. Install dependencies
14. Install all required Python packages:
15. pip install -r requirements.txt
    

16. Key dependencies installed
17. The requirements.txt includes:
18. # LangChain Ecosystem
    langchain>=0.2.0
    langchain-community>=0.2.0
    langchain-chroma>=0.1.0
    langchain-openai>=0.1.0
    
    # LLM & API
    openai>=1.0.0
    tiktoken>=0.5.0
    
    # Vector Database & Retrieval
    chromadb>=0.4.0
    rank-bm25>=0.2.0
    
    # Data processing
    pandas>=2.0.0
    numpy>=1.20.0
    
    # Utilities
    python-dotenv>=1.0.0
    pydantic>=2.0.0
    tqdm>=4.60.0
    
    # Evaluation
    ragas>=0.2.15,<0.3
    datasets>=2.15.0
    evaluate>=0.4.0
    

19. Configure environment variables
20. Create a .env file in the project root directory:
21. cp .env.example .env
    

22. Edit the .env file and add your API keys:
23. # OpenAI API configuration (Required)
    OPENAI_API_KEY=your_openai_api_key_here
    
    # Optional: HuggingFace models configuration
    # Only needed for specialized medical models
    HF_TOKEN=your_huggingface_token_here
    MEDIPHI_ENDPOINT_URL=https://your_mediphi_endpoint_url
    MEDGEMMA_ENDPOINT_URL=https://your_medgemma_endpoint_url
    
    # Optional: PageIndex configuration
    PAGEINDEX_API_KEY=your_pageindex_api_key_here
    PAGEINDEX_DOC_ID=your_pageindex_doc_id_here
    

24. Required: Only OPENAI_API_KEY is required for basic usage. Other API keys are optional and only needed if you plan to use specialized medical models or PageIndex RAG.

25. Verify project structure
26. Ensure the project structure is correct:
27. ls -la
    

28. You should see:
29. RAG-Benchmark/
    ├── src/
    │   ├── rag/                     # RAG implementations
    │   ├── evaluation/              # RAGAS orchestration
    │   └── common/                  # Shared utilities
    ├── scripts/
    │   ├── create_embeddings.py     # Embedding generation
    │   └── run_evaluation.py        # Evaluation CLI
    ├── data/
    │   ├── chunks/                  # Pre-chunked documents
    │   └── embeddings/              # Vector database storage
    ├── results/                     # Evaluation outputs
    ├── requirements.txt             # Python dependencies
    ├── .env.example                 # Environment template
    └── README.md
    

30. Create embeddings database
31. Generate embeddings for the obstetrics medical corpus:
32. python scripts/create_embeddings.py
    

33. What this does:
35. Expected output:
36. === STARTING EMBEDDING CREATION PROCESS ===
    Loading chunks from: /path/to/data/chunks/chunks_final.json
    Successfully loaded 150 chunks.
    Initializing OpenAI embeddings model...
    Creating and storing vector database at: /path/to/data/embeddings/chroma_db
    Collection: guia_embarazo_parto
    
    Embeddings created and stored successfully!
    Total vectors in database: 150
    Database saved at: /path/to/data/embeddings/chroma_db
    
    === PROCESS COMPLETED ===
    

37. The embedding creation process typically takes 1-2 minutes. It only needs to be run once unless you modify the source documents.

38. Verify installation
39. Run a test evaluation to verify everything is working:
40. python scripts/run_evaluation.py simple
    

41. If successful, you’ll see evaluation results printed to the console and saved to the results/ directory.

Troubleshooting

Common issues and solutions

ModuleNotFoundError: No module named 'langchain'

Problem: Dependencies not installed or wrong Python environment.

Solution:

# Ensure virtual environment is activated
source venv/bin/activate  # Linux/macOS
venv\Scripts\activate     # Windows

# Reinstall dependencies
pip install -r requirements.txt

OpenAI API authentication error

Problem: Invalid or missing OpenAI API key.

Solution:

1. Verify your API key is correct in .env:
   cat .env | grep OPENAI_API_KEY
   
2. Ensure no extra spaces or quotes around the key
3. Verify your OpenAI account has credits: platform.openai.com/account/billing

ChromaDB connection error

Problem: Vector database not initialized or corrupted.

Solution:

1. Delete existing database:
   rm -rf data/embeddings/chroma_db
   
2. Recreate embeddings:
   python scripts/create_embeddings.py
   

FileNotFoundError: chunks_final.json

Problem: Missing source data files.

Solution: Ensure you cloned the complete repository:

ls data/chunks/chunks_final.json

If missing, re-clone the repository or check that Git LFS files downloaded correctly.

Python version compatibility issues

Problem: Using Python version below 3.8.

Solution:

1. Install Python 3.8+ from python.org
2. Create virtual environment with correct version:
   python3.8 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   

Rate limit errors from OpenAI API

Problem: Exceeding OpenAI API rate limits.

Solution:

• Tier 1 accounts have lower rate limits
• Wait a few minutes between evaluation runs
• Consider upgrading your OpenAI account tier
• For comprehensive evaluations, use --debug flag to see detailed progress

Upgrading

To upgrade to the latest version:

# Pull latest changes
git pull origin main

# Update dependencies
pip install -r requirements.txt --upgrade

# Recreate embeddings if data changed
python scripts/create_embeddings.py

Uninstallation

To completely remove the project:

# Deactivate virtual environment
deactivate

# Remove project directory
cd ..
rm -rf RAG-Benchmark

Next steps

Quickstart Guide

Run your first evaluation in 5 minutes

RAG Architectures

Learn about the different RAG strategies

Evaluation Guide

Understanding RAGAS metrics and results

API Reference

Explore the Python API for custom workflows

Getting help

If you encounter issues not covered in this guide:

• Check the GitHub Issues
• Review the README
• Open a new issue with:
  • Your Python version (python --version)
  • Full error message
  • Steps to reproduce
  • Your operating system

System requirements summary

Component	Requirement	Notes
Python	3.8+	Tested on 3.8, 3.9, 3.10, 3.11
RAM	4GB minimum	8GB+ recommended for large evaluations
Storage	500MB	For embeddings and results
OpenAI API	Active account	With available credits
Internet	Required	For API calls

Estimated costs: Running a single RAG evaluation with 10 questions typically costs $0.05-0.15 USD depending on the model used.