RAGAS Evaluator

Professional evaluation module using RAGAS fundamental metrics for RAG system assessment

Overview

The RAGAS Evaluator module provides comprehensive evaluation capabilities for RAG systems using the RAGAS (Retrieval-Augmented Generation Assessment) framework. It supports multiple RAG architectures, various LLM models, and generates detailed performance metrics and cost analysis.

Module Path: src/evaluation/ragas_evaluator.py

Key Features

• Multiple RAG Support: Evaluate Simple, HyDE, Rewriter, Hybrid, Hybrid-RRF, and PageIndex RAG systems
• RAGAS Metrics: Faithfulness, Answer Relevancy, Context Precision, and Context Recall
• Multi-Model Evaluation: Compare performance across different LLM models
• Performance Tracking: Token usage, execution time, and cost analysis
• Obstetric Dataset: Built-in specialized medical dataset with 10 ground truth questions
• Comprehensive Reporting: JSON export with detailed question-by-question analysis

---

RAGASEvaluator Class

The main orchestrator class for RAG evaluation workflows.

Constructor

RAGASEvaluator(rag_type: str = "rewriter", debug: bool = False)

Initialize a RAGAS evaluator for a specific RAG architecture.

rag_type · str · default: "rewriter"

RAG architecture to evaluate. Supported values:

• "simple" - Simple Semantic RAG
• "hybrid" - Hybrid RAG (BM25 + Semantic)
• "hybrid-rrf" - Hybrid RAG with Reciprocal Rank Fusion
• "hyde" - HyDE RAG (Hypothetical Documents)
• "rewriter" - Multi-Query Rewriter RAG
• "pageindex" - PageIndex RAG

debug · bool · default: "False"

Enable debug output for detailed logging and error traces

Instance Attributes:

• metrics - List of RAGAS metric objects (faithfulness, answer_relevancy, context_precision, context_recall)
• results_dir - Path to results directory (project_root/results)
• query_function - RAG-specific query function for evaluation
• rag_name - Descriptive name of the RAG system
• rag_type - RAG type identifier
• llm_model - Default LLM model name (initially “gpt-4o”)
• performance_metadata - List storing execution metrics per query

Example:

from src.evaluation.ragas_evaluator import RAGASEvaluator

# Initialize evaluator for HyDE RAG
evaluator = RAGASEvaluator(rag_type="hyde", debug=True)

# Initialize with simple RAG
simple_evaluator = RAGASEvaluator(rag_type="simple")

---

set_models

set_models(llm_model: str = None, embeddings_model: str = None)

Update the LLM model used by the evaluator.

llm_model · str

New LLM model name (e.g., “gpt-4o”, “claude-3-5-sonnet”)

embeddings_model · str

New embeddings model name (currently not used in implementation)

Example:

evaluator = RAGASEvaluator(rag_type="hybrid")
evaluator.set_models(llm_model="claude-3-5-sonnet")

---

load_test_queries

load_test_queries(use_obstetric_dataset: bool = True) -> List[Dict]

Load test queries from the obstetric dataset.

use_obstetric_dataset · bool · default: "True"

Whether to use the built-in obstetric dataset (currently the only option)

return · List[Dict]

List of query dictionaries, each containing:

• question (str) - The test question
• ground_truth (str) - The expected answer

Example:

evaluator = RAGASEvaluator(rag_type="simple")
queries = evaluator.load_test_queries()
print(f"Loaded {len(queries)} test queries")
# Output: Loaded 10 test queries

---

prepare_dataset

prepare_dataset(test_queries: List[Dict]) -> Dataset

Prepare RAGAS dataset format by executing RAG queries and collecting results.

test_queries · List[Dict] · required

List of test query dictionaries with “question” and “ground_truth” keys

return · Dataset

RAGAS-compatible HuggingFace Dataset with columns:

• question - User queries
• answer - RAG-generated answers
• contexts - Retrieved context chunks
• ground_truth - Expected answers

Side Effects:

• Populates self.performance_metadata with execution metrics
• Prints progress information for each query
• Handles errors gracefully and continues processing

Example:

evaluator = RAGASEvaluator(rag_type="rewriter")
test_queries = evaluator.load_test_queries()
dataset = evaluator.prepare_dataset(test_queries)
print(f"Dataset size: {len(dataset)}")
# Output: Dataset size: 10

---

evaluate_rag

evaluate_rag(dataset: Dataset) -> Dict[str, Any]

Evaluate RAG using RAGAS metrics with automatic fallback to synchronous mode.

dataset · Dataset · required

Prepared RAGAS dataset with questions, answers, contexts, and ground truth

return · Dict[str, Any]

Evaluation results object with metric scores. Can be accessed as:

• Object attributes (e.g., results.faithfulness)
• Via to_pandas() method for DataFrame conversion

Behavior:

• Executes RAGAS evaluation with 8 parallel workers
• Disables per-call timeouts for Python 3.14+ compatibility
• Automatically detects all-NaN results and falls back to synchronous evaluation
• Handles async context issues gracefully

Example:

evaluator = RAGASEvaluator(rag_type="hybrid-rrf")
queries = evaluator.load_test_queries()
dataset = evaluator.prepare_dataset(queries)
results = evaluator.evaluate_rag(dataset)

# Access metrics
print(f"Faithfulness: {results.faithfulness:.3f}")
print(f"Answer Relevancy: {results.answer_relevancy:.3f}")

# Convert to DataFrame
df = results.to_pandas()
print(df[['faithfulness', 'answer_relevancy']].describe())

---

display_results

display_results(results)

Display evaluation results in a formatted console output.

results · required

Evaluation results from evaluate_rag()

Output Format:

• Individual metric scores (0-1 scale)
• Average score across all metrics
• Performance assessment (Excellent/Good/Needs improvement/Significant improvements needed)

Example:

evaluator = RAGASEvaluator(rag_type="simple")
evaluator.run_evaluation()  # Automatically calls display_results

# Manual display
results = evaluator.evaluate_rag(dataset)
evaluator.display_results(results)

Sample Output:

============================================================
RAGAS EVALUATION RESULTS
============================================================
Faithfulness: 0.872
Answer Relevancy: 0.845
Context Precision: 0.790
Context Recall: 0.823

Average Score: 0.833
Performance: Excellent

---

save_results

save_results(
    results,
    filename: str = None,
    return_data_only: bool = False,
    model_name: str = None
) -> Union[Path, Dict, None]

Save evaluation results to JSON file or return as dictionary.

results · required

Evaluation results from evaluate_rag()

filename · str

Output filename. If not provided, generates timestamped filename

return_data_only · bool · default: "False"

If True, returns dictionary instead of saving to file

model_name · str

LLM model name to include in metadata

return · Union[Path, Dict, None]

• File path if saved to disk
• Dictionary if return_data_only=True
• None if no results or error occurred

Output Structure:

{
  "metadata": {
    "timestamp": "2026-03-11T10:30:45.123456",
    "evaluation_type": "single_rag_evaluation_hybrid",
    "dataset_size": 10,
    "rags_evaluated": ["hybrid"],
    "model_used": "gpt-4o"
  },
  "pricing_config": { ... },
  "summary": {
    "hybrid": {
      "rag_name": "Hybrid RAG (BM25 + Semantic)",
      "metrics": {
        "faithfulness": 0.872,
        "answer_relevancy": 0.845,
        "context_precision": 0.790,
        "context_recall": 0.823
      },
      "performance": {
        "average_execution_time": 2.456,
        "total_input_tokens": 12450,
        "total_output_tokens": 3200,
        "total_cost": 0.045678,
        "average_cost_per_question": 0.004568,
        "overall_average_score": 0.833
      }
    }
  },
  "question_by_question": [ ... ]
}

Example:

# Save to file
evaluator = RAGASEvaluator(rag_type="hyde")
results = evaluator.evaluate_rag(dataset)
filepath = evaluator.save_results(results, model_name="gpt-4o")
print(f"Results saved to: {filepath}")

# Return as dictionary
data = evaluator.save_results(results, return_data_only=True)
print(f"Overall average: {data['summary']['hyde']['performance']['overall_average_score']}")

---

run_evaluation

run_evaluation() -> Any

Execute complete evaluation pipeline with the obstetric dataset.

Returns: Evaluation results object

Pipeline Steps:

1. Load test queries from obstetric dataset
2. Prepare RAGAS dataset by executing RAG queries
3. Run RAGAS evaluation with configured metrics
4. Display formatted results
5. Save results to JSON file

Example:

# Complete evaluation workflow
evaluator = RAGASEvaluator(rag_type="pageindex", debug=False)
results = evaluator.run_evaluation()

# Results are automatically displayed and saved

Console Output:

Starting RAGAS evaluation
System: PageIndex RAG
Dataset: Obstetric queries (10 questions)
============================================================
Processing 10 queries with PageIndex RAG
Dataset prepared: 10 queries processed
Starting RAGAS evaluation...
Evaluation completed

============================================================
RAGAS EVALUATION RESULTS
============================================================
...
Results saved to: results/ragas_evaluation_pageindex_20260311_103045.json

Evaluation completed - 10 queries processed

---

run_multi_model_evaluation

run_multi_model_evaluation(models_to_test: list = None)

Run evaluation for the current RAG type against multiple LLM models.

models_to_test · list

List of model keys from MODELS_REGISTRY. If None, tests all registered models.

Behavior:

• Iterates through each model in the list
• Creates custom LLM instances for each model
• Wraps RAG query functions to use the custom model
• Collects results from all models
• Generates consolidated multi-model report

Output File: ragas_multimodel_{rag_type}_{timestamp}.json

Example:

# Test specific models
evaluator = RAGASEvaluator(rag_type="simple")
models = ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro"]
evaluator.run_multi_model_evaluation(models_to_test=models)

# Test all registered models
evaluator.run_multi_model_evaluation()

Output Structure:

{
  "metadata": {
    "timestamp": "2026-03-11T10:30:45.123456",
    "evaluation_type": "multi_model_rag_comparison",
    "rag_type_evaluated": "simple",
    "dataset_size": 10,
    "models_evaluated": ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro"]
  },
  "pricing_config": { ... },
  "summary": {
    "gpt-4o": { ... },
    "claude-3-5-sonnet": { ... },
    "gemini-1.5-pro": { ... }
  },
  "question_by_question": [ ... ]
}

---

SyncEvaluationResult Class

Lightweight wrapper for synchronous evaluation results, compatible with RAGAS result format.

Constructor

SyncEvaluationResult(dataframe: pd.DataFrame, metrics: List[Any])

dataframe · pd.DataFrame · required

DataFrame containing metric scores for each evaluation sample

metrics · List[Any] · required

List of RAGAS metric objects used in evaluation

Behavior:

• Calculates mean value for each metric and stores as instance attribute
• Provides to_pandas() method for DataFrame access
• Compatible with downstream result processing pipelines

Example:

from src.evaluation.ragas_evaluator import SyncEvaluationResult
import pandas as pd

# Create result wrapper
df = pd.DataFrame({
    'faithfulness': [0.9, 0.8, 0.85],
    'answer_relevancy': [0.85, 0.9, 0.88]
})
result = SyncEvaluationResult(df, metrics)

# Access mean scores
print(f"Faithfulness: {result.faithfulness}")
print(f"Answer Relevancy: {result.answer_relevancy}")

# Get DataFrame
df = result.to_pandas()

---

Helper Functions

evaluate_rewriter_rag

evaluate_rewriter_rag(export_analysis: bool = False, debug: bool = False)

Evaluate Rewriter RAG specifically with optional analysis export.

export_analysis · bool · default: "False"

Export detailed analysis files (CSV, charts) using export_ragas_analysis utility

debug · bool · default: "False"

Enable debug output

Example:

from src.evaluation.ragas_evaluator import evaluate_rewriter_rag

# Basic evaluation
results = evaluate_rewriter_rag()

# With detailed analysis export
results = evaluate_rewriter_rag(export_analysis=True, debug=True)

---

evaluate_hybrid_rag

evaluate_hybrid_rag(export_analysis: bool = False, debug: bool = False)

Evaluate Hybrid RAG (BM25 + Semantic) specifically.

export_analysis · bool · default: "False"

Export detailed analysis files

debug · bool · default: "False"

Enable debug output

Example:

from src.evaluation.ragas_evaluator import evaluate_hybrid_rag

results = evaluate_hybrid_rag(export_analysis=True)

---

evaluate_hybrid_rrf_rag

evaluate_hybrid_rrf_rag(export_analysis: bool = False, debug: bool = False)

Evaluate Hybrid RAG with Reciprocal Rank Fusion.

export_analysis · bool · default: "False"

Export detailed analysis files

debug · bool · default: "False"

Enable debug output

Example:

from src.evaluation.ragas_evaluator import evaluate_hybrid_rrf_rag

results = evaluate_hybrid_rrf_rag()

---

evaluate_hyde_rag

evaluate_hyde_rag(export_analysis: bool = False, debug: bool = False)

Evaluate HyDE RAG (Hypothetical Documents) specifically.

export_analysis · bool · default: "False"

Export detailed analysis files

debug · bool · default: "False"

Enable debug output

Example:

from src.evaluation.ragas_evaluator import evaluate_hyde_rag

results = evaluate_hyde_rag(export_analysis=True)

---

evaluate_simple_rag

evaluate_simple_rag(export_analysis: bool = False, debug: bool = False)

Evaluate Simple Semantic RAG specifically.

export_analysis · bool · default: "False"

Export detailed analysis files

debug · bool · default: "False"

Enable debug output

Example:

from src.evaluation.ragas_evaluator import evaluate_simple_rag

results = evaluate_simple_rag()

---

evaluate_pageindex_rag

evaluate_pageindex_rag(export_analysis: bool = False, debug: bool = False)

Evaluate PageIndex RAG specifically.

export_analysis · bool · default: "False"

Export detailed analysis files

debug · bool · default: "False"

Enable debug output

Example:

from src.evaluation.ragas_evaluator import evaluate_pageindex_rag

results = evaluate_pageindex_rag()

---

evaluate_both_rags

evaluate_both_rags(export_analysis: bool = False, debug: bool = False)

Evaluate both original RAG systems sequentially (Rewriter and Hybrid).

export_analysis · bool · default: "False"

Export detailed analysis files for both systems

debug · bool · default: "False"

Enable debug output

return · Dict

Dictionary with keys “rewriter” and “hybrid” containing respective results

Example:

from src.evaluation.ragas_evaluator import evaluate_both_rags

results = evaluate_both_rags(export_analysis=True)
print(f"Rewriter score: {results['rewriter'].faithfulness}")
print(f"Hybrid score: {results['hybrid'].faithfulness}")

---

evaluate_all_rags

evaluate_all_rags(export_analysis: bool = False, debug: bool = False)

Evaluate all 6 RAG systems sequentially with comprehensive comparison report.

export_analysis · bool · default: "False"

Export detailed analysis files for all systems

debug · bool · default: "False"

Enable debug output

return · Dict

Dictionary with keys for each RAG type (“simple”, “hyde”, “rewriter”, “hybrid”, “hybrid-rrf”, “pageindex”)

Behavior:

• Evaluates all 6 RAG systems in sequence
• Generates individual result files for each RAG
• Creates consolidated comparison report with best performer analysis
• Includes 2-second pause between evaluations

Output Files:

• Individual: ragas_evaluation_{rag_type}_{timestamp}.json
• Comparison: ragas_comparison_all_rags_{timestamp}.json

Example:

from src.evaluation.ragas_evaluator import evaluate_all_rags

results = evaluate_all_rags(export_analysis=True, debug=False)

# Access individual results
for rag_type, result in results.items():
    print(f"{rag_type}: {result.faithfulness:.3f}")

---

run_all_models_all_rags_evaluation

run_all_models_all_rags_evaluation(export_analysis: bool = False, debug: bool = False)

Comprehensive evaluation: ALL RAG types against ALL LLM models.

export_analysis · bool · default: "False"

Export detailed analysis files (currently not used in this function)

debug · bool · default: "False"

Enable debug output

Behavior:

• Tests all 6 RAG types with all models in MODELS_REGISTRY
• Total evaluations: 6 RAGs × N models
• Generates comprehensive consolidated report
• Provides progress indicators and success statistics

Output File: ragas_comprehensive_all_rags_all_models_{timestamp}.json

Example:

from src.evaluation.ragas_evaluator import run_all_models_all_rags_evaluation

# Run comprehensive evaluation
results = run_all_models_all_rags_evaluation(debug=True)

Console Output:

🚀 Starting comprehensive evaluation: ALL RAGs vs ALL Models
RAG types to evaluate: ['simple', 'hybrid', 'hybrid-rrf', 'hyde', 'rewriter', 'pageindex']
Models to test: ['gpt-4o', 'claude-3-5-sonnet', 'gemini-1.5-pro', ...]
Total evaluations: 6 × 8 = 48
============================================================
...
================================================================================
🎉 COMPREHENSIVE EVALUATION COMPLETED!
📄 Report saved to: results/ragas_comprehensive_all_rags_all_models_20260311_103045.json
📊 Total evaluations: 6 RAGs × 8 models = 48
✅ Successful evaluations: 46/48
================================================================================

---

DATA_GT Dataset

Built-in obstetric and pregnancy-specific evaluation dataset with 10 ground truth questions.

Structure:

DATA_GT = [
    {
        "question": "¿En qué momento y quien debe reevaluar el riesgo clínico...",
        "ground_truth": "El Ginecobstetra en la semana 28 - 30 y semana 34 – 36."
    },
    # ... 9 more questions
]

Topics Covered:

• Prenatal care scheduling and timing
• Risk assessment protocols
• Clinical evaluation tools (Herrera & Hurtado scale)
• Postpartum depression screening
• Weight gain recommendations by BMI
• VBAC (Vaginal Birth After Cesarean) probabilities
• Nausea and vomiting treatment options

Language: Spanish (Colombian clinical guidelines)

Usage:

from src.evaluation.ragas_evaluator import DATA_GT

print(f"Dataset size: {len(DATA_GT)}")
for i, item in enumerate(DATA_GT, 1):
    print(f"Q{i}: {item['question'][:50]}...")

---

RAGAS Metrics

The evaluator uses four fundamental RAGAS metrics:

Faithfulness

Measures whether the answer is factually consistent with the retrieved contexts. Score range: 0-1 (higher is better).

Calculation: Checks if claims in the answer can be inferred from the contexts without hallucination.

---

Answer Relevancy

Measures how relevant the answer is to the original question. Score range: 0-1 (higher is better).

Calculation: Uses embeddings to compute semantic similarity between question and answer.

---

Context Precision

Measures the proportion of relevant contexts in the retrieved set. Score range: 0-1 (higher is better).

Calculation: Evaluates whether retrieved contexts are actually useful for answering the question.

---

Context Recall

Measures whether all necessary information from ground truth is present in retrieved contexts. Score range: 0-1 (higher is better).

Calculation: Checks if ground truth answer can be derived from the retrieved contexts.

---

Result Format

Single RAG Evaluation

{
  "metadata": {
    "timestamp": "2026-03-11T10:30:45.123456",
    "evaluation_type": "single_rag_evaluation_hybrid",
    "dataset_size": 10,
    "rags_evaluated": ["hybrid"],
    "model_used": "gpt-4o"
  },
  "pricing_config": {
    "gpt-4o": {
      "input": 0.0025,
      "output": 0.01
    }
  },
  "summary": {
    "hybrid": {
      "rag_name": "Hybrid RAG (BM25 + Semantic)",
      "metrics": {
        "faithfulness": 0.872,
        "answer_relevancy": 0.845,
        "context_precision": 0.790,
        "context_recall": 0.823
      },
      "performance": {
        "average_execution_time": 2.456,
        "total_input_tokens": 12450,
        "total_output_tokens": 3200,
        "total_cost": 0.045678,
        "average_cost_per_question": 0.004568,
        "overall_average_score": 0.833
      }
    }
  },
  "question_by_question": [
    {
      "question_id": 1,
      "question": "¿En qué momento y quien debe reevaluar...",
      "ground_truth": "El Ginecobstetra en la semana 28 - 30...",
      "rag_results": {
        "hybrid": {
          "answer": "Según las guías clínicas...",
          "contexts_count": 5,
          "metrics": {
            "faithfulness": 0.900,
            "answer_relevancy": 0.875,
            "context_precision": 0.800,
            "context_recall": 0.850
          },
          "performance": {
            "question": "¿En qué momento y quien debe reevaluar...",
            "execution_time": 2.345,
            "input_tokens": 1245,
            "output_tokens": 320,
            "total_cost": 0.004567,
            "cost_source": "precise"
          }
        }
      }
    }
  ]
}

Multi-Model Comparison

{
  "metadata": {
    "timestamp": "2026-03-11T10:30:45.123456",
    "evaluation_type": "multi_model_rag_comparison",
    "rag_type_evaluated": "simple",
    "dataset_size": 10,
    "models_evaluated": ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro"]
  },
  "summary": {
    "gpt-4o": {
      "model_name": "gpt-4o",
      "metrics": { ... },
      "performance": { ... }
    },
    "claude-3-5-sonnet": {
      "model_name": "claude-3-5-sonnet",
      "metrics": { ... },
      "performance": { ... }
    }
  },
  "question_by_question": [
    {
      "question_id": 1,
      "question": "...",
      "ground_truth": "...",
      "rag_results": {
        "gpt-4o": { ... },
        "claude-3-5-sonnet": { ... }
      }
    }
  ]
}

Comprehensive All RAGs All Models

{
  "metadata": {
    "timestamp": "2026-03-11T10:30:45.123456",
    "evaluation_type": "comprehensive_all_rags_all_models",
    "dataset_size": 10,
    "rags_evaluated": ["simple", "hybrid", "hybrid-rrf", "hyde", "rewriter", "pageindex"],
    "models_evaluated": ["gpt-4o", "claude-3-5-sonnet", ...],
    "total_evaluations": 48
  },
  "summary": {
    "simple": {
      "gpt-4o": { ... },
      "claude-3-5-sonnet": { ... }
    },
    "hybrid": { ... }
  },
  "question_by_question": [
    {
      "question_id": 1,
      "question": "...",
      "ground_truth": "...",
      "rag_results": {
        "simple": {
          "gpt-4o": { ... },
          "claude-3-5-sonnet": { ... }
        },
        "hybrid": { ... }
      }
    }
  ]
}

---

Command-Line Usage

The module can be executed directly from the command line:

# Evaluate specific RAG type
python src/evaluation/ragas_evaluator.py simple
python src/evaluation/ragas_evaluator.py hybrid-rrf

# Evaluate with detailed analysis export
python src/evaluation/ragas_evaluator.py hyde --export

# Evaluate with debug output
python src/evaluation/ragas_evaluator.py rewriter --debug

# Evaluate multiple RAGs
python src/evaluation/ragas_evaluator.py both        # Rewriter + Hybrid
python src/evaluation/ragas_evaluator.py all         # All 6 RAGs

# Multi-model evaluation
python src/evaluation/ragas_evaluator.py multi-model simple

# Comprehensive evaluation (all RAGs × all models)
python src/evaluation/ragas_evaluator.py all-models-all-rags

Available Flags:

• --export or -e - Export detailed analysis files
• --debug or -d - Enable debug output

---

Complete Usage Example

from src.evaluation.ragas_evaluator import RAGASEvaluator

# 1. Basic Evaluation
evaluator = RAGASEvaluator(rag_type="hybrid", debug=False)
results = evaluator.run_evaluation()

# 2. Custom Workflow
evaluator = RAGASEvaluator(rag_type="simple")
queries = evaluator.load_test_queries()
dataset = evaluator.prepare_dataset(queries)
results = evaluator.evaluate_rag(dataset)
evaluator.display_results(results)
filepath = evaluator.save_results(results, model_name="gpt-4o")

# 3. Multi-Model Comparison
evaluator = RAGASEvaluator(rag_type="hyde")
models = ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro"]
evaluator.run_multi_model_evaluation(models_to_test=models)

# 4. Change Model During Evaluation
evaluator = RAGASEvaluator(rag_type="pageindex")
evaluator.set_models(llm_model="claude-3-5-sonnet")
results = evaluator.run_evaluation()

# 5. Access Detailed Metrics
df = results.to_pandas()
print(df[['faithfulness', 'answer_relevancy', 'context_precision', 'context_recall']])
print(f"Mean Faithfulness: {df['faithfulness'].mean():.3f}")
print(f"Std Faithfulness: {df['faithfulness'].std():.3f}")

# 6. Programmatic Comparison
from src.evaluation.ragas_evaluator import evaluate_all_rags
all_results = evaluate_all_rags(export_analysis=True)

for rag_type, result in all_results.items():
    print(f"\n{rag_type.upper()} Performance:")
    print(f"  Faithfulness: {result.faithfulness:.3f}")
    print(f"  Answer Relevancy: {result.answer_relevancy:.3f}")

---

Performance Considerations

Execution Time

• Single RAG evaluation: ~3-5 minutes for 10 questions
• Multi-model evaluation: ~5-10 minutes per model
• All RAGs evaluation: ~20-30 minutes total
• Comprehensive (all RAGs × all models): 1-2 hours depending on number of models

Cost Implications

Evaluation costs depend on:

• LLM model pricing (input/output tokens)
• Number of RAG queries
• Retrieved context length
• Answer generation length

Example costs per 10-question evaluation:

• GPT-4o: ~$0.04-0.06
• Claude-3.5-Sonnet: ~$0.05-0.08
• GPT-3.5-turbo: ~$0.01-0.02

Async vs Sync Mode

The evaluator automatically handles async context issues:

• Default: Async evaluation with 8 workers (faster)
• Fallback: Synchronous evaluation with 1 worker (more stable)
• Trigger: All-NaN metrics or async exceptions

---

Error Handling

Common Issues

1. All-NaN Metrics

Detected all-NaN async metric output. Switching to synchronous fallback...

Solution: Automatically handled by fallback mechanism.

2. Model Not Found

Model 'invalid-model' not found in registry. Skipping.

Solution: Use valid model keys from MODELS_REGISTRY.

3. Query Execution Failure

Error processing query 5: Connection timeout

Solution: Individual query failures are logged but don’t stop evaluation.

4. Unsupported RAG Type

ValueError: Unsupported RAG type: custom. Use 'rewriter', 'hybrid', ...

Solution: Use one of the 6 supported RAG types.

---

Dependencies

Required Packages:

• ragas - RAGAS evaluation framework
• datasets - HuggingFace datasets library
• pandas - Data manipulation
• numpy - Numerical operations

Internal Dependencies:

• src.rag.* - RAG system implementations
• src.common.model_provider - LLM model management
• src.common.pricing - Cost tracking utilities
• src.common.utils - Analysis export utilities

---

See Also

• Evaluation Overview - Comprehensive evaluation system guide
• Model Provider - LLM model configuration and management
• RAG Systems - Documentation for all RAG implementations
• Pricing - Cost tracking and analysis utilities