Project 07 · Developer Tools · AI

OllamaBench

Automated performance and quality evaluation for locally deployed Large Language Models. Multi-model comparison with hardware-aware analysis.

View on GitHub ↗

Overview

OllamaBench is a full-stack AI benchmarking platform built for evaluating locally-deployed Large Language Models (LLMs) through Ollama. It provides automated benchmark suites, multi-model comparison, and hardware-aware performance analysis.

The platform enables developers to run comprehensive quality and performance evaluations entirely offline, with interactive dashboards for analyzing results and exportable benchmark reports for sharing findings.

Built with a modern stack — React with TypeScript for the frontend, FastAPI with Python for the backend — OllamaBench bridges the gap between simple model testing and production-grade evaluation infrastructure.

Architecture

React + TypeScript Frontend ↓ FastAPI REST Backend ↓ Ollama Local Model Server ↓ Benchmark Suite Engine ↓ Hardware Metrics Collector ↓ Results Aggregation & Analysis ↓ Interactive Dashboard & Reports

Key Features

Automated Benchmark Suites: Pre-configured test suites covering reasoning, coding, creative writing, and factual accuracy.
Multi-Model Comparison: Side-by-side performance comparison across different locally deployed LLM models.
Hardware-Aware Analysis: Correlates model performance with system hardware metrics including CPU, GPU, and memory utilization.
Offline Quality Evaluation: No cloud dependency — all evaluation runs locally with full data privacy.
Interactive Dashboards: Rich visualization of benchmark results with filtering, sorting, and drill-down capabilities.
Exportable Reports: Generate shareable benchmark reports for team collaboration and model selection decisions.

Platform Interface

Dashboard Overview

Main benchmarking dashboard with model selection and test configuration.

Benchmark Results

Detailed performance metrics with response quality scoring.

Model Comparison

Side-by-side comparison of multiple model performances.

Test Suite Configuration

Pre-configured and custom benchmark suite management.

Tech Stack

Frontend

React TypeScript Vite Tailwind CSS

Backend

Python FastAPI Benchmark Runner

Database & Storage

SQLite JSON Metric Storage

AI & Infrastructure

Ollama Local LLMs Hardware Monitor