Harvey Model Cache

Version 1.0 — Complete guide to model capability caching in Harvey

Overview

Harvey’s Model Cache is a SQLite-backed database that stores capability metadata for Ollama models. This caching system significantly speeds up Harvey’s startup time by avoiding the need to re-probe every model on each launch.

What It Solves

Key Benefits

1. Fast Startup — Harvey starts instantly even with dozens of installed models
2. Offline Operation — Cached capability data available without Ollama running
3. Consistent Behavior — Model capabilities remain stable across sessions
4. Automatic Management — Cache is created and updated automatically
5. Probe Levels — Supports both fast (heuristic) and thorough (live test) probing

Quick Start

The model cache works automatically — no configuration required:

# First run: probes all installed models and caches results
harvey

# Subsequent runs: loads from cache, much faster
harvey

# Force re-probe a specific model
harvey> /ollama probe llama3.2:latest

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                      MODEL CACHE ARCHITECTURE                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────────┐     ┌─────────────────┐     ┌─────────────┐    │
│  │   Ollama        │     │  Model Cache    │     │   Harvey    │    │
│  │   Server        │     │  (SQLite DB)    │     │   Startup   │    │
│  │                 │     │                 │     │             │    │
│  │ /api/show       │     │ model_cache.db  │     │ Load cache  │    │
│  │ /api/embed      │     │                 │     │             │    │
│  └─────────────────┘     └─────────────────┘     └─────────────┘    │
│          │                       │                         │        │
│           │ Probe (fast/thorough)  │ Update cache                   │
│           ▼                       ▼                                 │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │                    ModelCapability                            │  │
│  │  - name, family, parameter_size, quantization                 │  │
│  │  - size_bytes, context_length                                 │  │
│  │  - supports_tools, supports_embed (CapabilityStatus enum)     │  │
│  │  - probe_level ("none", "fast", "thorough")                   │  │
│  │  - probed_at (timestamp)                                      │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Data Flow

1. Harvey Startup:
   • Open model cache database (agents/model_cache.db)
   • Load all cached ModelCapability entries
   • Use cached data for model selection UI
2. Model Probe (on demand):
   • Call FastProbeModel() or ThoroughProbeModel()
   • Fetch model metadata from Ollama /api/show
   • For thorough probe: test /api/embed endpoint
   • Store results in cache with timestamp
3. Cache Query:
   • Lookup model by name
   • Return cached ModelCapability or nil if not found
   • Display capability status (✓, —, ?)

Database Schema

SQLite Schema

CREATE TABLE IF NOT EXISTS model_capabilities (
    name                   TEXT PRIMARY KEY,
    family                 TEXT    NOT NULL DEFAULT '',
    parameter_size         TEXT    NOT NULL DEFAULT '',
    quantization           TEXT    NOT NULL DEFAULT '',
    size_bytes             INTEGER NOT NULL DEFAULT 0,
    context_length         INTEGER NOT NULL DEFAULT 0,
    supports_tools         INTEGER NOT NULL DEFAULT -1,
    supports_embed         INTEGER NOT NULL DEFAULT -1,
    supports_tagged_blocks INTEGER NOT NULL DEFAULT -1,
    tool_mode              TEXT    NOT NULL DEFAULT '',
    probe_level            TEXT    NOT NULL DEFAULT 'none',
    probed_at              DATETIME DEFAULT CURRENT_TIMESTAMP
);

PRAGMA foreign_keys = ON;
PRAGMA journal_mode = WAL;

Columns added after the initial schema are migrated automatically via ALTER TABLE … ADD COLUMN when OpenModelCache is called. Existing rows receive the column default (-1 for integer capabilities, '' for tool_mode).

Table Columns

Indexes

• Primary Key: name — fast lookup by model name
• No secondary indexes — table is small enough for full scans

Go API Reference

Types

CapabilityStatus

An enum representing whether a model capability is confirmed, denied, or unknown.

type CapabilityStatus int

const (
    CapUnknown CapabilityStatus = -1  // Not yet probed
    CapNo      CapabilityStatus = 0   // Confirmed absent
    CapYes     CapabilityStatus = 1   // Confirmed present
)

String Representation: - CapUnknown → “?” - CapNo → “—”
- CapYes → “✓”

ModelCapability

Holds all cached metadata for a single model.

type ModelCapability struct {
    Name                 string           // Full model identifier
    Family               string           // Model family
    ParameterSize        string           // Human-readable size
    Quantization         string           // Quantization level
    SizeBytes            int64            // Bytes on disk
    ContextLength        int              // Context window in tokens
    SupportsTools        CapabilityStatus // Tool/function calling support
    SupportsEmbed        CapabilityStatus // Embedding support
    SupportsTaggedBlocks CapabilityStatus // Path-tagged code block support
    ToolMode             string           // Explicit execution strategy; see ToolMode* constants
    ProbeLevel           string           // "none", "fast", or "thorough"
    ProbedAt             time.Time        // When last probed
}

ToolMode constants

The ToolMode field overrides the auto-detected SupportsTools capability when choosing a tool-execution strategy. The zero value ("") means auto.

const (
    ToolModeAuto       = ""           // Use SupportsTools==CapYes to decide
    ToolModeStructured = "structured" // Force OpenAI tool_calls (RunToolLoop)
    ToolModeProse      = "prose"      // Force JSON-fence prose fallback
    ToolModeInject     = "inject"     // Pre-inject file content; no tool calls
    ToolModeNone       = "none"       // Plain text; no tools, no injection
)

Set via /model mode inside Harvey. Re-probing a model does not overwrite a user-set ToolMode — only an explicit /model mode command changes it.

ModelCache

The main handle for the model cache database.

type ModelCache struct {
    db   *sql.DB
    path string
}

// OpenModelCache opens (or creates) the model cache database.
// customPath overrides the default location (agents/model_cache.db).
func OpenModelCache(ws *Workspace, customPath string) (*ModelCache, error)

// Path returns the absolute path of the cache file.
func (mc *ModelCache) Path() string

// Close releases the database connection.
func (mc *ModelCache) Close() error

Methods

Get(name string) (*ModelCapability, error)

Returns the cached capability for the named model, or nil if not found.

cap, err := mc.Get("llama3.2:latest")
if cap == nil {
    // Model not in cache
}
if cap.SupportsTools == CapYes {
    fmt.Println("Model supports tools")
}

Parameters: - name — Full model identifier (e.g., “llama3.2:latest”)

Returns: - *ModelCapability — cached entry; nil if not found - error — non-nil on database error (not on missing row)

Set(cap *ModelCapability) error

Upserts a ModelCapability into the cache. Existing entries are completely replaced.

cap := &ModelCapability{
    Name:          "llama3.2:latest",
    Family:        "llama",
    SupportsTools: CapYes,
    SupportsEmbed: CapNo,
    ProbeLevel:    "fast",
    ProbedAt:      time.Now(),
}
err := mc.Set(cap)

Parameters: - cap — Capability record to store

Returns: - error — non-nil on database write failure

Delete(name string) error

Removes the cache entry for the named model.

err := mc.Delete("old-model:tag")

Parameters: - name — Full model name

Returns: - error — non-nil on database write failure

All() ([]ModelCapability, error)

Returns all cached model capabilities, ordered by name.

allCaps, err := mc.All()
for _, cap := range allCaps {
    fmt.Printf("%s (%s): tools=%s embed=%s\n",
        cap.Name, cap.Family, cap.SupportsTools, cap.SupportsEmbed)
}

Returns: - []ModelCapability — all cached entries; empty slice if cache is empty - error — non-nil on database read failure

Capability Probing

Harvey uses two levels of capability probing, implemented in ollama.go:

Fast Probe (FastProbeModel)

Uses heuristics to determine capabilities from the model’s /api/show response:

Tool Support Detection: 1. Checks the Capabilities array from /api/show (authoritative on Ollama ≥ 0.3) 2. Falls back to checking for known tool-call template markers: - {% if tools %} (Llama 3, Granite - Jinja2) - [TOOL_CALLS], [AVAILABLE_TOOLS] (Mistral, Ministral) - <tool_call>, ✿FUNCTION✿ (Qwen 2.x variants) - <function_calls> (Gemma 4 and others)

Embedding Support Detection: - Checks if model name contains known embedding-model keywords: - embed, e5-, bge-, gte-, minilm, nomic, mxbai, jina

Advantages: - Single API call (/api/show) - Fast execution - No embedding model test required

Limitations: - Embedding detection is heuristic-based - May produce false positives for models with embedding keywords in name

Thorough Probe (ThoroughProbeModel)

First runs FastProbeModel, then makes a live /api/embed request to confirm embedding support:

1. Calls FastProbeModel for initial capability detection
2. Sends a test embedding request to /api/embed
3. If successful with non-empty embeddings array → SupportsEmbed = CapYes
4. If any error or empty response → SupportsEmbed = CapNo

Advantages: - Definitive embedding support confirmation - Accurate results

Limitations: - Requires embedding model to be loaded in Ollama - Slower (additional API call) - Still uses heuristics for tool support

Probe Levels Summary

Usage Patterns

Automatic Probing on Startup

Harvey automatically probes models when needed:

// In harvey initialization
if model, ok := knownModels[name]; !ok {
    cap, err := FastProbeModel(ctx, ollamaURL, name)
    if err == nil {
        cache.Set(cap)
    }
}

Manual Probing

# Probe a specific model
harvey> /ollama probe llama3.2:latest

# Probe all installed models
harvey> /ollama probe --all

Setting Tool Mode

Override the auto-detected tool execution strategy for a model:

# Show the current mode for the active model
harvey> /model mode

# Force file-injection mode for a small model that ignores the tools schema
harvey> /model mode phi4:latest inject

# Force structured tool_calls for a model whose probe result was wrong
harvey> /model mode granite4.1:8b structured

# Reset to auto-detection (clears the override)
# Note: /model mode auto is planned for v0.0.16; until then, use:
sqlite3 agents/model_cache.db "UPDATE model_capabilities SET tool_mode='' WHERE name='phi4:latest'"

Tool modes survive /ollama probe — re-probing a model does not overwrite a mode set via /model mode.

Checking Model Capabilities

# List all models with their capabilities
harvey> /ollama list

# The output shows:
# - Model name and family
# - Parameter size and quantization
# - Context length
# - Tool support (✓, —, ?)
# - Embedding support (✓, —, ?)

Programmatic Access

// Open the cache
mc, err := OpenModelCache(ws, "")
defer mc.Close()

// Get a specific model's capabilities
cap, err := mc.Get("llama3.2:latest")
if cap != nil {
    fmt.Printf("Tools: %s, Embed: %s\n", cap.SupportsTools, cap.SupportsEmbed)
}

// Iterate all cached models
all, _ := mc.All()
for _, c := range all {
    if c.SupportsEmbed == CapYes {
        fmt.Println(c.Name, "supports embeddings")
    }
}

Configuration

Database Location

Default: agents/model_cache.db in the workspace root

Custom Path:

mc, err := OpenModelCache(ws, "custom/path/model_cache.db")

YAML Configuration:

# In harvey.yaml
model_cache:
  path: custom/path/model_cache.db

SQLite Settings

The database is configured with: - Journal Mode: WAL (Write-Ahead Logging) - Foreign Keys: ON - Max Connections: 1 (prevents locking issues)

Best Practices

When to Use Each Probe Level

1. Fast Probe (Default):
   • Use for initial model discovery
   • Use when embedding model is known from name
   • Use for quick capability checks
2. Thorough Probe:
   • Use when embedding capability is uncertain
   • Use before relying on embedding functionality
   • Use when fast probe returns unknown for embedding

Cache Management

1. Let Harvey manage the cache:
   • Cache is automatically created on first use
   • Probing happens automatically when needed
2. Clear cache when needed:
   • If Ollama is updated, consider clearing the cache
   • If model definitions change, re-probe specific models
3. Backup the cache:
   • agents/model_cache.db contains valuable metadata
   • Backup before major Ollama upgrades

Model Name Format

• Use full model identifiers including tags:
  • ✅ llama3.2:latest
  • ✅ nomic-embed-text:latest
  • ✅ granite-code:3b
  • ❌ llama3.2 (missing tag)
  • ❌ nomic-embed-text (missing tag)
• Tags matter: different tags may have different capabilities

Troubleshooting

Common Issues

Force Re-probe

To force a fresh probe of a model:

// Delete the old entry
mc.Delete("model:tag")

// Run a new probe
cap, err := FastProbeModel(ctx, ollamaURL, "model:tag")
if err == nil {
    mc.Set(cap)
}

Database Corruption

If the database file is corrupted:

# Remove the corrupted file
rm agents/model_cache.db

# Harvey will create a new one on next startup
# All capabilities will be re-probed

Verify Cache Contents

# Check the database directly
sqlite3 agents/model_cache.db "SELECT name, probe_level, supports_tools, supports_embed FROM model_capabilities"

# Count entries
sqlite3 agents/model_cache.db "SELECT COUNT(*) FROM model_capabilities"

Performance

Cache Size

• Each entry: ~200-500 bytes
• 100 models: ~20-50KB
• 1000 models: ~200-500KB
• Memory: Only loaded entries are in memory

Query Performance

• Lookup by name: O(log n) via primary key index
• Full scan (All()): O(n) but fast for typical model counts
• Typical lookup: < 1ms
• Typical full scan: < 10ms for 100 models

Probe Performance

Migration Guide

From No Cache (v0.1 and earlier)

Harvey v0.2+ includes automatic model caching:

1. Start Harvey — cache is created automatically
2. Models are probed as they’re discovered
3. Cache grows as you use more models

From Old Cache Format

The cache schema is automatically migrated on open: - Missing columns are added - Indexes are created - Existing data is preserved

No manual migration needed.

Related Files

See Also

• CONFIGURATION.md — Configuration file reference
• OLLAMA.md — Ollama integration guide (if exists)
• ARCHITECTURE.md — Technical architecture
• RAG_Support_Design.md — RAG design document with embedding model info

Documentation generated from model_cache.go and ollama.go source code. Version 1.0.