Scimax VS Code

✅ Overview

Scimax VS Code includes a powerful database and search system that indexes your org, markdown, and Jupyter notebook files. The database provides:

  • Full-text search with FTS5 (SQLite's Full-Text Search) and BM25 ranking

  • Semantic search using vector embeddings for meaning-based queries

  • Hybrid search combining keyword and semantic approaches

  • Advanced search with query expansion, weighted RRF, and LLM reranking (SOTA)

  • Structured queries for headings, TODOs, tags, properties, and links

  • Agenda views for scheduled items and deadlines

  • Code block search filtered by programming language

The database is built on SQLite (via @libsql/client) with support for vector similarity search, making it both fast and capable of sophisticated semantic queries.

✅ What Gets Indexed

File Types

The database indexes the file types listed in scimax.db.fileExtensions, which defaults to:

  • .org files - Org-mode documents

  • .md files - Markdown documents

Add others (e.g. "markdown") to that setting; anything that isn't .org is parsed as Markdown. The extension and the scimax CLI both read this one setting. See scimax.db.fileExtensions.

✅ Indexed Content

For each file, the database extracts and indexes:

✅ Headings

  • Heading text and level (*, **, ***, etc.)

  • TODO states (TODO, DONE, IN-PROGRESS, etc.)

  • Priority markers ([#A], [#B], [#C])

  • Tags (both direct and inherited)

  • Properties (CUSTOM_ID, CATEGORY, etc.)

  • Scheduling information (SCHEDULED, DEADLINE, CLOSED)

  • Line numbers for navigation

✅ Source Blocks

  • Programming language

  • Complete code content

  • Header arguments (:results, :exports, etc.)

  • Line numbers

  • For notebooks: cell indices

✅ Hashtags

  • Inline hashtags (e.g., #research, #todo)

  • Associated file paths

✅ Full Text

  • Complete document content for full-text search

  • Indexed with FTS5 virtual tables

  • Porter stemming and Unicode normalization

Here is a hashtag #FullTextSearch

✅ Text Chunks (for Semantic Search)

  • Document divided into ~2000 character chunks

  • 3-line overlap between chunks for context

  • Vector embeddings (if embedding service configured)

  • Line ranges for each chunk

✅ File Watching

The database automatically watches for file changes:

  • New files are indexed when created

  • Modified files are re-indexed (debounced with 500ms delay)

  • Deleted files are removed from the index

  • Changes are queued and processed sequentially

✅ Ignore Patterns

Some paths are always ignored. This baseline cannot be turned off, because indexing these directories produces duplicates of real notes rather than notes:

Pattern groupPatterns
Dependencies**/node_modules/**
Version control**/.git/**, **/.hg/**, **/.svn/**
Build output**/dist/**, **/build/**, **/__pycache__/**
Notebook checkouts**/.ipynb_checkpoints/**
System**/.Trash/**, **/Library/Caches/**
Editor state**/Code/User/History/**, **/Code/User/globalStorage/**,
**/Code/User/workspaceStorage/**, **/Code/agent-host/**,
the same paths for Code - Insiders, VSCodium and Cursor, and
**/.vscode/extensions/**
Claude app state**/Library/Application Support/Claude/**

Editor state matters most. VS Code's Local History keeps a timestamped copy of every edit under the original file extension, so a single project.org edited 200 times becomes 200 more .org files in the index — each with its own stale copy of your TODOs, deadlines and scheduled items, all of which then appear in the agenda.

scimax.db.exclude adds to this baseline; it does not replace it. That is deliberate: a VS Code settings array overrides the contributed default outright, so without a hard floor, adding one personal exclude pattern would silently re-enable indexing of node_modules and editor history.

If entries that the current rules would reject are already in the database — after an upgrade that adds a baseline pattern, or after you tighten scimax.db.exclude — clear them with:

scimax db prune --dry-run   # show what would go, grouped by pattern
scimax db prune             # remove them

scimax db sync will not do this for you: it keeps a database entry whose file still exists on disk, so that files added by other means are not lost.

The excludes apply to every way a file reaches the index, not only the directory scan: a save, a file-watcher event or a re-index of an excluded file is skipped, and if the file was indexed before, its entry is dropped.

cmd:workbench.action.openSettings2

Open Scimax Settings

Choosing What Gets Indexed

By default the index covers four sources: every registered scimax project (scimax.db.includeProjects), the folders open in VS Code (scimax.db.includeWorkspace), the journal (scimax.db.includeJournal) and scimax.db.include. Every folder you open in VS Code is registered as a project and stays one, so a few broad folders (~/Dropbox, ~/projects) can put far more in scope than you meant: cloud-drive mirrors, cloned repositories and build output, each with its own copies of TODOs that then show up in the agenda.

Your home directory and the folders above it (/Users, /) are never registered automatically when opened, and are never used as indexing roots even if an older version registered them, since that would index everything under ~.

scimax.db.include takes single .org / .md files as well as folders, in both the extension and the CLI (scimax db sync).

To index only what you choose, turn off the projects and workspace sources and list the folders and files you want:

{
  "scimax.db.include": ["~/Dropbox/projects", "~/Dropbox/proposals", "~/Dropbox/dashboard.org"],
  "scimax.db.exclude": ["**/_build/**", "**/_site/**", "**/venv/**"],
  "scimax.db.includeJournal": true,
  "scimax.db.includeProjects": false,
  "scimax.db.includeWorkspace": false
}

How a file gets into the index

RouteWhat is indexed
The indexed foldersScanned, watched and kept up to date (scimax.db.include, the journal, ...)
Saving a fileThat file, wherever it is (opening it is not enough); it stays until removed
A file changing on diskOnly if it is in the indexed folders or already in the index
scimax.db.excludeKept out by every route, except a file forced in by its exact path

So a file you work on outside your folders is indexed when you save it, and stays up to date after that; files that merely change on disk elsewhere (a git pull, a build, another program writing files) are left out.

A file beats a folder: Scimax: Always Index File (cmd:scimax.db.alwaysIndexFile, also on an editor tab's right-click menu) puts the file's exact path in scimax.db.include, and it is indexed even if its folder is excluded, e.g. one notes file in an ignored repository. Scimax: Don't Index File (cmd:scimax.db.ignoreFile) does the opposite (and takes the file out of scimax.db.include if it was there). The built-in excludes (node_modules, VS Code's history, ...) apply to every file.

Files outside the indexed folders stay until you remove them. Scimax: Remove Files Outside Indexed Folders... (cmd:scimax.db.removeOutOfScope) lists them by folder, all ticked; untick the ones to keep and press Enter to remove the rest from the index (the files themselves are not touched). It is also the way to tidy up after narrowing scimax.db.include. scimax db prune on the command line removes indexed files that match an exclude pattern.

Search View

C-c d v (cmd:scimax.search.view, Scimax: Search View) opens a search page beside your files, in the manner of the agenda view. Results update as you type, from the database, grouped by file with the lines that match.

The Query and the Modes

Type in the query field at the top. Three checkboxes choose how to search; any mix may be on, and a file found by several shows a badge for each:

ModeFinds
Full textLines with the query words (FTS5 syntax: AND, OR, NOT,
"phrases", prefix*), with stem variants marked
SemanticPassages close in meaning; needs embeddings (see Semantic
Search). Results arrive a moment after the others
HeadingsHeadings whose title holds every query word

Full text and Headings are on at first. The Semantic box is greyed out when semantic search is not available, and its tooltip says why. Files are ranked by reciprocal-rank fusion over the modes that found them.

The query and filters are remembered for the next time the view opens. With text selected on one line, C-c d v starts with that text as the query.

Filters

FilterHow it works
TagsClick a tag once to require it (✓), again to exclude it (✕),
and again to clear it. Inherited tags count
TODOSame clicks as tags, on the states in the database, plus
Any open and Any done. Required states are OR'd: a heading
needs one of them. Done means a done state in its file's
#+TODO line, or one in scimax.agenda.doneStates
PropertiesSTATUS=draft, AUTHOR: a key alone means the property is set;
KEY=value means its value contains the text (any case)
PathsComma-separated. Plain text matches anywhere in the path; a
glob (*.md, notes/**) matches the file name or the path
relative to its workspace folder

A line counts under the heading it sits in, so tag, TODO and property filters keep only lines in matching sections. With filters and an empty query, the view lists the headings that pass them, so requiring Any open with an empty query lists your open tasks. Clear filters resets them all, and Database… opens the Database View.

Results

The Sort menu at the bottom orders the files:

SortOrder
Recently modifiedNewest file first (the default)
Best matchReciprocal-rank fusion over the modes
File nameBy file name, with numbers in numeric order (b9 before b10)

Recently modified and File name reorder the best matches: the view fetches three times as many files as it shows, then sorts those, so a weak match in a recently edited file can still be left out of a long result list.

Each file shows its name, folder, when it was modified and match badges, then its hits: the line number, the outline of headings above the line, and the line with the matches marked. Five hits show per file, with Show N more for the rest, and More files at the bottom loads another 50 files. Saving an org or Markdown file runs the search again once the database has reindexed it.

KeyAction
↓ or Enter (query)Move into the results
n, j, ↓Next hit
p, k, ↑Previous hit (↑ on the first goes back up)
Enter, oOpen the file at the hit, beside the view
/, EscBack to the query
qClose the view

Click a hit to select it and double-click to open it.

✅ Full-Text Search

✅ Overview

Full-text search uses SQLite's FTS5 (Full-Text Search version 5) with:

  • BM25 ranking - Industry-standard relevance scoring

  • Porter stemming - Matches word variations (e.g., "run" matches "running")

  • Unicode normalization - Handles accented characters correctly

  • Snippet generation - Shows matching context with highlighting

✅ Usage

✅ Command

Run Scimax: Search All Files (FTS5) or use command scimax.db.search

cmd:scimax.db.search

✅ Query Syntax

FTS5 supports rich query syntax:

✅ Basic Queries

machine learning          # Match both words (in any order)
"machine learning"        # Match exact phrase
neural OR artificial      # Match either word

Boolean Operators

python AND jupyter        # Must contain both
python NOT tensorflow     # Contains python but not tensorflow
deep OR machine learning  # Contains "deep" or the phrase "machine learning"

Prefix Matching

comput*                   # Matches: computer, computing, computation
data*                     # Matches: data, database, dataset

Column Queries

title: introduction       # Search only in titles
content: python           # Search only in content

Proximity Queries

NEAR(neural network, 5)   # Words within 5 tokens of each other

Punctuation in Queries

Words with punctuation are searched as phrases, so citation keys, file names and hyphenated words can be typed as they are:

kitchin-2025-beyon-fourt  # Same as "kitchin-2025-beyon-fourt"
cite:&kitchin-2025        # Finds the key's text; the cite: prefix is ignored
e-mail c++                # Phrases "e mail" and "c"

Half-typed queries are repaired rather than reported as errors: an unclosed phrase or parenthesis is closed, a stray ) and a trailing AND/OR/NOT are dropped. If a query still is not valid, its words are searched as plain text. title: word and title:word are the same column filter.

Results

Search results include:

  • File path and basename

  • Line number

  • Preview snippet with <mark> tags highlighting matches

  • BM25 relevance score

  • Up to 100 results (configurable)

Example Searches

# Find all references to "gradient descent"
gradient descent

# Find Python-related TODO items
TODO python

# Find documents about data science or machine learning
"data science" OR "machine learning"

# Find recent mentions of TensorFlow (excluding Keras)
tensorflow NOT keras

# Find all documents with "introduction" in title
title: introduction

✅ Advanced Search (SOTA Pipeline)

Overview

Advanced search implements a state-of-the-art (SOTA) search pipeline inspired by modern search engines like qmd. It combines multiple techniques for maximum recall and precision:

  1. Query Expansion - Generates alternative query formulations

  2. Parallel Retrieval - Runs FTS5 and vector search concurrently

  3. Weighted Reciprocal Rank Fusion (RRF) - Combines results intelligently

  4. LLM Reranking - Uses AI to improve final ranking (optional)

Usage

Command

Run Scimax: Advanced Search (cmd:scimax.db.searchAdvanced)

When to Use

  • Complex research queries

  • When you want maximum recall

  • When hybrid search isn't finding relevant content

  • For important searches where accuracy matters more than speed

Query Expansion

Query expansion improves recall by generating alternative formulations of your query.

Pseudo-Relevance Feedback (PRF)

No LLM required - Analyzes top results to find related terms.

How it works:

  1. Runs initial search with original query

  2. Extracts key terms from top 5 results

  3. Creates expanded query with additional terms

  4. Searches again with expanded query

Example:

Original: "machine learning"
Expanded: "machine learning neural network training models"

LLM Query Expansion

Requires Ollama - Uses a language model to generate query variants.

How it works:

  1. Sends query to LLM (e.g., qwen3:1.7b)

  2. LLM generates 3 alternative phrasings

  3. All variants searched in parallel

  4. Original query gets 2× weight in ranking

Example:

Original: "database optimization"
Variants:
  - "improve SQL query performance"
  - "speed up database queries"
  - "index tuning for databases"

Weighted RRF

Standard RRF assigns scores based on rank position. Advanced search enhances this with:

Position Bonuses

Top-ranked results get bonuses:

  • Rank 1: +15% bonus

  • Rank 2: +10% bonus

  • Rank 3: +5% bonus

Original Query Weight

The original query's results receive 2× weight compared to expanded query results.

Score Normalization

Different backends produce different score ranges:

  • BM25: Negative values (normalized to 0-1)

  • Vector: Cosine distance (converted to similarity)

  • All scores normalized before fusion

LLM Reranking

Requires Ollama - Uses AI to score relevance after initial retrieval.

How It Works

  1. Takes top 30 candidates from RRF fusion

  2. LLM scores each document's relevance (0-10)

  3. Scores blended with retrieval scores using position-aware weights

Position-Aware Blending

High-confidence retrieval matches are preserved:

  • Ranks 1-3: 75% retrieval, 25% reranker

  • Ranks 4-10: 60% retrieval, 40% reranker

  • Ranks 11+: 40% retrieval, 60% reranker

Performance Considerations

Reranking adds latency (~1-2 seconds for 30 documents). Disable for fast searches.

Capabilities Check

Run Scimax: Show Search Capabilities (scimax.db.searchCapabilities) to see:

✓ Full-Text Search (FTS5/BM25) - Available
✓ Semantic/Vector Search - Available (Ollama)
✓ Query Expansion (PRF) - Available (no LLM required)
✗ Query Expansion (LLM) - Unavailable - check Ollama
✗ LLM Reranking - Unavailable - pull qwen3:0.6b

Graceful Degradation

Advanced search works without all features:

If UnavailableFallback
Vector searchFTS-only
LLM expansionPRF-only
RerankingSkip reranking
All LLM featuresEquivalent to hybrid search

Configuration

Search Mode

{
  "scimax.search.defaultMode": "hybrid",  // or "fast", "semantic", "advanced"
  "scimax.search.defaultLimit": 20
}

Query Expansion

{
  "scimax.search.queryExpansion.enabled": true,
  "scimax.search.queryExpansion.method": "prf",  // or "llm", "both"
  "scimax.search.queryExpansion.prfTopK": 5,
  "scimax.search.queryExpansion.prfTermCount": 5,
  "scimax.search.queryExpansion.llmModel": "qwen3:1.7b"
}

Reranking

{
  "scimax.search.reranking.enabled": false,  // Enable for better accuracy
  "scimax.search.reranking.model": "qwen3:0.6b",
  "scimax.search.reranking.topK": 30,
  "scimax.search.reranking.usePositionBlending": true
}

Hybrid Weights

{
  "scimax.search.hybrid.ftsWeight": 0.5,
  "scimax.search.hybrid.vectorWeight": 0.5,
  "scimax.search.hybrid.usePositionBonus": true,
  "scimax.search.hybrid.k": 60  // RRF constant
}

Caching

{
  "scimax.search.caching.enabled": true,
  "scimax.search.caching.ttlSeconds": 900,  // 15 minutes
  "scimax.search.caching.maxEntries": 500
}

Setting Up LLM Features

To enable query expansion and reranking with Ollama:

  1. Install Ollama: https://ollama.ai

  2. Start Ollama: ollama serve

  3. Pull models: #+BEGIN_SRC bash ollama pull qwen3:1.7b # For query expansion ollama pull qwen3:0.6b # For reranking #+END_SRC

  4. Enable in settings: #+BEGIN_SRC json { "scimax.search.queryExpansion.method": "both", "scimax.search.reranking.enabled": true } #+END_SRC

Performance Comparison

ModeSpeedRecallPrecisionWhen to Use
Fast<50msLowHighExact matches
Semantic~200msHighMediumConceptual queries
Hybrid~300msHighHighGeneral purpose
Advanced1-3sHighestHighestImportant searches

Structured Queries

Beyond free-text search, the database supports structured queries for specific content types.

✅ File Browser

Browse all indexed files:

✅ Command

Scimax: Browse Indexed Files (scimax.db.browseFiles). cmd:scimax.db.browseFiles

✅ Features

  • Lists all files in database

  • Shows last indexed date

  • Sorted by most recently indexed

  • Displays file type (org, md, ipynb)

✅ Agenda and Time Management

✅ Agenda View

The agenda shows scheduled items and deadlines:

✅ Command

Scimax: Show Agenda (scimax.db.agenda). cmd:scimax.db.agenda

✅ Time Periods

  • Next 2 weeks (default)

  • Next month

  • Next 3 months

  • All items (no time limit)

✅ Options

  • Include unscheduled TODOs

  • Filter by time range

  • Sorted by urgency (overdue first)

✅ Item Types

✅ Deadline

Items with DEADLINE timestamps:

* TODO Submit report
DEADLINE: <2026-01-20 Mon>

✅ Scheduled

Items with SCHEDULED timestamps:

* TODO Team meeting
SCHEDULED: <2026-01-15 Wed 14:00>

✅ Unscheduled TODOs

Items with TODO state but no scheduling

✅ Deadline View

Show only upcoming deadlines:

Command

Scimax: Show Deadlines (scimax.db.deadlines). cmd:scimax.db.deadlines

✅ Features

  • Next 2 weeks of deadlines

  • Overdue items highlighted

  • Shows days until deadline

  • Excludes DONE and CANCELLED

Display Format 

⚠️  Overdue: Submit TPS Report (3 days ago)
🔔 Today: Code Review
🔔 Tomorrow: Documentation Update
🔔 In 5 days: Project Demo

Date Formats

Scheduling in Org Files

# Simple date
SCHEDULED: <2026-01-20>

# Date with time
SCHEDULED: <2026-01-20 Mon 14:00>

# Date with time range
SCHEDULED: <2026-01-20 Mon 14:00-16:00>

# Deadline with warning period
DEADLINE: <2026-01-20 Mon -3d>

# Closed timestamp
CLOSED: [2026-01-13 Mon 10:30]

Relative Dates

+2w    # 2 weeks from now
+1m    # 1 month from now
+3d    # 3 days from now
+1y    # 1 year from now

Search Scope

Limit searches to specific directories:

Commands

Scimax: Set Search Scope (scimax.db.setScope)

Scope Types

All Files (Default)

  • Searches entire indexed database

  • Includes all workspace folders

  • Includes additional configured directories

Current Directory

  • Limits to active file's directory

  • Includes subdirectories

  • Useful for project-focused searches

Current Scope Indicator

The current scope is shown when setting scope:

Search scope: all
Search scope: directory (my-project)

Database View

C-c d i (cmd:scimax.db.view, Scimax: Database View) opens a page about the database. The Database panel in the sidebar has buttons for it and for the Search View.

Overview

Cards show the number of files (org and Markdown), headings, source blocks, chunks, the size of the database file, when a file was last indexed, whether semantic search is on (and how far the embedding queue has got), the search scope and the schema version. The path of the database file is below them, with Reveal to show it in the Finder or Explorer. The numbers update while a sync or the embedding queue runs; Refresh reads them again.

Actions

Buttons run the database commands: sync, remove files outside the indexed folders, cancel sync, rebuild, optimize, verify, back up, restore and clear (rebuild and clear ask first), and open the Search View, the file browser, the search scope picker and the embedding setup.

Folders and Recent Files

Folders with the most files lists the 15 folders with the most indexed files. Exclude adds a folder to scimax.db.exclude and offers to remove its files from the index now. Recently changed files opens a file on click.

Settings

Every scimax.db.* and scimax.search.* setting is listed by section (what gets indexed, embeddings, syncing, limits, search), with its description and a control: a checkbox, number or text field, a menu, or a list of entries with Add (and Browse… for scimax.db.include). Changes save as you make them, to the workspace settings when the setting is set there and to your user settings otherwise. A changed setting has a bar at its left and a Reset link; ⚙ opens it in VS Code's Settings editor. Open settings and settings.json at the top open the editor and the JSON file. Type in the filter box to narrow the list.

Database Management

Refreshing the Index

Sync Files (daily use)

Command: Scimax: Sync Files (scimax.db.reindex). cmd:scimax.db.reindex

  • Scans all configured directories (scimax.db.includeJournal, includeWorkspace, includeProjects, plus scimax.db.include)

  • Checks file modification times against the database

  • Only re-indexes changed files

  • This is the everyday command for picking up edits, especially when syncing across machines via Dropbox.

  • scimax.db.refresh is the canonical implementation; scimax.db.reindex is the user-facing alias.

Full Rebuild

Command: Scimax: Rebuild Database (from scratch) (scimax.db.rebuild). cmd:scimax.db.rebuild

  • Wipes the database and re-indexes everything

  • Use after schema changes or to recover from corruption

✅ Auto-Indexing

{
  "scimax.db.autoIndex": true
}

Warning: Disable for very large workspaces (>10,000 files) to prevent memory issues.

Indexing Sources

By default, the database indexes:

  • Journal directory (scimax.db.includeJournal: true)

  • Workspace folders (scimax.db.includeWorkspace: true)

  • Scimax projects (scimax.db.includeProjects: true)

Add additional directories (or individual .org/.md files) with scimax.db.include:

{
  "scimax.db.include": [
    "/home/user/research",
    "/home/user/notes",
    "/home/user/notes/inbox.org",
    "~/Documents/org"
  ]
}

The Scimax: Configure Agenda Files command (cmd:scimax.agenda.configure) appends to this list from a folder/file picker and refreshes the index in one step.

Optimization

✅ Command

Scimax: Optimize Database (scimax.db.optimize). cmd:scimax.db.optimize

Operations

  • Removes entries for deleted files

  • Runs VACUUM to reclaim space

  • Reports the file size before and after, for example Database optimized: 47.0 GB -> 1.9 GB

Earlier versions kept a vector index on the embeddings that semantic search never used. It could grow to tens of gigabytes and made semantic search take over a minute. The database now drops it when it is opened, which can take a few minutes once on a large database; run optimize afterwards to give the space back.

  • Should be run periodically (monthly)

VACUUM rewrites the whole file, so a large database needs free disk space about its own size and can take minutes. The CLI equivalent is scimax db optimize (close VS Code first). See CLI.

Clearing Database

Command

Scimax: Clear Database (scimax.db.clear)

Warning

This is destructive and requires confirmation:

  • Removes all indexed data

  • Clears embeddings

  • Resets statistics

  • Requires full reindex to restore

When to Clear

  • Database corruption

  • Major schema changes

  • Troubleshooting issues

  • Fresh start needed

Statistics

Command

Scimax: Show Database Stats (scimax.db.stats)

Information Displayed

Scimax DB: 127 files (98 org, 23 md, 6 ipynb),
1,234 headings, 456 code blocks, 789 links.
Semantic search: Enabled (243 chunks).
Last indexed: 2026-01-13 14:30:00

Stats Include

  • File count by type

  • Heading count

  • Code block count

  • Link count

  • Chunk count (for semantic search)

  • Embedding status

  • Last index timestamp

Performance Considerations

Indexing Performance

File Size

  • Small files (<100KB): ~10-50ms

  • Medium files (100KB-1MB): ~50-200ms

  • Large files (>1MB): ~200ms-1s

Batch Indexing

  • 100 small files: ~2-5 seconds

  • 1,000 small files: ~20-60 seconds

  • With embeddings: 2-5x slower

Optimization Tips

  1. Use ignore patterns for large non-content directories

  2. Disable auto-indexing for huge workspaces

  3. Index incrementally (only changed files)

  4. Run optimization monthly

Search Performance

Full-Text Search (FTS5)

  • Query time: 10-50ms (typical)

  • Scales well to 10,000+ files

  • BM25 scoring is highly optimized

  • Results returned in rank order

Database Size

Typical Sizes

100 files:      ~5-10 MB
1,000 files:    ~50-100 MB
10,000 files:   ~500 MB-1 GB

With Embeddings

+50-100% size increase for chunks and vectors
384-dim embeddings: ~1.5 KB per chunk
768-dim embeddings: ~3 KB per chunk
1536-dim embeddings: ~6 KB per chunk

Memory Usage

Indexing

  • Base: ~50-100 MB

  • Peak during large batch: ~200-500 MB

  • Embedding generation: +100-300 MB

Searching

  • FTS5: ~10-50 MB

  • Vector search: ~50-200 MB (loads embeddings)

  • Minimal memory footprint when idle

Scaling Guidelines

Small Workspace (<100 files)

  • Enable auto-indexing

  • Use any embedding provider

  • Full reindex in seconds

Medium Workspace (100-1,000 files)

  • Enable auto-indexing

  • Local or Ollama embeddings recommended

  • Full reindex in under a minute

Large Workspace (1,000-10,000 files)

  • Consider disabling auto-indexing

  • Ollama embeddings recommended

  • Reindex incrementally

Very Large Workspace (>10,000 files)

  • Disable auto-indexing (manual reindex)

  • Use selective directory indexing

  • Consider multiple smaller databases

  • Ollama with a fast model recommended

Configuration Reference

Database Settings

scimax.db.includeJournal

Type: boolean Default: true

Include journal directory in database indexing.

scimax.db.includeWorkspace

Type: boolean Default: true

Include workspace folders in database indexing.

scimax.db.includeProjects

Type: boolean Default: true

Include all scimax projects in database indexing.

scimax.db.include

Type: string[] Default: []

Additional directories or files to index (supports ~ for home directory).

{
  "scimax.db.include": [
    "/home/user/notes",
    "~/Documents/research"
  ]
}

scimax.db.exclude

Type: string[] Default: []

Additional patterns or paths to exclude from indexing (globs and absolute paths). These are added to the always-on baseline described under Ignore Patterns — you do not need to repeat **/node_modules/** and friends here.

{
  "scimax.db.exclude": [
    "**/temp/**",
    "**/*.backup.org",
    "~/notes/scratch.org"
  ]
}

`scimax.db.autoIndex'

Type: boolean Default: false

Automatically index workspace on activation. Disable for large workspaces.

{
  "scimax.db.autoIndex": true
}

Embedding Settings

`scimax.db.embeddingProvider'

Type: enum Values: "none" | "ollama" Default: "none"

Embedding provider for semantic search. With the default none, semantic search is off; set it to ollama (or run `Scimax: Configure Embedding Service') to turn it on.

{
  "scimax.db.embeddingProvider": "ollama"
}

`scimax.db.ollamaUrl'

Type: string Default: "http:localhost:11434"

Ollama server URL.

{
  "scimax.db.ollamaUrl": "http://localhost:11434"
}

`scimax.db.ollamaModel'

Type: string Default: "nomic-embed-text"

Ollama embedding model name.

{
  "scimax.db.ollamaModel": "nomic-embed-text"
}

Command Reference

All Database Commands (C-c d ?)

C-c d ? (cmd:scimax.db.help) lists every database command. The ones bound under C-c d show their key, and typing it runs the command at once, so C-c d ? s does what C-c d s does. The commands with no key follow under More: type to narrow the list and press Enter.

KeyAction
dDatabase hydra menu
sSearch all files
vSearch View
iDatabase View
hSearch headings
fBrowse indexed files
tShow TODOs
aShow agenda
-Code block, hashtag, tag, property, semantic, hybrid and
advanced search; deadlines; search scope
-Refresh, sync, rebuild, cancel sync, auto-scan, remove or
ignore files, stats, embeddings, optimize, verify, backup
and restore, clear

Search Commands

CommandDescription
scimax.db.searchFull-text search (FTS5)
scimax.db.searchSemanticSemantic search (vector)
scimax.db.searchHybridHybrid search (FTS + vector)
scimax.db.searchAdvancedAdvanced search (full pipeline)
scimax.db.searchCapabilitiesShow search capabilities
scimax.db.searchHeadingsSearch headings
scimax.db.searchByTagSearch by org tag
scimax.db.searchByPropertySearch by property value
scimax.db.searchBlocksSearch code blocks
scimax.db.searchHashtagsSearch by hashtag

View Commands

CommandDescription
scimax.db.showTodosShow TODO items
scimax.db.agendaShow agenda
scimax.db.deadlinesShow upcoming deadlines
scimax.db.browseFilesBrowse indexed files
scimax.search.viewSearch View
scimax.db.viewDatabase View

Management Commands

CommandDescription
scimax.db.refreshRefresh database (incremental)
scimax.db.rebuildRebuild database (from scratch)
scimax.db.optimizeOptimize database
scimax.db.clearClear database
scimax.db.statsShow database statistics
scimax.db.setScopeSet search scope
scimax.db.configureEmbeddingsConfigure embedding service
scimax.db.backupBackup database to file
scimax.db.restoreRestore database from file
scimax.db.rebuildRebuild database completely
scimax.db.verifyVerify database integrity

✅ Database Maintenance

✅ Backup and Restore

The database can be backed up and restored to prevent data loss and enable migration between machines.

✅ Backup

Command: Scimax: Backup Database (scimax.db.backup)

Creates a portable backup file containing:

  • All indexed file paths (not file contents)

  • Project information

  • Database metadata

Backup is stored in JSON format for portability.

# Example backup location
~/.scimax/backup-2026-01-22.json

✅ Restore

Command: Scimax: Restore Database (scimax.db.restore)

Restores database from a backup file:

  • Imports project list

  • Queues files for reindexing

  • Preserves original creation timestamps

Note: Actual file content must still be reindexed after restore.

✅ Database Rebuild

Command: Scimax: Rebuild Database (scimax.db.rebuild)

Completely rebuilds the database from scratch:

  • Drops and recreates all tables

  • Re-scans all configured directories

  • Regenerates all indexes

  • Regenerates embeddings (if configured)

Use when:

  • Database appears corrupted

  • Major schema changes after update

  • Switching embedding providers

  • Performance issues after many incremental updates

Options

OptionDescription
Full rebuildComplete reindex of all files
Projects onlyOnly rebuild project table

✅ Database Verification

Command: Scimax: Verify Database (scimax.db.verify)

Checks database integrity and freshness:

Checks Performed

  1. File existence - Verifies indexed files still exist on disk

  2. Modification time - Detects files modified since indexing

  3. Index integrity - Validates FTS5 and vector indexes

  4. Project validity - Checks project directories exist

Result Format

Database Verification Results:
- Total files: 127
- Missing files: 2
- Stale files: 5
- Projects: 8 (7 valid, 1 missing)
- Status: NEEDS_REINDEX

Status Values

StatusMeaning
OKDatabase is current and valid
NEEDS_REINDEXSome files are stale or missing
CORRUPTEDIndex integrity check failed

✅ Project Integration

The database now stores project information, integrated with the Projectile project manager:

Benefits

  • Projects persist across VS Code restarts

  • Shared project list between Projectile and Database

  • Fast project switching using indexed data

  • Projects can be associated with indexed files

Project Commands

Projects are managed through Projectile commands (C-c p), but the database provides the persistence layer.

See Projectile for project management commands.

Troubleshooting

Semantic Search Not Working

Problem

Semantic search returns no results, shows "unavailable", or displays an error.

If it works but is very slow (tens of seconds) or times out, check the database's size in Scimax: Show Database Stats. A file far larger than your notes is most likely the old vector index (see Optimization); open the database with this version, then run Scimax: Optimize Database.

First: Check Vector Search Support

Run Scimax: Show Database Stats (scimax.db.stats) to see the semantic search status:

Status MessageMeaning
Semantic search: Enabled (N chunks)Working, N chunks with embeddings
Semantic search: Ready (no embeddings)Supported, but no provider configured
Semantic search: Unavailable (error)Vector search not supported by database

Vector Search Unavailable

If you see "Semantic search: Unavailable", the libsql database doesn't support vector operations. This can happen if:

  • The libsql version doesn't include the vector functions (vector32, vector_distance_cos)

Workaround: Use full-text search (scimax.db.search) instead. FTS5 is always available and fast.

Embedding Provider Issues

If vector search is supported but not working:

  1. Check embedding provider is configured: scimax.db.embeddingProvider

  2. Test connection: Run Scimax: Configure Embedding Service

  3. Run Scimax: Sync Files after configuring embeddings (it queues files that have no embeddings)

  4. Check console for errors (Help: Toggle Developer Tools)

"Embeddings failed for N file(s)"

When the embedding queue finishes and some files could not be embedded, Scimax shows a warning:

Scimax: embeddings failed for 3 file(s). Last error: <message>

The last error usually points at the cause: Ollama not running, the model not pulled, or a dimension mismatch. Fix the cause and run Scimax: Sync Files again; files without embeddings are queued again.

Files Saved While Being Embedded

Embedding a file can take a few seconds. If you save the file during that time, it is re-indexed and queued for embedding again. The embeddings computed for the old version are discarded rather than stored, so the file ends up embedded from its latest content. This is expected and not an error.

Ollama Issues

  • Ensure Ollama is running: ollama serve

  • Pull model: ollama pull nomic-embed-text

  • Check URL is correct in settings

  • Test connection: curl http:localhost:11434apiembeddings

Search Returns No Results

Problem

Searches return empty results despite having files.

Solutions

  1. Run Scimax: Show Database Stats to check file count

  2. If files = 0, run Scimax: Sync Files (scimax.db.reindex)

  3. Check file extensions (.org, .md, .ipynb)

  4. Verify files aren't in ignored directories

  5. Check search scope is set to "All files"

Slow Indexing

Problem

Indexing takes very long or appears stuck.

Solutions

  1. Check workspace size (number of files)

  2. Add ignore patterns for large non-content directories

  3. Disable embedding generation if not needed

  4. Index directories incrementally

  5. Check disk I/O and available memory

Database Corruption

Problem

Errors mentioning "database is locked" or "disk I/O error".

Solutions

  1. Close other VS Code windows accessing same workspace

  2. Restart VS Code

  3. Run Scimax: Clear Database and reindex

  4. Check disk space is available

  5. Verify database file permissions

High Memory Usage

Problem

VS Code uses excessive memory during indexing or searching.

Solutions

  1. Disable auto-indexing

  2. Reduce number of indexed directories

  3. Use more aggressive ignore patterns

  4. Clear and reindex database

  5. Restart VS Code between large indexing operations

Examples and Workflows

Research Paper Management

# Organize papers with properties
* TODO Read: Attention Is All You Need
:PROPERTIES:
:CUSTOM_ID: vaswani2017attention
:AUTHOR: Vaswani et al.
:YEAR: 2017
:CATEGORY: research
:END:

#transformers #attention #nlp

SCHEDULED: <2026-01-15 Wed>

# Search by property
Property: CATEGORY
Value: research

# Search by hashtag
#nlp

# Semantic search
transformer architecture papers

Project Todo Management

# Use tags for organization
* TODO Implement login feature :work:backend:
DEADLINE: <2026-01-20 Mon>

* TODO Write API documentation :work:docs:
SCHEDULED: <2026-01-18 Sat>

# Search by tag
:work: -> shows all work items
:backend: -> shows backend tasks

# View agenda
Next 2 weeks -> prioritized by deadline

# Search TODOs
Filter by: TODO (in progress)

Code Snippet Library

# Store reusable code blocks
* Data Processing Utils

#+BEGIN_SRC python
def normalize_data(df):
    """Normalize numeric columns"""
    return (df - df.mean()) / df.std()
#+END_SRC

# Search code blocks
Language: python
Query: normalize

# Or semantic search
how to standardize dataframe columns

Personal Knowledge Base

# Use hybrid search for discovery
Query: "improve code performance"

Results will include:
- Exact matches: "code performance" articles
- Related concepts: optimization, profiling, caching
- Similar topics: algorithm efficiency, memory management

# Use properties for metadata
:PROPERTIES:
:CREATED: [2026-01-13 Mon 10:00]
:MODIFIED: [2026-01-13 Mon 15:30]
:CATEGORY: programming
:END:

Best Practices

Indexing Strategy

  1. Start selective - Index specific directories first

  2. Use ignore patterns - Exclude build artifacts and dependencies

  3. Index incrementally - Don't reindex everything on changes

  4. Schedule optimization - Run monthly for large databases

  5. Monitor statistics - Check file counts and sizes regularly

Search Strategy

  1. Start broad - Use semantic or hybrid search for exploration

  2. Refine with keywords - Switch to FTS for specific terms

  3. Use structured queries - Filter by tags/properties when possible

  4. Set scope appropriately - Narrow to directories for focused work

  5. Combine approaches - Use multiple search types for thorough research

Organization Tips

  1. Use consistent tags - Establish tag naming conventions

  2. Add properties - Include metadata for filtering

  3. Set schedules - Use SCHEDULED/DEADLINE for time management

  4. Include hashtags - Quick inline categorization

  5. Write descriptive headings - Better search results

Performance Tips

  1. Disable auto-indexing - For large workspaces (manual trigger)

  2. Choose appropriate embeddings - Balance quality vs. speed

  3. Limit result counts - Don't request thousands of results

  4. Use search scope - Narrow searches to relevant directories

  5. Cache frequent queries - Database has 15-minute result cache

Technical Architecture

Database Schema

Files Table

Tracks indexed files with modification tracking:

  • path, file_type, mtime, hash, size, indexed_at

Headings Table

Org/markdown headings with full metadata:

  • level, title, todo_state, priority

  • tags, inherited_tags, properties

  • `scheduled', `deadline', `closed'

  • line_number, begin_pos

Source Blocks Table

Code blocks with language and content:

  • `language', `content', `headers'

  • line_number, cell_index

Hashtags Table

Inline hashtags:

  • tag, file_path

Chunks Table

Text chunks for semantic search:

  • content, line_start, line_end

  • embedding (F32_BLOB vector)

FTS Content (Virtual Table)

Full-text search index:

  • file_path, title, content

  • Porter stemming, Unicode normalization

  • BM25 ranking support

Indexes

Performance indexes on:

  • headings.file_id, headings.todo_state

  • `headings.deadline', `headings.scheduled'

  • source_blocks.language

  • `hashtags.tag'

  • files.file_type

Parsers

Org Mode

UnifiedParserAdapter - Full AST parser compatible with org-element:

  • Recursive heading parsing with inheritance

  • Property drawer extraction

  • Timestamp parsing (scheduled, deadline, closed)

  • Source block with headers

  • Link extraction

Markdown

Simplified parser:

  • ATX heading syntax (`#')

  • Fenced code blocks

  • Inline links

Jupyter Notebooks

ipynbParser - Notebook-specific parser:

  • Markdown cells → headings and links

  • Code cells → source blocks

  • Cell indices tracked for navigation

  • Hashtags from markdown cells

Navigation