✅ Headings

• Heading text and level (*, **, ***, etc.)

• TODO states (TODO, DONE, IN-PROGRESS, etc.)

• Priority markers ([#A], [#B], [#C])

• Tags (both direct and inherited)

• Properties (CUSTOM_ID, CATEGORY, etc.)

• Scheduling information (SCHEDULED, DEADLINE, CLOSED)

• Line numbers for navigation

✅ Source Blocks

• Programming language

• Complete code content

• Header arguments (:results, :exports, etc.)

• Line numbers

• For notebooks: cell indices

✅ Links

• Link type (file, http, https, id, etc.)

• Target path or URL

• Optional description text

• Line number

✅ Hashtags

• Inline hashtags (e.g., #research, #todo)

• Associated file paths

✅ Full Text

• Complete document content for full-text search

• Indexed with FTS5 virtual tables

• Porter stemming and Unicode normalization

Here is a hashtag #FullTextSearch

✅ Text Chunks (for Semantic Search)

• Document divided into ~2000 character chunks

• 3-line overlap between chunks for context

• Vector embeddings (if embedding service configured)

• Line ranges for each chunk

✅ Command

Run Scimax: Search All Files (FTS5) or use command scimax.db.search

[[cmd:scimax.db.search]]

✅ Query Syntax

FTS5 supports rich query syntax:

machine learning          # Match both words (in any order)
"machine learning"        # Match exact phrase
neural OR artificial      # Match either word

python AND jupyter        # Must contain both
python NOT tensorflow     # Contains python but not tensorflow
deep OR machine learning  # Contains "deep" or the phrase "machine learning"

comput*                   # Matches: computer, computing, computation
data*                     # Matches: data, database, dataset

title: introduction       # Search only in titles
content: python           # Search only in content

NEAR(neural network, 5)   # Words within 5 tokens of each other

Checking Availability

Run Scimax: Show Database Stats (scimax.db.stats) to see:

• Whether vector search is supported

• Number of embeddings stored

• Any error messages

If vector search is unavailable, the stats will show:

Semantic search: Unavailable (vector search not supported)

Fallback to Full-Text Search

If semantic search is unavailable, use full-text search (`scimax.db.search') instead. FTS5 is always available and very fast for keyword-based queries.

Ollama

• Pros: Free, private, local control, high quality

• Cons: Requires Ollama installed and running

• Models:

• Setup: Install Ollama, then ollama pull nomic-embed-text

✅ Interactive Setup

Run Scimax: Configure Embedding Service (scimax.db.configureEmbeddings)

This wizard will:

1. Let you choose a provider

2. Select a model

3. Test the connection

4. Update your settings

5. Prompt you to reindex files

✅ Manual Configuration

Add to your VS Code settings (settings.json):

{
  "scimax.db.embeddingProvider": "ollama",
  "scimax.db.ollamaUrl": "http://localhost:11434",
  "scimax.db.ollamaModel": "nomic-embed-text"
}

✅ Command

Run Scimax: Semantic Search or use command scimax.db.searchSemantic

✅ Query Examples

Semantic search uses natural language:

# Conceptual searches
explain neural network architecture
how to optimize database queries
project management best practices

# Related concept discovery
clustering algorithms    # Finds: k-means, hierarchical, DBSCAN
data visualization       # Finds: matplotlib, charts, plots, graphs

# Question answering
what is gradient descent
how does attention mechanism work

Command

Run Scimax: Hybrid Search or use command scimax.db.searchHybrid

How It Works

1. Query runs through both FTS5 and vector search

2. Results from each are ranked

3. RRF algorithm combines rankings:

4. Final results sorted by combined score

Weights

Default weights are 50/50, but can be adjusted:

// Internal API (for extension developers)
db.searchHybrid(query, {
  limit: 20,
  ftsWeight: 0.5,      // 50% weight on keywords
  vectorWeight: 0.5    // 50% weight on semantics
});

When to Use Hybrid

• Default choice for general searches

• When you want both exact matches and related concepts

• When query has both specific terms and broad concepts

• Example: "python sklearn classification accuracy" - finds both sklearn-specific code and general ML accuracy discussions

Command

Run Scimax: Advanced Search ([[cmd:scimax.db.searchAdvanced]])

When to Use

• Complex research queries

• When you want maximum recall

• When hybrid search isn't finding relevant content

• For important searches where accuracy matters more than speed

Pseudo-Relevance Feedback (PRF)

How it works:

1. Runs initial search with original query

2. Extracts key terms from top 5 results

3. Creates expanded query with additional terms

4. Searches again with expanded query

Example:

Original: "machine learning"
Expanded: "machine learning neural network training models"

LLM Query Expansion

How it works:

1. Sends query to LLM (e.g., qwen3:1.7b)

2. LLM generates 3 alternative phrasings

3. All variants searched in parallel

4. Original query gets 2× weight in ranking

Example:

Original: "database optimization"
Variants:
  - "improve SQL query performance"
  - "speed up database queries"
  - "index tuning for databases"

Position Bonuses

Top-ranked results get bonuses:

• Rank 1: +15% bonus

• Rank 2: +10% bonus

• Rank 3: +5% bonus

Original Query Weight

The original query's results receive 2× weight compared to expanded query results.

Score Normalization

Different backends produce different score ranges:

• BM25: Negative values (normalized to 0-1)

• Vector: Cosine distance (converted to similarity)

• All scores normalized before fusion

How It Works

1. Takes top 30 candidates from RRF fusion

2. LLM scores each document's relevance (0-10)

3. Scores blended with retrieval scores using position-aware weights

Position-Aware Blending

High-confidence retrieval matches are preserved:

• Ranks 1-3: 75% retrieval, 25% reranker

• Ranks 4-10: 60% retrieval, 40% reranker

• Ranks 11+: 40% retrieval, 60% reranker

Performance Considerations

Reranking adds latency (~1-2 seconds for 30 documents). Disable for fast searches.

Search Mode

{
  "scimax.search.defaultMode": "hybrid",  // or "fast", "semantic", "advanced"
  "scimax.search.defaultLimit": 20
}

Query Expansion

{
  "scimax.search.queryExpansion.enabled": true,
  "scimax.search.queryExpansion.method": "prf",  // or "llm", "both"
  "scimax.search.queryExpansion.prfTopK": 5,
  "scimax.search.queryExpansion.prfTermCount": 5,
  "scimax.search.queryExpansion.llmModel": "qwen3:1.7b"
}

Reranking

{
  "scimax.search.reranking.enabled": false,  // Enable for better accuracy
  "scimax.search.reranking.model": "qwen3:0.6b",
  "scimax.search.reranking.topK": 30,
  "scimax.search.reranking.usePositionBlending": true
}

Hybrid Weights

{
  "scimax.search.hybrid.ftsWeight": 0.5,
  "scimax.search.hybrid.vectorWeight": 0.5,
  "scimax.search.hybrid.usePositionBonus": true,
  "scimax.search.hybrid.k": 60  // RRF constant
}

Caching

{
  "scimax.search.caching.enabled": true,
  "scimax.search.caching.ttlSeconds": 900,  // 15 minutes
  "scimax.search.caching.maxEntries": 500
}

Command

Scimax: Search Headings (scimax.db.searchHeadings)

Features

• Search by heading text

• Filter by TODO state

• Filter by tag

• Shows heading level, tags, TODO state

• Displays deadlines and scheduled dates

Example Use Cases

# Find all headings about Python
python

# Find TODOs with specific tag (use Tag Search)
:work:

# Browse document structure
<empty query>

Command

Scimax: Search By Tag (scimax.db.searchByTag)

Features

• Lists all tags found in indexed files

• Shows heading count per tag

• Supports both direct and inherited tags

• Tags displayed as :tagname:

Command

Scimax: Search By Property (scimax.db.searchByProperty)

Common Properties

CUSTOM_ID    # Unique identifiers for linking
ID           # Auto-generated UUIDs
CATEGORY     # Classification
CREATED      # Creation timestamp
MODIFIED     # Last modification time

Examples

# Find all entries with CATEGORY property
Property: CATEGORY
Value: <empty>

# Find entries with specific CATEGORY
Property: CATEGORY
Value: research

# Find entries with CUSTOM_ID
Property: CUSTOM_ID
Value: <empty or specific>

Command

Scimax: Show TODOs (scimax.db.showTodos)

Features

• Lists all TODO items across workspace

• Filter by state (TODO, DONE, IN-PROGRESS, etc.)

• Shows priority, tags, scheduling

• Excludes DONE and CANCELLED by default in other views

Common TODO States

TODO          # Not started
IN-PROGRESS   # Currently working on
NEXT          # Up next
WAIT          # Waiting on something
DONE          # Completed
CANCELLED     # Abandoned

✅ Command

Scimax: Search Code Blocks (scimax.db.searchBlocks). [[cmd:scimax.db.searchBlocks]]

✅ Features

• Filter by programming language

• Optional text search within code

• Shows first line of code as preview

• Includes both org files and notebook cells

✅ Example Workflow

1. Select language (or "All languages")
2. Optionally enter search text
3. Browse matching code blocks
4. Jump to file location

✅ Command

Scimax: Search Hashtags (scimax.db.searchHashtags). [[cmd:scimax.db.searchHashtags]]

✅ Features

• Lists all hashtags found (`#tagname')

• Shows file count per hashtag

• Displays files containing selected hashtag

• Case-insensitive matching

✅ Hashtag Format

# In your documents
This is a #research note about #machinelearning

# Database indexes as
research
machinelearning

✅ Command

Scimax: Browse Indexed Files (scimax.db.browseFiles). [[cmd:scimax.db.browseFiles]]

✅ Features

• Lists all files in database

• Shows last indexed date

• Sorted by most recently indexed

• Displays file type (org, md, ipynb)

✅ Command

Scimax: Show Agenda (scimax.db.agenda). [[cmd:scimax.db.agenda]]

✅ Time Periods

• Next 2 weeks (default)

• Next month

• Next 3 months

• All items (no time limit)

✅ Options

• Include unscheduled TODOs

• Filter by time range

• Sorted by urgency (overdue first)

✅ Item Types

,* TODO Submit report
DEADLINE: <2026-01-20 Mon>

,* TODO Team meeting
SCHEDULED: <2026-01-15 Wed 14:00>

Command

Scimax: Show Deadlines (scimax.db.deadlines). [[cmd:scimax.db.deadlines]]

✅ Features

• Next 2 weeks of deadlines

• Overdue items highlighted

• Shows days until deadline

• Excludes DONE and CANCELLED

Display Format

⚠️  Overdue: Submit TPS Report (3 days ago)
🔔 Today: Code Review
🔔 Tomorrow: Documentation Update
🔔 In 5 days: Project Demo

Scheduling in Org Files

# Simple date
SCHEDULED: <2026-01-20>

# Date with time
SCHEDULED: <2026-01-20 Mon 14:00>

# Date with time range
SCHEDULED: <2026-01-20 Mon 14:00-16:00>

# Deadline with warning period
DEADLINE: <2026-01-20 Mon -3d>

# Closed timestamp
CLOSED: [2026-01-13 Mon 10:30]

Relative Dates

+2w    # 2 weeks from now
+1m    # 1 month from now
+3d    # 3 days from now
+1y    # 1 year from now

All Files (Default)

• Searches entire indexed database

• Includes all workspace folders

• Includes additional configured directories

Current Directory

• Limits to active file's directory

• Includes subdirectories

• Useful for project-focused searches

Full Reindex

Command: Scimax: Reindex Files (scimax.db.reindex). [[cmd:scimax.db.reindex]]

• Scans all workspace folders

• Checks file modification times

• Only reindexes changed files

• Shows progress notification

• Reports statistics on completion

✅ Auto-Indexing

{
  "scimax.db.autoIndex": true
}

Warning: Disable for very large workspaces (>10,000 files) to prevent memory issues.

✅ Indexing Sources

By default, the database indexes:

• Journal directory (scimax.db.includeJournal: true)

• Workspace folders (scimax.db.includeWorkspace: true)

• Scimax projects (scimax.db.includeProjects: true)

Add additional directories with scimax.db.include:

{
  "scimax.db.include": [
    "/home/user/research",
    "/home/user/notes",
    "~/Documents/org"
  ]
}

✅ Command

Scimax: Optimize Database (scimax.db.optimize). [[cmd:scimax.db.optimize]]

Operations

• Removes entries for deleted files

• Runs VACUUM to reclaim space

• Rebuilds indexes for performance

• Should be run periodically (monthly)

Command

Scimax: Clear Database (scimax.db.clear)

Warning

This is destructive and requires confirmation:

• Removes all indexed data

• Clears embeddings

• Resets statistics

• Requires full reindex to restore

When to Clear

• Database corruption

• Major schema changes

• Troubleshooting issues

• Fresh start needed

Command

Scimax: Show Database Stats (scimax.db.stats)

Information Displayed

Scimax DB: 127 files (98 org, 23 md, 6 ipynb),
1,234 headings, 456 code blocks, 789 links.
Semantic search: Enabled (243 chunks).
Last indexed: 2026-01-13 14:30:00

Stats Include

• File count by type

• Heading count

• Code block count

• Link count

• Chunk count (for semantic search)

• Embedding status

• Last index timestamp

File Size

• Small files (<100KB): ~10-50ms

• Medium files (100KB-1MB): ~50-200ms

• Large files (>1MB): ~200ms-1s

Batch Indexing

• 100 small files: ~2-5 seconds

• 1,000 small files: ~20-60 seconds

• With embeddings: 2-5x slower

Optimization Tips

1. Use ignore patterns for large non-content directories

2. Disable auto-indexing for huge workspaces

3. Index incrementally (only changed files)

4. Run optimization monthly

Full-Text Search (FTS5)

• Query time: 10-50ms (typical)

• Scales well to 10,000+ files

• BM25 scoring is highly optimized

• Results returned in rank order

Semantic Search

• Query time: 50-500ms depending on provider

• Local embeddings: slower but private

• Ollama: moderate speed

• OpenAI: fastest but requires network

Hybrid Search

• Query time: Combined FTS + vector time

• Typically 100-600ms

• Runs searches in parallel

• RRF fusion adds ~10ms

Typical Sizes

100 files:      ~5-10 MB
1,000 files:    ~50-100 MB
10,000 files:   ~500 MB-1 GB

With Embeddings

+50-100% size increase for chunks and vectors
384-dim embeddings: ~1.5 KB per chunk
768-dim embeddings: ~3 KB per chunk
1536-dim embeddings: ~6 KB per chunk

Indexing

• Base: ~50-100 MB

• Peak during large batch: ~200-500 MB

• Embedding generation: +100-300 MB

Searching

• FTS5: ~10-50 MB

• Vector search: ~50-200 MB (loads embeddings)

• Minimal memory footprint when idle

Small Workspace (<100 files)

• Enable auto-indexing

• Use any embedding provider

• Full reindex in seconds

Medium Workspace (100-1,000 files)

• Enable auto-indexing

• Local or Ollama embeddings recommended

• Full reindex in under a minute

Large Workspace (1,000-10,000 files)

• Consider disabling auto-indexing

• Ollama embeddings recommended

• Reindex incrementally

Very Large Workspace (>10,000 files)

• Disable auto-indexing (manual reindex)

• Use selective directory indexing

• Consider multiple smaller databases

• Ollama with a fast model recommended

scimax.db.includeJournal

Type: boolean Default: true

Include journal directory in database indexing.

scimax.db.includeWorkspace

Type: boolean Default: true

Include workspace folders in database indexing.

scimax.db.includeProjects

Type: boolean Default: true

Include all scimax projects in database indexing.

scimax.db.include

Type: string[] Default: []

Additional directories or files to index (supports ~ for home directory).

{
  "scimax.db.include": [
    "/home/user/notes",
    "~/Documents/research"
  ]
}

scimax.db.exclude

Type: string[] Default: ["**/node_modules/**", "**/.git/**", "**/dist/**", "**/build/**"]

Patterns or paths to exclude from indexing (globs and absolute paths).

{
  "scimax.db.exclude": [
    "**/node_modules/**",
    "**/.git/**",
    "**/dist/**",
    "**/temp/**",
    "**/*.backup.org",
    "~/notes/scratch.org"
  ]
}

`scimax.db.autoIndex'

Type: boolean Default: false

Automatically index workspace on activation. Disable for large workspaces.

{
  "scimax.db.autoIndex": true
}

`scimax.db.embeddingProvider'

Type: enum Values: "none" | "ollama" Default: "ollama"

Embedding provider for semantic search.

{
  "scimax.db.embeddingProvider": "ollama"
}

`scimax.db.ollamaUrl'

Type: string Default: "http://localhost:11434"

Ollama server URL.

{
  "scimax.db.ollamaUrl": "http://localhost:11434"
}

`scimax.db.ollamaModel'

Type: string Default: "nomic-embed-text"

Ollama embedding model name.

{
  "scimax.db.ollamaModel": "nomic-embed-text"
}

✅ Backup

Command: Scimax: Backup Database (scimax.db.backup)

Creates a portable backup file containing:

• All indexed file paths (not file contents)

• Project information

• Database metadata

Backup is stored in JSON format for portability.

# Example backup location
~/.scimax/backup-2026-01-22.json

✅ Restore

Command: Scimax: Restore Database (scimax.db.restore)

Restores database from a backup file:

• Imports project list

• Queues files for reindexing

• Preserves original creation timestamps

Note: Actual file content must still be reindexed after restore.

Options

Checks Performed

1. File existence - Verifies indexed files still exist on disk

2. Modification time - Detects files modified since indexing

3. Index integrity - Validates FTS5 and vector indexes

4. Project validity - Checks project directories exist

Result Format

Database Verification Results:
- Total files: 127
- Missing files: 2
- Stale files: 5
- Projects: 8 (7 valid, 1 missing)
- Status: NEEDS_REINDEX

Status Values

Benefits

• Projects persist across VS Code restarts

• Shared project list between Projectile and Database

• Fast project switching using indexed data

• Projects can be associated with indexed files

Project Commands

Projects are managed through Projectile commands (C-c p), but the database provides the persistence layer.

See Projectile for project management commands.

Problem

Semantic search returns no results, shows "unavailable", or displays an error.

First: Check Vector Search Support

Run Scimax: Show Database Stats (scimax.db.stats) to see the semantic search status:

Vector Search Unavailable

If you see "Semantic search: Unavailable", the libsql database doesn't support vector operations. This can happen if:

• The libsql version doesn't include vector support

• The vector index failed to create

Embedding Provider Issues

If vector search is supported but not working:

1. Check embedding provider is configured: scimax.db.embeddingProvider

2. Test connection: Run Scimax: Configure Embedding Service

3. Ensure files are reindexed after configuring embeddings

4. Check console for errors (Help: Toggle Developer Tools)

Local Provider Issues

• First use downloads model (~30MB), wait for completion

• Check extension cache directory has write permissions

• Try different model if one fails

Ollama Issues

• Ensure Ollama is running: ollama serve

• Pull model: ollama pull nomic-embed-text

• Check URL is correct in settings

• Test connection: curl http://localhost:11434/api/embeddings

Problem

Searches return empty results despite having files.

Solutions

1. Run Scimax: Show Database Stats to check file count

2. If files 0, run Scimax: Reindex Files

3. Check file extensions (.org, .md, .ipynb)

4. Verify files aren't in ignored directories

5. Check search scope is set to "All files"

Problem

Indexing takes very long or appears stuck.

Solutions

1. Check workspace size (number of files)

2. Add ignore patterns for large non-content directories

3. Disable embedding generation if not needed

4. Index directories incrementally

5. Check disk I/O and available memory

Problem

Errors mentioning "database is locked" or "disk I/O error".

Solutions

1. Close other VS Code windows accessing same workspace

2. Restart VS Code

3. Run Scimax: Clear Database and reindex

4. Check disk space is available

5. Verify database file permissions

Problem

VS Code uses excessive memory during indexing or searching.

Solutions

1. Disable auto-indexing

2. Reduce number of indexed directories

3. Use more aggressive ignore patterns

4. Clear and reindex database

5. Restart VS Code between large indexing operations

Files Table

Tracks indexed files with modification tracking:

• path, file_type, mtime, hash, size, indexed_at

Headings Table

Org/markdown headings with full metadata:

• level, title, todo_state, priority

• tags, inherited_tags, properties

• `scheduled', `deadline', `closed'

• line_number, begin_pos

Source Blocks Table

Code blocks with language and content:

• `language', `content', `headers'

• line_number, cell_index

Links Table

All link types (file, http, id, etc.):

• link_type, target, description

• line_number

Hashtags Table

Inline hashtags:

• tag, file_path

Chunks Table

Text chunks for semantic search:

• content, line_start, line_end

• embedding (F32_BLOB vector)

FTS Content (Virtual Table)

Full-text search index:

• file_path, title, content

• Porter stemming, Unicode normalization

• BM25 ranking support

Org Mode

UnifiedParserAdapter - Full AST parser compatible with org-element:

• Recursive heading parsing with inheritance

• Property drawer extraction

• Timestamp parsing (scheduled, deadline, closed)

• Source block with headers

• Link extraction

Markdown

Simplified parser:

• ATX heading syntax (`#')

• Fenced code blocks

• Inline links

Jupyter Notebooks

ipynbParser - Notebook-specific parser:

• Markdown cells → headings and links

• Code cells → source blocks

• Cell indices tracked for navigation

• Hashtags from markdown cells