Indexing and Search
MAR supports sidecar index files (.mai) that enable advanced search capabilities without modifying the original .mar archive. Indices are built with mar index and queried with mar search.
Note: Indexing and search are under active development. The CLI is relatively stable but may change. Consider these features in beta.
Overview
| Index Type | Use Case |
|---|---|
minhash |
Structural similarity — find files with similar content |
vector |
Semantic search — natural language queries via embeddings |
bm25 |
Keyword search — probabilistic lexical retrieval |
genomic |
Genomic sequence indexing and region queries |
email |
Email archive search by headers, dates, and keywords |
timeseries |
Time-series CSV/TSV data filtering and anomaly detection |
Index files are named <archive>.<type>.mai by default (e.g., data.mar.minhash.mai).
Building Indices
mar index -i <archive> --type <type> [--with key=value ...]
Common options:
| Option | Description |
|---|---|
-i, --input <archive> |
Path to the .mar archive |
--type <type> |
Index type (see table above) |
--with <key=value> |
Type-specific parameter (repeatable) |
-o, --output <file> |
Custom output path |
--align <log2> |
Section alignment as 2^n bytes |
Use mar index --type <type> --help for type-specific build and search parameters.
MinHash (Similarity)
MinHash indices find files with structurally similar content using locality-sensitive hashing.
Build Options
| Parameter | Description |
|---|---|
hashes=N |
Number of hash functions (default: 128) |
bit_width=W |
Hash bit width: 8, 16, 32, or 64 (default: 64) |
seed=S |
Base seed (default: 42) |
threads=N |
Build threads (default: CPU cores) |
Search Options
| Parameter | Description |
|---|---|
file=NAME |
Find files similar to archive file NAME |
topk=N |
Maximum results (default: 10) |
format=X |
Output format: text, json, filenames (default: text) |
Examples
# Build index
mar index -i data.mar --type minhash --with bit_width=32 --with hashes=256
# Search by query file
mar search -i data.mar --index data.mar.minhash.mai --with file=report.txt
# Search with positional query
mar search -i data.mar --index data.mar.minhash.mai query.txt --with topk=5
Vector (Semantic Search)
Vector indices use embeddings and an HNSW graph for approximate nearest-neighbor semantic search. Building a vector index requires a running mar-embed-server (or compatible embedding endpoint).
Build Options
| Parameter | Description |
|---|---|
url=URL |
mar-embed-server URL (required) |
model=MODEL |
Embedding model name (optional; server default used otherwise) |
chunk_size=N |
Characters per chunk (default: 1024) |
chunk_overlap=N |
Overlap between adjacent chunks (default: 128) |
chunk_mode=fixed|semantic |
Chunking strategy (default: fixed) |
semantic_threshold=F |
Cosine similarity threshold for semantic boundaries (default: 0.75) |
dtype=float32|int8 |
Vector dtype (default: float32; int8 recommended for 4× smaller index) |
hnsw_M=N |
HNSW M parameter (default: 16) |
hnsw_ef_construction=N |
HNSW ef_construction (default: 200) |
batch_size=N |
Embedding batch size (default: 32) |
parallel_embedders=N |
Parallel embedding workers (default: 4) |
Search Options
| Parameter | Description |
|---|---|
url=URL |
mar-embed-server URL (required for natural language queries) |
topk=N |
Maximum results (default: 10) |
mode=files|chunks |
File-level (default) or chunk-level results |
format=text|json|filenames |
Output format (default: text) |
file=NAME |
Nearest-neighbor from in-archive file (no server needed) |
rerank=true |
Enable cross-encoder reranking |
rerank_candidates=N |
Candidates for reranking (default: 100) |
rerank_top_n=N |
Results after reranking (default: topk) |
hybrid=true |
Enable hybrid search with BM25 |
bm25_index=PATH |
Path to BM25 index for hybrid search |
Examples
# Build a vector index with int8 quantization
mar index -i docs.mar --type vector \
--with url=http://localhost:7998 \
--with model=voyageai/voyage-4-nano \
--with dtype=int8
# Semantic search with natural language
mar search -i docs.mar --index docs.vector.mai \
"How is Singular Value Decomposition implemented?" \
--with url=http://localhost:7998 --with topk=5
# Return matching chunks (for RAG)
mar search -i docs.mar --index docs.vector.mai \
"matrix inverse implementation" \
--with url=http://localhost:7998 --with mode=chunks --with topk=2
# Find files similar to an existing archive file (no server needed)
mar search -i docs.mar --index docs.vector.mai \
--with file=reference.py --with topk=5
Semantic Chunking
For better retrieval quality with long documents, use semantic chunking to detect topic boundaries:
mar index -i docs.mar --type vector \
--with url=http://localhost:7998 \
--with chunk_mode=semantic \
--with semantic_threshold=0.75 \
--with dtype=int8
Semantic chunking embeds sliding windows to find where topics shift, creating chunks aligned with natural boundaries rather than fixed sizes. This typically improves answer relevance for RAG applications by 15–25%, at the cost of slower indexing.
BM25 (Keyword Search)
BM25 is a probabilistic retrieval model that scores documents based on term frequency and document length normalization. It requires no external server.
Build Options
| Parameter | Description |
|---|---|
bm25_k1=F |
Term frequency saturation parameter (default: 1.2) |
bm25_b=F |
Length normalization parameter (default: 0.75) |
Examples
mar index -i docs.mar --type bm25
mar index -i docs.mar --type bm25 --with bm25_k1=1.5 --with bm25_b=0.5
Hybrid Search (Vector + BM25)
Vector search excels at semantic similarity but can miss exact keyword matches. Hybrid search combines both using Reciprocal Rank Fusion:
# Build both indexes
mar index -i docs.mar --type bm25
mar index -i docs.mar --type vector --with url=http://localhost:7998 --with dtype=int8
# Hybrid search
mar search -i docs.mar --index docs.vector.mai \
"eigenvalue decomposition algorithm" \
--with url=http://localhost:7998 \
--with hybrid=true \
--with bm25_index=docs.mar.bm25.mai \
--with topk=5
Documents appearing in both result lists receive higher combined scores. Hybrid search is especially useful for technical documentation where exact function names and APIs matter alongside semantic relevance.
Genomic
Genomic indices support sequence similarity search and region extraction from FASTA/VCF archives.
Build Options
| Parameter | Description |
|---|---|
k=N |
K-mer size for similarity sketching (default: 21) |
num_hashes=N |
Sketch size (default: 256) |
seed=S |
Hash seed (default: 42) |
stranded=0|1 |
0 = canonical k-mers (default), 1 = forward only |
vcf_bin_size=N |
VCF region bin size in bp (default: 65536) |
Query Formats
| Format | Description |
|---|---|
FILE.fa |
Similarity search (external reference) |
chr1:1000-2000 |
Region query (1-based inclusive; FASTA / VCF) |
chrM |
Whole contig |
--with file=X |
Similarity search (in-archive file) |
Examples
mar index -i ref.mar --type genomic
# Region extraction
mar search -i ref.mar --index ref.genomic.mai chr1:1000000-2000000 --extract
# Similarity search
mar search -i ref.mar --index ref.genomic.mai query.fa --with topk=5
Email indices enable search over archived email messages by headers, dates, threads, and keywords.
Search Options
| Parameter | Description |
|---|---|
from=ADDR |
Filter by From: address (substring match) |
to=ADDR |
Filter by To: address (substring match) |
subject=TEXT |
Filter by Subject: (substring match) |
since=DATE |
Earliest date (ISO 8601 or epoch seconds) |
until=DATE |
Latest date (ISO 8601 or epoch seconds) |
thread=MSGID |
Return all messages in the same thread |
topk=N |
Maximum results (default: 10) |
format=X |
Output format: text, json, filenames |
Positional query arguments perform keyword full-text search (space-separated tokens are ORed).
Examples
mar index -i mail.mar --type email
mar search -i mail.mar --index mail.email.mai "project deadline" \
--with since=2024-01-01 --with format=json
mar search -i mail.mar --index mail.email.mai \
--with thread="<abc@mail.example.com>"
TimeSeries
TimeSeries indices support filtering and anomaly detection over CSV/TSV data archived in MAR.
Build Options
| Parameter | Description |
|---|---|
ts_col=NAME|INDEX |
Timestamp column name or 0-based index (required) |
ts_format=FORMAT |
Timestamp format (required): iso8601, epoch_s, epoch_ms, epoch_us, auto, or strptime string |
delim=CHAR |
Delimiter character (default: ,; use \t for TSV) |
has_header=true|false |
Whether the first row is a header (default: true) |
value_cols=A,B,C |
Columns to compute stats for (default: all) |
skip_rows=N |
Rows to skip before header/data (default: 0) |
Search Options
| Parameter | Description |
|---|---|
since=TS |
Earliest timestamp (epoch ms, epoch s, or ISO 8601) |
until=TS |
Latest timestamp |
col=NAME |
Filter to files with a matching column (partial) |
zscore=N |
Files where any column's max deviates > N sigma from its mean |
min=V, max=V |
Value range filter applied to all numeric columns |
topk=N |
Maximum results (default: 10) |
format=X |
Output format: text, json, filenames |
Examples
mar index -i sensors.mar --type timeseries \
--with ts_col=timestamp \
--with ts_format=iso8601 \
--with value_cols=temperature,humidity
mar search -i sensors.mar --index sensors.timeseries.mai \
--with since=2024-01-01 --with until=2024-01-31 --with format=json
mar search -i sensors.mar --index sensors.timeseries.mai --with zscore=3.0
Embedding Server
Vector indexing requires an embedding server. The recommended approach is to run mar-embed-server locally or point to a remote instance:
# Start the embedding server
mar-embed-server --port 7998
# Index with the server
mar index -i docs.mar --type vector --with url=http://localhost:7998
For production deployments, configure provider credentials (VoyageAI, OpenAI, HuggingFace) in the embedding server rather than passing API keys to mar directly. See the embedding providers documentation in the repository for details.
Index File Format
Sidecar .mai files share a common 64-byte header followed by type-specific sections. See the index format specification in the repository for the complete on-disk format.