Getting Started with MAR

MAR is a modern archive format and data archive toolset designed for modern computing. It is built to leverage advances in modern hardware—multi-core processors, NVMe storage, and fast compression algorithms—while supporting new interfaces to data archives such as semantic search, S3, MCP, and web-based browsing.

Think of MAR as "tar for the 21st century."

Note: The current release is v0.2.0. Indexing and search features are in active development; the CLI is relatively stable but may change.

Key Features

• Fast: Optimized for modern hardware with efficient I/O patterns on NVMe and SATA SSDs, hard drives, and multi-core architectures.
• Random Access: Retrieve individual files without decompressing the entire archive.
• Flexible Compression: Supports ZSTD (default), LZ4, Gzip, Bzip2, or no compression, with configurable compression levels.
• Integrity Verification: Built-in checksums using XXHash3 (default), XXHash32, CRC32C, BLAKE3, or none.
• POSIX Metadata: Optionally preserves permissions, ownership, and timestamps, just like tar.
• Flexible File/Block Layout: Files can be stored in their own block, shared with other files, or across multiple blocks.
• Parallel Ready: Multi-threaded creation, extraction, and validation using all available CPU cores.
• Content Deduplication: Optional BLAKE3-based deduplication of identical files within an archive.
• Sidecar Indices: .mai index files enable MinHash similarity, vector semantic search, BM25 keyword search, genomic, email, and time-series queries.
• Modern Interfaces: Designed for S3, MCP, and web-based archive browsing.

Installation

Pre-built Binaries (Recommended)

Download a static binary from the GitHub Releases page:

# Example: install on macOS (Apple Silicon)
curl -sL -o mar https://github.com/EarthFrame/mar/releases/download/v0.2.0/mar-macos-arm64
chmod +x mar
sudo mv mar /usr/local/bin/

# Verify
mar version

Homebrew

# From the EarthFrame tap (when available)
brew install earthframe/tap/mar

# Or build from the repository formula
git clone https://github.com/EarthFrame/mar
cd mar
brew install --build-from-source ./.homebrew/mar.rb

Build from Source

Dependencies (for all features):

• C++17 compatible compiler (GCC 8+, Clang 7+)
• zlib, libzstd, liblz4, libbz2, libdeflate
• libblake3 (vendored in deps/; no system install required)

macOS (Homebrew):

brew install gcc zstd lz4 bzip2 libdeflate

Debian / Ubuntu / Linux Mint:

sudo apt update
sudo apt install build-essential cmake libzstd-dev liblz4-dev libbz2-dev zlib1g-dev libdeflate-dev

Fedora / RHEL / CentOS:

sudo dnf install gcc-c++ cmake zstd-devel lz4-devel bzip2-devel zlib-devel libdeflate-devel

Arch Linux:

sudo pacman -S gcc cmake zstd lz4 bzip2 zlib libdeflate

Build:

git clone https://github.com/EarthFrame/mar
cd mar

make system-deps   # Install system dependencies (macOS/Linux)
make               # Build the tool
make test          # Run unit tests

Quick Start

The basic command form is:

mar <command> [options] <arguments>

Use mar -h for a list of commands, or mar <command> --help for command-specific options.

Create an Archive

# Create an archive from a directory
mar create archive.mar mydir/

# Use LZ4 compression (faster)
mar create -c lz4 archive.mar mydir/

# Use BLAKE3 checksums and enable deduplication
mar create --checksum blake3 --hashes --dedup archive.mar mydir/

# Deterministic output (sorted files, fixed timestamps)
mar create --deterministic archive.mar mydir/

# Read file list from stdin
find . -name "*.txt" | mar create -T - archive.mar

List Contents

# Simple list
mar list archive.mar

# Table with metadata (mode, owner, size, checksum)
mar list --table --header archive.mar

# JSON output
mar list --format json archive.mar

Extract an Archive

# Extract all files to the current directory
mar extract archive.mar

# Extract to a specific directory
mar extract -o ./output archive.mar

# Strip leading path components
mar extract --strip-components 1 archive.mar

Get Specific Files

# Extract a single file to stdout
mar get -c archive.mar path/to/file.txt

# Extract specific files to a directory
mar get -o /tmp archive.mar file1.txt file2.txt

Validate Integrity

mar validate archive.mar
mar validate -v archive.mar   # Verbose output

Indexing and Search

MAR supports sidecar index files (.mai) for advanced search without modifying the original archive. Available index types include MinHash (structural similarity), Vector (semantic search), BM25 (keyword search), Genomic, Email, and TimeSeries.

# Build a MinHash similarity index
mar index -i data.mar --type minhash --with bit_width=32 --with hashes=256

# Search for files similar to a query file
mar search -i data.mar --index data.mar.minhash.mai --with file=report.txt

See Indexing and Search for full documentation on vector search, hybrid BM25+vector queries, and other index types.

Format Specification

This implementation follows the MAR format specification v0.1.0. Key properties:

• 48-byte fixed header with magic number, version, and metadata offsets
• Section-based metadata container with NAME_TABLE, FILE_TABLE, FILE_SPANS, etc.
• 32-byte block headers with compression type and checksum
• 64-byte block alignment for efficient I/O

See the format specification in the repository for the complete spec.

Next Steps

• Explore the CLI Reference for all commands and options.
• Learn about Indexing and Search for semantic and similarity queries.
• Read the format specification on GitHub.