Data Go Open Source

mksqlite — Convert Anything to SQLite

mksqlite is a zero-config command-line tool written in Go that converts virtually any tabular data format — CSV, XLSX, Parquet, JSON, HTML tables — into a portable, query-ready SQLite database file. No schemas. No boilerplate.

View on GitHub ← All Projects

go install github.com/darianmavgo/mksqlite@latest
mksqlite data.csv output.db
mksqlite spreadsheet.xlsx finance.db
sqlite3 output.db 'SELECT * FROM data LIMIT 10'

Features

Ingest CSV, Excel, Parquet, JSON, HTML, and more into a query-ready SQLite database with a single command.

⚡

Zero Config

Point at any file. mksqlite detects format, schema, and column types automatically.

📦

Universal Formats

Supports CSV, TSV, XLSX, Parquet, JSON arrays, HTML tables, and more out of the box.

🔄

Streaming Ingestion

Streams arbitrarily large files into SQLite without loading everything into memory first.

🔍

Schema Inference

Automatically infers and casts column types: integers, floats, dates, booleans, and strings.

🚀

Go Concurrency

Leverages Go's goroutines to parallelize ingestion across multiple tables and sheets.

🔗

Ecosystem Ready

Works seamlessly alongside banquet, wasmcanvas, aggrid, and the rest of the Mavgo SQLite suite.

Documentation & Architecture

A robust library and command-line tool designed to convert various file formats and data streams into SQLite databases or SQL statements.

Features Supported

Multi-Format Conversion:
- PDF (.pdf): Automatically extracts tables with spatial layout analysis and multi-line row aggregation.
- CSV: Converts delimiters, handles headers, sanitizes column names.
- Excel (.xlsx, .xls): Converts each sheet into a separate table.
- HTML: Extracts data from standard HTML <table> elements.
- JSON: Converts JSON data into structured tables.
- Markdown (.md): Extracts data from Markdown tables.
- Text (.txt): Parses text files into tables.
- ZIP (.zip): Processes formats contained within ZIP archives.
- Filesystem: Recursively crawls directories to create a metadata index (path, size, etc.) in SQLite.
Dual Output Modes:
- SQLite Database: Direct binary creation of .db files.
- SQL Dump: Generates CREATE TABLE and INSERT statements to stdout (great for piping).
Flexible Usage: Available as both a standalone CLI tool and a Go library (package converters).
Stream Processing: capable of processing data streams without loading entire files into memory.

Installation

Install mksqlite as a system-wide command:

./install.sh

Or install to a custom directory:

./install.sh --prefix /usr/local/bin

Area of Responsibility

mksqlite is the Ingestion Engine. Its job is to take unstructured or semi-structured data from the “wild” (files, scrapes, spreadsheets) and normalize it into the universal structured format: SQLite. It bridges the gap between raw data files and SQL-capable tools.

Scope (What it explicitly doesn’t do)

No Long-Running Service: mksqlite is a task-based tool. It runs, converts, and exits. It is not an HTTP server or a daemon.
No Query Execution: It does not run user queries (SELECT, etc.). It only performs CREATE and INSERT operations necessary for conversion.
No Visualization: It does not provide a UI to view the data; it only prepares the data for other tools (like sqliter) to view.

Quick Usage

Create a Database

# Convert a CSV to a SQLite DB (output DB is optional, defaults to data.csv.db)
mksqlite data.csv [data.db]

# Index a directory
mksqlite ./documents/ index.db

# Convert with error logging
mksqlite --log data.csv

# Resume an interrupted directory conversion from a specific path
mksqlite --resume-path ./documents/some/file.txt ./documents/ index.db

Generate SQL

# Pipe SQL output to stdout
mksqlite --sql data.csv > dump.sql

# Export directly to an output file
mksqlite --sql data.csv dump.sql