Split Markdown into reproducible chunks

Split Markdown with an exact size model

“Characters” and “tokens” are not interchangeable. A JavaScript string counts UTF-16 code units, UTF-8 stores one to four bytes per Unicode scalar, and every model tokenizer has its own vocabulary. This tool therefore offers two reproducible measurements only: exact UTF-8 bytes and Unicode code points. The manifest names the selected model and records exact byte ranges beside it. It never labels either number as model tokens.

The target applies to source owned by a chunk. Prefix overlap is separate and contains only complete prior source atoms. That makes both numbers auditable: a chunk can own 4,000 bytes and export more because it carries explicit prior context. The actual overlap can be less than requested when the previous atom would not fit whole.

Recognized source structures remain atomic

The scanner uses physical source offsets rather than a rendered Markdown tree. ATX and Setext headings are recognized conservatively, while heading-looking lines in front matter, fenced or indented code, list containers, block quotes and raw HTML stay body source. A heading remains with its first following body atom so a split does not leave an orphan heading at the end of one file.

Backtick and tilde fences use their actual marker and run length. A three-backtick example inside a four-backtick block is not a closer. Strict GFM pipe tables treat escaped pipes and pipes inside matched code spans as cell content. An ambiguous or ragged candidate remains ordinary opaque source instead of being forced into a table.

A single recognized atom can be larger than the target. Keep whole, the default, places it alone and records the oversize measurement. Strict target stops export and identifies the exact atom. Neither mode silently cuts a fenced block or table, and neither rejects the input merely because the complete file is large.

The manifest distinguishes overlap from ownership

Every chunk carries a content byte range and an owned byte range. Reassembly checks the chunk SHA-256, removes only the declared prefix overlap, concatenates owned bytes in source order and compares the result with the original source hash and byte length. BOM, CRLF, LF, lone CR, mixed endings and final-newline state remain observable.

Stable chunk IDs derive from the schema, source identity, settings and exact byte spans. They are stable for the same inputs, paths, order and options; editing or renaming a source creates a new identity instead of pretending it is the same artifact. The JSON has no timestamp, random value or locale-dependent sorting.

Files and folders stay separate

Each selected file is chunked independently. This route never joins the end of one document to the start of another and overlap never crosses a file boundary. If you need one combined Markdown document first, use Merge Markdown files. For reorganizing headings within one authored document, use the Markdown editor.

A folder input supplies browser-reported relative paths. Those paths retain their provenance in the receipt, while unsafe, reserved or portable case/Unicode collisions receive deterministic export aliases. A web page cannot prove an absolute filesystem path, inode or whether an operating system followed a symlink, so the product does not claim that evidence.

Verified local export, not AI advice

Generate chunks creates one lazy local worker for every input size. It decodes files with fatal UTF-8 validation, scans exact source spans, hashes chunks, writes a deterministic STORE ZIP and reopens every entry to verify CRC, size and SHA-256. Preview is escaped raw text and can be clipped; copy, individual download, manifest and ZIP use complete content.

No embeddings, model calls or network fetches occur. The result is structure-aware only at the documented recognized boundaries. It is not semantic chunking, an “AI-readiness” score, an optimal RAG strategy or evidence that retrieval quality will improve. Those outcomes depend on a model, corpus and evaluation outside this deterministic source operation.