# Chunk Desk > Plan how a retrieval corpus should be chunked and embedded, then design its keyword, dense or hybrid search. The free check runs in the browser and uploads nothing: it splits a pasted or dropped corpus sample into documents and blocks (headings, paragraphs, lists, fenced code, tables) with exact character offsets, estimates tokens, chunks the sample with a fixed token window, recursive separators, heading sections or sentence packing at the chosen size and overlap, shows every chunk that ends mid-sentence, splits a code block or a table, cuts a heading off from its text or is a fragment, compares all four strategies at the same settings, finds duplicate chunks, boilerplate repeated across documents and mixed languages, and runs Okapi BM25 over the chunks for every pasted query - flagging queries that share no term with the sample and queries that carry exact identifiers. The paid plan lane picks the chunking, the cleaning and the embedding model and builds a golden set from the queries; the paid retrieval lane chooses the legs, the fusion, the candidate counts and reranking and routes every query to the leg it depends on. The page re-chunks the sample with the plan and checks every route against its own BM25. App: https://chunk-desk.skillsafe.ai/ API tutorial: https://chunk-desk.skillsafe.ai/api.html ## Input - A corpus sample: one or more documents, each introduced by a line `==> name <==` (the format `head a b` prints). Markdown, text, HTML (reduced to text), CSV and JSON files can be dropped; they are joined in that format. - Queries: one per line, what users really type (the first 20 are searched). - Chunking to test: strategy (`fixed`, `recursive`, `heading`, `sentence`), size and overlap in estimated tokens. - An optional note: a goal or a constraint ("self-hosted only", "Postgres with pgvector", "Spanish and English"). ## The free check (flags CK-01 to CK-15) - CK-01 chunks longer than common embedding inputs (512 tokens for BERT-family encoders such as bge-*-en-v1.5 and e5-*-v2, 256 for all-MiniLM-L6-v2, 8,191 for OpenAI's embedding models) - CK-02 chunks that end in the middle of a sentence - CK-03 fenced code blocks split across chunks with no chunk holding them whole - CK-04 Markdown tables split across chunks (the rows after the cut lose their header) - CK-05 headings cut off from their text, or chunks that hold only headings - CK-06 fragment chunks under a fifth of the target size - CK-07 overlap of half the chunk size or more (most text embedded twice), or no overlap at all - CK-08 duplicate chunks after whitespace normalisation - CK-09 queries with a vocabulary gap: no term shared with any chunk, or under half their terms in the sample - CK-10 queries carrying exact identifiers (error codes, part numbers, versions, CLI flags, standards such as DIN 933) that embeddings rank poorly - CK-11 more than one language in the sample, or a non-English sample - CK-12 chunks that sit under a heading they do not contain (a contextual prefix keeps the section) - CK-13 a sample too small to judge - CK-14 no queries pasted - CK-15 lines repeated verbatim across documents (navigation, footers, banners) Token counts are an estimate close to the cl100k tokenizer. BM25 is Okapi BM25 (k1 1.2, b 0.75) with light stemming and accent folding. ## Paid lanes (gpt-terra, routed on the `task` field) - `plan` (from embedding-strategies): verdict `ready` (keep the settings), `adjust` (same strategy, other size, overlap, prefix or cleaning) or `rework` (another strategy); a content profile; the chunking (strategy, size, overlap, contextual prefix, rules); preprocessing; one to three embedding models with the primary first (dimensions, input limit, multilingual, hosting, why, trade-off); every plan flag confirmed or dismissed; an evaluation with metrics and a golden set naming the chunk ids that answer each query. - `retrieval` (from hybrid-search-implementation): verdict `hybrid`, `dense` or `keyword` (the legs used); the keyword leg (engine, fields), the dense leg (model), the fusion (Reciprocal Rank Fusion with k, or normalised weights), candidates per leg, final k, reranking (model, top n) and filters; one route per query (keyword, dense or both, the answering chunks, whether keyword search finds them); an implementation with a snippet; every retrieval flag answered; an evaluation. - Handoff: a plan's settings can be applied to the free chunker, and the plan (chunking and primary model) can be carried into the retrieval lane. ## What the page checks in every reply - Plan: the verdict fits the change (ready changes nothing, adjust keeps the strategy, rework changes it); the primary model reads at least the planned chunk size and is multilingual when the sample is; every plan flag is answered once; every golden chunk id exists; the sample is re-chunked with the plan and the chunking flags are counted before and after. - Retrieval: the verdict matches the legs; RRF has a k and weights sum to 1; candidates per leg cover the final k and the reranker's top n; every query has one verbatim route; a query whose identifier occurs in the sample is never dense-only; a query with no keyword match is never keyword-only; every "keyword finds it" agrees with the page's BM25; the snippet carries no credential. ## Exports The check as Markdown, the flags as CSV, the chunks as JSONL (id, document, heading path, estimated tokens, offsets, text); a result as Markdown, JSON, and the golden set or the query routes as CSV. The golden set carries each answer's document and character offsets and the planned chunks that hold it, and a plan exports its planned chunks as JSONL with `embed_text` (the plan's contextual prefix applied, repeated boilerplate removed). A pinned baseline shows how chunk counts, breaks and flags change as the strategy, size or overlap changes. Partial exports are named `-partial`. ## Sources Derived from two agent skills by Seth Hobson, github.com/wshobson/agents (MIT): - embedding-strategies - https://skillsafe.ai/skill/@wshobson/embedding-strategies - hybrid-search-implementation - https://skillsafe.ai/skill/@wshobson/hybrid-search-implementation Background: Robertson and Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond" (2009); Cormack, Clarke and Buettcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods" (SIGIR 2009).