← Chunk Desk / API
Tokens

Drive Chunk Desk from your own code

The page does its free work in the browser: it splits a corpus sample into documents and blocks, estimates tokens, chunks it with the strategy, size and overlap you choose (fixed, recursive, heading, sentence), measures what that breaks, compares all four strategies, runs BM25 over the chunks for every query and raises flags CK-01..CK-15. Its two paid lanes are available over HTTP. You send one JSON object with task set to "plan" or "retrieval", and the model replies with one JSON object. The easiest way to build the facts string is the page itself: Chunkit.buildInput(lane, Chunkit.analyze(sample, queries, settings), note) in chunkit.js, which also runs under Node.

Base URL: https://api.skillsafe.ai/v1/app-api. Every call sends Content-Type: application/json and, once you have a token, Authorization: Bearer <token>. Both lanes run on gpt-terra with a 10% publisher markup. A run needs a signed-in (user) token and is metered: the platform holds the estimate's hold_credits while the run is in flight and then charges only what it used (charged_credits). /estimate is free. A plain-text summary is at /llms.txt.

The response envelope

Every response is {"data": ..., "error": null} on success or {"data": null, "error": {"code", "message"}} on failure.

Error codes

HTTPcodemeaning
400VALIDATION_ERRORThe body is not a JSON object, or a field is the wrong type. Every field of the run input is a string.
401UNAUTHORIZEDNo token, or an expired one: get a new one (step 2).
402INSUFFICIENT_CREDITSThe balance is below the run's minimum. Top up at skillsafe.ai/account/billing/.
403FORBIDDENA guest token tried a metered run: sign in for a personal token.
404NOT_FOUNDUnknown job id.
429RATE_LIMITEDBack off and retry with the same Idempotency-Key.

The task field: two lanes

taskwhat the model does
"plan"Picks the chunking (strategy, size, overlap, contextual prefix, rules), the cleaning to do before embedding and the embedding model (primary first), answers every plan flag in lane_flags, and builds a golden set from your queries. Skill: embedding-strategies.
"retrieval"Chooses keyword, dense or hybrid, the fusion (RRF or weighted), candidates per leg, the final k and reranking, routes every query to the leg it depends on, and gives an implementation with a snippet. Skill: hybrid-search-implementation.

Input fields (all strings)

fieldwhencontent
taskrequiredplan or retrieval
samplerequiredThe corpus sample, each document introduced by a ==> name <== line. Past 30,000 characters the page cuts the middle on a paragraph boundary and leaves a [chunk-desk: ...] marker.
factsrequiredA JSON string: lane, summary (documents, tokens, structure, languages, boilerplate), settings (strategy, size, overlap), stats, compare (all four strategies), chunk_count, chunks (id, doc, tokens, heading_path, breaks, text preview), queries (n, query, identifiers, coverage, missing_terms, bm25_matched_chunks, bm25_top), flags (uid, id, sev, lanes, text, chunks, query), lane_flags, notes. It is Chunkit.factsOf(lane, A).
queriesoptionalThe queries, one per line (the page searches the first 20).
plan_handoffretrieval onlyA JSON string of a plan reply's verdict, chunking, primary model and preprocessing; send it with facts built at the plan's settings.
noteoptionalYour goal or constraint, up to 2,000 characters.
retry_noteoptionalOnly on a reformat retry: what was wrong with the previous reply.

The reply contracts

Both lanes share an envelope: task, verdict, headline, flag_answers[{flag, status: confirmed|dismissed, reason}], evaluation{metrics[], golden[{query, expected_chunks[], why}], steps[]}, risks[], questions[], summary.

task: "plan"

verdict (ready: keep the settings, adjust: same strategy with other numbers, prefix or cleaning, rework: another strategy), content_profile{kind, structure, languages[], notes}, chunking{strategy, size_tokens, overlap_tokens, contextual_prefix: none|heading_path|doc_title|summary, why, rules[]}, preprocessing[], models[{name, role: primary|alternative|budget, dimensions, max_input_tokens, multilingual, hosting: api|self-hosted, why, tradeoff}].

task: "retrieval"

verdict (hybrid, dense, keyword - the legs used), design{keyword_leg{use, engine, fields[], notes}, dense_leg{use, model, notes}, fusion{method: rrf|weighted|none, rrf_k, weights{dense, keyword}, why}, candidates_per_leg, final_k, rerank{use, model, top_n, why}, filters[]}, query_routes[{n, query, needs: keyword|dense|both, expected_chunks[], keyword_finds_it, why}], implementation{stack, steps[], snippet{language, code}}.

Worked examples

These are the page's own bundled examples (invented companies), with the input as app.js builds it and the saved model reply the page replays for free.

Plan: a NAS help center with an error-code table and CLI blocks

Input (sample and facts shortened here):

{
 "task": "plan",
 "sample": "==> errors.md <==\n# Bramblewyn NAS error and warning codes\n\nThis page lists the codes that BWOS shows on the front-panel display, in the web console under System > Event Log, and in the output of `bwctl`. Codes that begin with E are errors that stop the affected service until they are resolved. Codes that begin with W are warnings: the service keeps running, but the unit is telling you that something needs attention before it turns into an error. If a code is not listed here, write it down toget... (8767 characters in all)",
 "queries": "what does E2210 mean\nhow do I clear E1043 after replacing the disk\nbwctl backup status --verbose\nmy nightly copies quit working after I updated the software\nthe box is way slower than it should be when moving files around\nwhich code means the NAS is getting too hot",
 "facts": "(a JSON string; decoded: heading 160/0 over 23 chunks, 6 queries searched with BM25, 6 flags, lane_flags F1, F2, F6)",
 "note": "Self-hosted is fine; we already run Postgres 16 for the product."
}

Reply:

{
 "task": "plan",
 "verdict": "adjust",
 "headline": "Keep heading chunking, but raise size to 400 with 40 overlap, prefix the heading path, and split the code table by rows with its header repeated.",
 "content_profile": {
  "kind": "NAS product help center: error-code reference plus how-to and troubleshooting articles",
  "structure": "3 Markdown docs (~2,600 tokens) with 15 headings, 16 paragraphs, 3 fenced bwctl command blocks and one 10-row error-code table; most sections are 80-230 tokens",
  "languages": [
   "English"
  ],
  "notes": "The 6% 'other' share is code and identifiers (E2210, W0301, bwctl flags, BW-420), not a second language. Queries are split between exact codes/commands and loose paraphrases ('box', 'too hot', 'updated the software'), so identifiers must survive preprocessing verbatim."
 },
 "chunking": {
  "strategy": "heading",
  "size_tokens": 400,
  "overlap_tokens": 40,
  "contextual_prefix": "heading_path",
  "why": "Heading is already the best fit here: 0% mid-sentence, no split code, and only 2 orphan headings where recursive leaves 8 (fixed cuts 47% of chunks mid-sentence). The problem is the 160-token cap. It splits whole sections like 'Scheduled backup jobs fail following a firmware upgrade' (C15/C16), the bwctl status section (C13/C14) and Link speed (C21/C22) that fit easily in 400 tokens, and it cuts the code table into four pieces where only the first has the header. At 400 every section except the ~550-token table stays whole. A 40-token overlap (10%) only applies where prose has to split, and the heading-path prefix gives the headerle
... (shortened here; the page shows it whole)

Retrieval: the same help center, on the plan's chunking

Input (with the plan handed over):

{
 "task": "retrieval",
 "sample": "==> errors.md <==\n# Bramblewyn NAS error and warning codes\n\nThis page lists the codes that BWOS shows on the front-panel display, in the web console under System > Event Log, and in the output of `bwctl`. Codes that begin with E are errors that stop the affected service until they are resolved. Codes that begin with W are warnings: the service keeps running, but the unit is telling you that something needs attention before it turns into an error. If a code is not listed here, write it down toget... (8767 characters in all)",
 "queries": "what does E2210 mean\nhow do I clear E1043 after replacing the disk\nbwctl backup status --verbose\nmy nightly copies quit working after I updated the software\nthe box is way slower than it should be when moving files around\nwhich code means the NAS is getting too hot",
 "facts": "(a JSON string; decoded: heading 400/40 over 9 chunks, 6 queries searched with BM25, 4 flags, lane_flags F2, F3, F4)",
 "note": "Self-hosted is fine; we already run Postgres 16 for the product.",
 "plan_handoff": "{\"verdict\":\"adjust\",\"chunking\":{\"strategy\":\"heading\",\"size_tokens\":400,\"overlap_tokens\":40,\"contextual_prefix\":\"heading_path\"},\"primary_model\":{\"name\":\"BAAI/bge-m3\",\"dimensions\":1024,\"max_input_tokens\":8192,\"multilingual\":true},\"preprocessing\":[\"Preserve identifiers verbatim: do not lowercase-fold or strip punctuation ..."
}

Reply:

{
 "task": "retrieval",
 "verdict": "hybrid",
 "headline": "Hybrid in Postgres 16: tsvector keyword leg for codes and bwctl flags, pgvector bge-m3 dense leg for paraphrases, RRF k=60, bge-reranker-v2-m3 on the top 20.",
 "design": {
  "keyword_leg": {
   "use": true,
   "engine": "Postgres 16 full-text search (tsvector + GIN, ts_rank_cd)",
   "fields": [
    "text",
    "heading_path"
   ],
   "notes": "Queries 1-3 are carried by exact tokens (E2210, E1043, bwctl backup status --verbose) that the 'english' parser keeps as single lexemes (e2210, e1043, verbos). OR the query lexemes together instead of websearch_to_tsquery: a natural-language query ANDed would match nothing, and websearch treats a leading '-' in --verbose as negation. Weight heading_path as A so section titles like 'Clearing a code after you fix it' count. ts_rank_cd has no IDF, so treat it as a candidate generator and let fusion and the reranker order the results."
  },
  "dense_leg": {
   "use": true,
   "model": "BAAI/bge-m3 (1024 dims, dense output, the plan's primary model)",
   "notes": "Store the heading_path-prefixed chunk text the plan embedded; embed queries with the same model and no prefix. Queries 4-6 are paraphrases ('nightly copies quit working after I updated the software', 'getting too hot', 'the box is way slower') whose key words (firmware upgrade, temperature, link speed) are not in the query, so only the dense leg can reach C6 and C3 for them. Use cosine distance (vector_cosine_ops) with an HNSW index."
  },
  "fusion": {
   "method": "rrf",
   "rrf_k": 60,
   "weights": null,
   "why": "
... (shortened here; the page shows it whole)

Retrieval: a fastener catalogue searched by part number

Input:

{
 "task": "retrieval",
 "sample": "==> hex-bolts.md <==\n# Kestrel Fixings catalogue: hex bolts and set screws\n\nAll prices are per 100 pieces, excluding VAT, in EUR. Part numbers follow the pattern type-thread-length-material, so HX-M8-40-A2 is a hex head fastener, M8 thread, 40 mm long, in A2 stainless. Fully threaded set screws are supplied to DIN 933 / ISO 4017; partially threaded bolts to DIN 931 / ISO 4014.\n\n### HX-M6-20-A2\nHex set screw, A2 stainless (304), M6 x 20 mm, fully threaded.\nStandard: DIN 933 / ISO 4017. Pack of 20... (4695 characters in all)",
 "queries": "HX-M10-60-A4\nM8 x 40 A4 hex bolt DIN 933\nDIN 125 washer M10\nNT-M8-A2\ntightening torque for M12 8.8\nstainless nuts",
 "facts": "(a JSON string; decoded: heading 150/0 over 14 chunks, 6 queries searched with BM25, 5 flags, lane_flags F1, F2, F3, F4, F5)",
 "note": "Customers paste part numbers and specs from old invoices; exact matches matter more than nuance, and latency must stay low."
}

Reply:

{
 "task": "retrieval",
 "verdict": "keyword",
 "headline": "Keyword-only BM25 with an exact part-number field: every query is carried by codes or catalogue words that already occur in the sample.",
 "design": {
  "keyword_leg": {
   "use": true,
   "engine": "OpenSearch BM25 (or Elasticsearch) with a keyword sub-field for part numbers",
   "fields": [
    "text",
    "heading_path",
    "part_numbers"
   ],
   "notes": "Extract every ### heading (HX-..., NT-..., WS-...) into an exact-match part_numbers keyword field and boost it far above the text match so a pasted code ranks its own chunk first. Keep the standard tokenizer on text so HX-M10-60-A4 also splits into hx / m10 / 60 / a4 for partial pastes, and stem so 'nuts' matches 'nut'. Keep '8.8', 'DIN 933', 'M12' as tokens (the standard tokenizer does)."
  },
  "dense_leg": {
   "use": false,
   "model": null,
   "notes": "Not used: all six queries are part numbers, sizes, standards or catalogue words present in the sample, and a dense model ranks near-miss codes (HX-M10-60-A2 vs -A4, NT-M8-A2 vs NT-M8-A4) as equally close. Skipping it also keeps latency low, as the note asks. Revisit only if real logs show paraphrased queries whose words are not in the catalogue."
  },
  "fusion": {
   "method": "none",
   "rrf_k": null,
   "weights": null,
   "why": "Single keyword leg, nothing to fuse; the exact part_numbers match and the BM25 text match are combined inside one bool query."
  },
  "candidates_per_leg": 20,
  "final_k": 5,
  "rerank": {
   "use": false,
   "model": null,
   "top_n": null,
   "why": "BM25 already puts th
... (shortened here; the page shows it whole)

The page re-checks the reply; the API does not

The page holds every reply to account (recon.js): a plan's verdict against the settings it changes, the primary model's input limit against the chunk size and its languages against the sample, every flag answered once, every chunk id real, and the plan re-run on the sample; a retrieval design's verdict against its legs, fusion and candidate numbers, one verbatim route per query, and every keyword_finds_it against the page's BM25. Over the API you get the model's reply as it is; run the same checks yourself with Recon.reconcile in recon.js, or paste the same sample into the page.

Step 1 — a tiny client helper

One function that sends JSON and unwraps the envelope.

BASE="https://api.skillsafe.ai/v1/app-api"
TOKEN="paste-your-token-here"   # from https://chunk-desk.skillsafe.ai/tokens.html
api() { # api METHOD PATH [JSON-BODY]
  curl -s -X "$1" "$BASE$2" -H "Content-Type: application/json" \
    ${TOKEN:+-H "Authorization: Bearer $TOKEN"} ${3:+--data "$3"}
}

Step 2 — a token

A guest token is enough for /me and /estimate. A run is metered, so it needs a personal token: sign in on /tokens.html and copy it from there.

# a guest token (enough for /me and /estimate); a run needs your personal token from /tokens.html
TOKEN=$(curl -s -X POST "$BASE/guest" -H "Content-Type: application/json" -d '{"slug":"chunk-desk"}' | jq -r '.data.token')

Step 3 — GET /me

Who the token is (subject_type: user or guest) and the credit balance.

api GET /me | jq '.data | {subject_type, credits}'

Step 4 — POST /estimate

Free. The body is the run input itself (not wrapped in {"input": ...}). hold_credits is a reservation, not the price; min_credits is what the balance must cover; model reads gpt-5.6-terra and model_alias gpt-terra.

# input.json: the run input object - {"task": "plan", "sample": "...", "queries": "...", "facts": "..."}
api POST /estimate "$(cat input.json)" | jq '.data | {hold_credits, min_credits, model, model_alias, markup_bps, warnings}'

Step 5 — POST /run, then poll the job

Send an Idempotency-Key built from the lane and the input: a retry with the same key never bills twice. The page uses chunk-desk:<lane>:<inputHash>:a<attempt>. The job's output.output is the reply text; truncated: true means the balance cut it short.

KEY="chunk-desk:plan:$(shasum -a 256 input.json | cut -c1-16):a1"
JOB=$(curl -s -X POST "$BASE/run" -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" \
  -H "Idempotency-Key: $KEY" --data @input.json | jq -r '.data.job_id')
while :; do
  J=$(api GET "/jobs/$JOB"); S=$(echo "$J" | jq -r '.data.status')
  case "$S" in succeeded|failed|cancelled) break;; esac; sleep 2
done
echo "$J" | jq -r '.data.output.output' > reply.json   # one JSON object, the lane's contract
echo "$J" | jq '.data | {charged_credits, truncated}'

Step 6 — POST /run-stream (retrieval)

The same run as Server-Sent Events. retrieval.json is a retrieval input, with plan_handoff when it follows a plan.

# Server-Sent Events: tick/delta events while the job runs, then a final event with the job
curl -N -s -X POST "$BASE/run-stream" -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" \
  -H "Idempotency-Key: chunk-desk:retrieval:$(shasum -a 256 retrieval.json | cut -c1-16):a1" --data @retrieval.json