Forms are a different problem from letters or diaries. With a letter, you want the words. With a form, you want the data — name, date, amount, checkbox state — in a shape your software can read. Transcribing a form to a wall of prose and then re-parsing it by hand defeats the point. The better move is to have the AI return structured data directly, in a format you can drop straight into a database, spreadsheet, or pipeline.
This guide is about doing exactly that with prompts: reading completed handwritten forms and getting clean, validated JSON back. It’s a deep-dive companion to the complete guide to transcribing handwritten scans if you haven’t set up your scan quality and basic workflow yet, start there.
Why JSON, and Why Ask for It Directly
A multimodal model can read a form and describe it, but description isn’t structure. If you ask for prose, you’ll spend the time you saved on transcription writing a parser to pull the fields back out — and that parser breaks the moment the model phrases something differently.
JSON solves this. It’s a format the model knows intimately, it maps cleanly onto database rows and API payloads, and it forces the output into named fields you define in advance. You decide the schema; the model fills it. That single decision — schema in the prompt, JSON out — is what turns “transcription” into “data entry.”
The Core Prompt
Define the exact shape you want and insist the model return nothing but that shape. Everything in [brackets] is yours to replace.
You are a data-entry specialist transcribing a completed [patient intake form].
Task: Read the handwritten entries in the image and return them as JSON
matching the schema below.
Output format — return ONLY valid JSON, no other text, no Markdown fences:
{
"full_name": "",
"date_of_birth": "",
"address": "",
"phone": "",
"email": "",
"notes": ""
}
Rules:
- Transcribe only what is handwritten in the fields; ignore the
pre-printed field labels and any instructions on the form.
- For any field left blank on the form, use null.
- For any value you cannot read with confidence, use the string "[illegible]".
- For a value you can partly read, give your best guess and append " [?]".
- Do not normalize or reformat values — keep dates, phone numbers, and
spelling exactly as written.
- Do not invent values for fields that are not present on the form.
Three things make this reliable. The schema is explicit, so the model can’t drift into a structure your code doesn’t expect. Blank is distinct from illegible (null vs "[illegible]") — these mean very different things downstream, and collapsing them hides real problems. And “keep exactly as written” stops the model from silently “helpfully” reformatting a date or trimming a phone number, which is one of the sneakiest error sources in form processing.
Handling the Tricky Field Types
Plain text fields are the easy case. Forms are full of fields that aren’t.
Checkboxes and radio buttons. Tell the model what the options are and what counts as selected, so it returns a clean value rather than a description of a tick mark.
- "marital_status" is a checkbox group with options:
single, married, divorced, widowed.
Return the single option whose box is marked. If none is marked, use null.
If more than one is marked, return them as an array and add a
"marital_status_note": "multiple boxes marked".
Dates and numbers. Decide up front whether you want them verbatim or normalized. Verbatim is safer for an audit trail; normalized is handier for a database. If you normalize, make the rule explicit and keep the original too:
- For "date_of_birth", return the value exactly as written in
"date_of_birth_raw", AND a normalized ISO form in "date_of_birth"
(YYYY-MM-DD). If the date is ambiguous (e.g. unclear day vs. month),
leave "date_of_birth" as null and explain in "date_of_birth_note".
Signatures and stamps. Don’t ask the model to transcribe a signature as text — it’ll guess a name that may be wrong. Capture presence instead:
- "signature_present": true if a signature appears in the signature field,
otherwise false. Do not attempt to read the signed name.
Processing a Batch of Forms
One form at a time is the most accurate approach, but you rarely have just one. The pattern that scales: process each scan independently, append a stable identifier, and collect the results into one array on your side rather than asking the model to handle many pages at once.
Add an identifier so every record traces back to its source:
- Add a field "source_file" with this exact value: "[form_047.jpg]".
Then your code calls the prompt per image and accumulates the JSON objects. This keeps each call focused (better accuracy), makes failures isolatable to a single form, and gives you a clean list to import. Resist the urge to upload a stack and ask for “all of them” — accuracy drops and a single misread cascades through the whole response.
Validate in Code, Not on Faith
The JSON looks clean, but treat it as a draft until your code has checked it. A few lightweight validations catch most problems before they reach your database:
- Schema check. Confirm every expected key is present and no unexpected keys appeared. If the model returned malformed JSON, strip any stray Markdown fences and re-parse; if it still fails, re-run that single form.
- Flag the markers. Any field containing
"[illegible]"or ending in" [?]"goes into a review queue. These are the model telling you where to look — route them to a human instead of importing them blindly. - Sanity rules. Cheap, domain-specific checks go a long way: a date of birth in the future, a phone number with too few digits, an email without an
@. Don’t auto-correct — flag.
The goal isn’t to trust the model; it’s to make the model tell you, in a structured way, exactly where not to trust it. The null / [illegible] / [?] distinction is what makes that triage possible.
Where This Approach Struggles
Forms with dense, overlapping handwriting in tight fields are hard — people write outside the lines, and the model can misattribute a value to the wrong field. Multi-column or grid-style forms (think timesheets) are closer to a table problem; for those, a table-oriented prompt often does better than a flat schema. And as always, numbers and names carry the highest error risk precisely because the model can’t infer them from context, so those fields deserve the closest review.
None of this makes the approach unreliable — it makes the review step non-negotiable for anything that matters. For high-stakes data (medical, financial, legal), the JSON pipeline is a fast first pass that a human confirms, not a replacement for one.
A Note on Privacy
Forms are among the most sensitive documents you’ll handle — they exist to collect personal data. Before uploading completed forms to any AI service, confirm it’s permissible under your data-protection obligations and check how the provider treats the data. Where you can, redact identifiers you don’t actually need to extract, or use a configuration where uploads aren’t retained or used for training. With forms, this isn’t a footnote; it’s a prerequisite.
Wrap-Up
Getting structured data out of handwritten forms comes down to three moves: define the schema in the prompt, make the model distinguish blank from illegible from uncertain, and validate the JSON in code before you trust it. Do that, and form transcription stops being typing and becomes a pipeline.
For the broader workflow, scan-quality tips, and prompts for letters, ledgers, and historical scripts, see the complete guide to transcribing handwritten scans with AI.
Leave a Reply