Lesson 14.414.4 Data models and schemas

In everyday life. A loading dock checks each delivery at the door: the labels, the paperwork, any damage. A broken pallet is refused there, not found three days later on a shelf. The dock does not open every box, so a box with the right label and the wrong goods inside still gets through.

What you will learn. How to write down what a valid record looks like, and what those rules let through.

Why it matters. Your tests cover the rows you can see. Written rules, checked on every record as it comes in, also cover next month's rows.

What is a schema?

A schema is a written description of what valid data looks like: which fields, which types, which values. It is the dock's checklist at the door. A schema validates data: it either returns the data, now trusted and typed, or returns a list of what is wrong. A data model is the type of a valid record, often produced from the schema.

In 14.2, isTerms checked one field by hand as the text came in. A schema checks every field of a record that way. In TypeScript, Brightline uses a library called Zod. In Python it uses Pydantic.

An excerpt from the schema file, then one bad invoice at the door. Each line of the schema is one rule from the table, and its comment names the rule. Predict which field the first problem names.

import * as z from "zod";

export const InvoiceSchema = z.object({
  // a pattern: INV-, then digits, and nothing else
  number: z.string().regex(/^INV-\d+$/, "must look like INV-8301"),
  // a real date
  invoiceDate: z.iso.date("must be a real date, YYYY-MM-DD"),
  // a real date, or empty
  dueDate: z.iso.date("must be a real date, YYYY-MM-DD").nullable(),
  // one of a fixed set
  terms: z.enum(TERMS, "must be one of the known terms"),
  // a whole number, zero or more
  amountCents: z.int("must be whole cents").min(0, "must be zero or more"),
  // ...vendorId and status
});

// at the door: a copy of INV-8304, with its amount changed to -23450
const result = InvoiceSchema.safeParse({ ...inv8304, amountCents: -23450 });
// did it pass, which field, and why. ?. reads the next part only if there is one
console.log(result.success, result.error?.issues[0].path, result.error?.issues[0].message);

false [ 'amountCents' ] must be zero or more

safeParse returns success, and either the typed data or the list of issues. model_validate raised one ValidationError holding both problems, and errors() lists them by field.

Why check at the edge?

Because every record passes there, on every run. An edge is wherever data comes in: a file, an API, a form or another program. Validate once, at the edges you marked in 14.1, and the code inside can rely on what it holds, as long as the schema is right and nothing later changes the data without checking it again.

At work. Ana decides the test cases first, from rows she can see, and the agent writes them. One good invoice, INV-8301, passes. Each kind of bad value is refused: a negative amount, a fraction of a cent, unknown terms and month 13. Then the agent adds Zod and Pydantic, and writes the schemas that pass. She compares every rule with the table above, and checks that the lock files changed in the same commit.

What can a schema not check?

Two kinds of rule. The first needs more than the one record being checked. The invoice schema does not have the vendor list, so it cannot know that vendor V099 does not exist. Rules that compare records, or compare a record with other data, are code you write after the schema. 14.5 shows where they go.

The second is a wrong value of the right shape. A schema checks shape, not meaning. If INV-8301's amount arrives as 48250 instead of 482500, it is whole cents, zero or more, and it passes. Your count and total from Chapter 12 catch it, and so does a test that names the right amount.

Does the library change values for me?

Pydantic does, by default: it turns the text "482500" into the number 482500. Zod does not: it refuses text where a number is expected.

The safe pattern in both languages is controlled conversion, then validation. The loader, src/load-invoices.ts, which reads the file and checks each row, turns text into a number only when the text is a plain whole number. Anything else it passes on for the schema to refuse.

Brightline's own invoiceFromRow does not do this. In JavaScript, Number("") and Number(" ") are both 0, and this function converts every amount with Number and checks nothing. due-by-friday, the list of Friday's payments, reads its invoices through it. If next week's export leaves INV-8301's amount blank, the list still shows 8 invoices, with INV-8301 at $0.00, a total of $14,404.53, and no warning. Read which conversions happen before you trust a schema.

How does a schema become JSON Schema?

Both libraries can write their schema out in a standard format called JSON Schema, a description of valid data that other programs read. Chapter 17 uses it to describe the inputs of a tool an agent can call.

Try it

The file in 14-4 checks two amounts with a one-field schema. Run it and investigate, as in 13.1.

Modify. Write in your notes whether each of these should be accepted: "482500" as text, the same text with spaces around it, 4825.5, and INV-8301's amount with one zero missing, 48250. Then send: "Please change the list of values in the file to these four. Don't run it yet." Say what the schema will do with each one. Then ask the agent to run it.

Make. Send: "Please make a new file in 14-4 called mine, with a schema for one field, the terms, which must be Net 15 or Net 30. Check three values of my choice. Don't run it yet." Read its rule line, say what it will print, and then ask the agent to run it.

What you should see. Zod refuses the two text values and the fraction. Pydantic turns the text into 482500, and refuses the fraction. Both accept 48250, because a schema checks shape, not meaning.

The operating rule: validate every record where it enters the program, with a schema whose rules you have read. Keep the rules that compare records in code right after it, and keep your own count and total for the wrong values a schema lets through.

Figure 14.5. Check at the edge. Inside the program, every record has already passed the schema. The amounts start as text, and the loader turns plain whole numbers into numbers before the schema checks them.

Next. A schema can refuse a record. 14.5 decides what happens next, and who is told.