Core concepts

Samples

Everything you send to SDVM is a list of samples. Each sample declares a task_type so the service knows how to audit and fix it.

Sample	`task_type`	What it holds
`TextSample`	`text`	A single piece of free text, for example a pre-training document or a chat utterance.
`MultipleChoiceCompletionSample`	`multiple_choice`	A sentence stem the correct choice continues, as in HellaSwag.
`MultipleChoiceQuestionAnswerSample`	`multiple_choice`	A complete question the correct choice answers, as in MMLU.
`MultipleChoiceSample`	`multiple_choice`	The generic form with a `style` flag; the two classes above are its specialisations.
`QuestionAnswerSample`	`question_answer`	A question and the answer the dataset publishes, with no choices to pick from.
`ConversationSample`	`conversation`	A chat transcript: a list of `Message(role, content)` turns in the OpenAI shape, as in an SFT corpus.

Multiple-choice samples carry a context, a list of choices, the answer_index of the one correct choice, and an extra dict of passthrough metadata that comes back untouched. A question-answer sample is just the two fields, with the same extra passthrough. A conversation is its messages list, with the same extra. Each type has its own page under Sample types with the fields it holds and its shape on the wire.

The models

Model	Client	What it does
`sdvm-audit-1`	`Auditor`	Rates a sample and returns one verdict per check on `.audit`. `AuditorConfig(votes=N)` majority-votes the verdicts. Changes nothing.
`sdvm-fix-1`	`Fixer`	Fixes a sample and records what changed on `.fix`. `FixerConfig(max_attempts=N)` gives it up to N tries at an accepted fix. Never leaves it worse. Does not reaudit.
both, chained	`Refinery`	`sdvm-audit-1 -> sdvm-fix-1 -> sdvm-audit-1` in one call, one config per stage, so `.reaudit` says whether the fix helped.

Each has an async twin (AsyncAuditor, AsyncFixer, AsyncRefinery) with the same run method.

The audit block

After an audit, every sample carries .audit: a dict of dimension name to verdict (true = healthy; false = defect; null = the check does not apply). Alongside the shape and content checks each type lists on its own page, sdvm-audit-1 returns these dimensions, per sample type:

Dimension	Question it answers	Sample types
`ctx_grammatical`	Is the context (the question, or the text) grammatical?	text, multiple-choice, question-answer
`ctx_coherent`	Does it make sense on its own? On a conversation, this is the final user turn, and the only verdict it gets.	all
`ctx_fluent`	Does it read naturally?	text, multiple-choice, question-answer
`ctx_complete`	Is it complete, not cut off? `null` on a completion stem, which is truncated by design.	text, multiple-choice, question-answer
`choice_grammatical`	Is every choice grammatical? `true` only if all choices pass.	multiple-choice
`choice_coherent`	Does every choice make sense?	multiple-choice
`choice_fluent`	Does every choice read naturally?	multiple-choice
`choice_complete`	Is every choice complete, not cut off?	multiple-choice
`choice_defects`	The indices of the choices that failed any of the four above. A list, not a verdict.	multiple-choice
`answer_grammatical`	Is the answer’s own text grammatical? A bare value like `72` passes; on a conversation, the final reply’s markdown and code are formatting.	question-answer, conversation
`answer_coherent`	Does the answer’s text make sense? Rated on the writing, never on correctness.	question-answer, conversation
`answer_fluent`	Does the answer read naturally?	question-answer, conversation
`answer_complete`	Is the answer complete, not cut off? A bare value passes; a reply cut off mid-list or mid-code-block fails.	question-answer, conversation
`label_correct`	Is the marked (or published) answer actually correct?	multiple-choice, question-answer
`response_grounded`	Does the final reply answer the last user turn, stay consistent with the transcript and follow the system message? Not a truth judgement.	conversation
`single_valid_answer`	Is exactly one choice valid?	multiple-choice
`distractors_discriminating`	Do the wrong choices actually discriminate, or are they trivially wrong?	multiple-choice

A verdict is a reading of the sample, not a lookup: on a borderline item it can differ between requests. Where it has to be stable, votes audits the sample several times and keeps the majority.

Auditor.aggregate(samples) adds a local, dataset-level view: per-field distributions, a per-dimension pass/fail rollup, and an answer-position bias check that tells you whether the correct answer sits in one slot too often.

The fix block

After a fix, every sample carries .fix:

{"changes": [...], "flagged": bool, "attempts": int}

changes lists what the fix did. flagged marks a sample the fix could not fully resolve, surfaced for review rather than shipped as a silent partial edit. attempts is how many tries the fixer got at it (see max_attempts). Filtering on flagged gives you a review queue:

flagged = [s for s in fixed if s.fix and s.fix["flagged"]]

The never-worse contract

sdvm-fix-1 fixes what the audit verdicts say is wrong. If a fix would introduce a new defect — a structural one, an empty field, mojibake or PII — it is withheld and the sample is flagged. A fix never leaves a sample worse than its input. Fixer does not judge whether a fix helped; that takes a reaudit, which is Refinery’s job.

The refine pipeline

Refinery runs, server-side, one pass of three stages:

audit(votes) -> fix(max_attempts) -> reaudit(votes)

Each stage has its own config — AuditorConfig(votes) for the two audits, FixerConfig(max_attempts) for the fix — and each returned sample carries one block per stage: .audit is the verdict before the fix (the one the fix was routed from, and the same block Auditor returns), .fix records what the fix changed, and .reaudit is the verdict after it, which is how you tell whether the fix helped. A sample the fix left alone is not re-audited; its .reaudit is None and its .audit still holds.

Limits

Up to 100 samples per request.
100 requests per minute per key.
Token-based pricing with a minimum of $0.01 per request; see Pricing.

Get started

Sample types

Models

Guides

SDK

HTTP API

Pricing and limits

Core concepts

Samples

The models

The audit block

The fix block

The never-worse contract

The refine pipeline

Limits

​Samples

​The models

​The audit block

​The fix block

​The never-worse contract

​The refine pipeline

​Limits

Samples

The models

The audit block

The fix block

The never-worse contract

The refine pipeline

Limits