sdvm-audit-1

TextThinking

sdvm-audit-1 reads a sample and returns an audit. The sample is not changed. Each sample type is a schema the model knows: text, multiple choice, question and answer, conversation. The audit follows the schema: one field per property, true when healthy, false when not, null when the property does not apply.

import json
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.sdvm.ai/v1",
    api_key=os.environ["SDVM_API_KEY"],
)

sample = {
    "task_type": "multiple_choice",
    "style": "continuation",
    "context": "A woman sits at a bathroom mirror with a tube of mascara. "
               "She unscrews the wand and leans in close to the glass. she",
    "choices": [
        "drives off.",
        "exits.",
        "carefully applies the mascara to her upper and lower lashes, blinking between coats.",
        "mixes a bowl.",
    ],
    "answer_index": 2,
}

response = client.chat.completions.create(
    model="sdvm/audit-1",
    messages=[{"role": "user", "content": json.dumps({"data": [sample]})}],
)
audit = json.loads(response.choices[0].message.content)[0]["audit"]
print(audit["no_length_shortcut"])  # False

With the SDK, Auditor(api_key=...).run(samples) attaches the audit to each sample’s .audit.

A real request and response

{
  "data": [
    {
      "task_type": "multiple_choice",
      "style": "continuation",
      "context": "A woman sits at a bathroom mirror with a tube of mascara. She unscrews the wand and leans in close to the glass. she",
      "choices": [
        "drives off.",
        "exits.",
        "carefully applies the mascara to her upper and lower lashes, blinking between coats.",
        "mixes a bowl."
      ],
      "answer_index": 2
    }
  ]
}

{
  "audited_data": [
    {
      "task_type": "multiple_choice",
      "style": "continuation",
      "context": "A woman sits at a bathroom mirror with a tube of mascara. She unscrews the wand and leans in close to the glass. she",
      "choices": [
        "drives off.",
        "exits.",
        "carefully applies the mascara to her upper and lower lashes, blinking between coats.",
        "mixes a bowl."
      ],
      "answer_index": 2,
      "audit": {
        "num_choices": 4,
        "has_multiple_choices": true,
        "context_non_empty": true,
        "choices_non_empty": true,
        "valid_answer_index": true,
        "no_length_shortcut": false,
        "no_mojibake": true,
        "no_pii": true,
        "ctx_grammatical": false,
        "ctx_coherent": true,
        "ctx_fluent": true,
        "ctx_complete": null,
        "choice_grammatical": true,
        "choice_coherent": false,
        "choice_fluent": true,
        "choice_complete": true,
        "choice_defects": [0, 3],
        "label_correct": true,
        "single_valid_answer": true,
        "distractors_discriminating": true
      }
    }
  ],
  "original_count": 1,
  "audited_count": 1,
  "cost_usd": 0.01,
  "input_tokens": 1217,
  "output_tokens": 4621
}

ctx_grammatical: false: the stem ends in a lowercase she.
no_length_shortcut: false: the correct choice is far longer than the distractors, so the item is solvable by length alone. label_correct is true at the same time: the label is right, the item is still bad.
choice_defects: [0, 3]: the choices choice_coherent objected to, drives off. and mixes a bowl..
ctx_complete: null: a completion stem ends mid-sentence by design, so completeness does not apply.

Options

str | None

default:"None"

Dataset conventions the model must not read as defects, in free text: uniform lowercasing, markup tokens, a context truncated by design. See Dataset conventions.

int

default:"1"

Audit each sample this many times and keep the majority per field. Use an odd number. On AuditorConfig; on the HTTP API, config: {"votes": 3}. See Denoising with votes.

run takes a list of samples, up to 100 per request.

from sdvm import Auditor, AuditorConfig

auditor = Auditor(
    api_key=os.environ["SDVM_API_KEY"],
    config=AuditorConfig(votes=3),
)
audited = auditor.run(
    samples,
    conventions="The text is uniformly lowercased.",
)

Get started

Sample types

Models

Guides

SDK

HTTP API

Pricing and limits

sdvm-audit-1

A real request and response

Options

​A real request and response

​Options

A real request and response

Options