Refine

Refine runs sdvm-audit-1, sdvm-fix-1 and sdvm-audit-1 again in one call:

audit(votes) -> fix(max_attempts) -> reaudit(votes)

Each returned sample carries audit, the audit the fix was made from; fix, what the fix did; and reaudit, the audit of the fixed sample. Only samples the fix changed are re-audited; an unchanged sample has reaudit: null.

import os

from sdvm import AuditorConfig, FixerConfig, Refinery

refinery = Refinery(
    api_key=os.environ["SDVM_API_KEY"],
    audit=AuditorConfig(votes=3),
    fix=FixerConfig(max_attempts=2),
    reaudit=AuditorConfig(votes=1),
)

for sample in refinery.run(samples):
    print(sample.audit, sample.fix, sample.reaudit)

On the OpenAI-compatible endpoint the model id is sdvm/refine-1.

A real request and response

{
  "data": [
    {
      "task_type": "multiple_choice",
      "style": "continuation",
      "context": "A woman sits at a bathroom mirror with a tube of mascara. She unscrews the wand and leans in close to the glass. she",
      "choices": [
        "drives off.",
        "exits.",
        "carefully applies the mascara to her upper and lower lashes, blinking between coats.",
        "mixes a bowl."
      ],
      "answer_index": 2
    }
  ]
}

{
  "refined_data": [
    {
      "task_type": "multiple_choice",
      "style": "continuation",
      "context": "A woman sits at a bathroom mirror with a tube of mascara. She unscrews the wand and leans in close to the glass. She",
      "choices": [
        "turns away from the mirror and walks out of the bathroom immediately.",
        "steps back from the sink and leaves the room abruptly without applying anything.",
        "carefully applies the mascara to her upper and lower lashes, blinking between coats.",
        "grabs a mixing bowl from the cabinet and starts preparing ingredients for dinner."
      ],
      "answer_index": 2,
      "audit": {
        "num_choices": 4,
        "has_multiple_choices": true,
        "context_non_empty": true,
        "choices_non_empty": true,
        "valid_answer_index": true,
        "no_length_shortcut": false,
        "no_mojibake": true,
        "no_pii": true,
        "ctx_grammatical": false,
        "ctx_coherent": true,
        "ctx_fluent": true,
        "ctx_complete": null,
        "choice_grammatical": true,
        "choice_coherent": false,
        "choice_fluent": true,
        "choice_complete": true,
        "choice_defects": [0, 3],
        "label_correct": true,
        "single_valid_answer": true,
        "distractors_discriminating": true
      },
      "fix": {
        "changes": [
          "ctx: Capitalized final word 'she' to 'She'",
          "Rewrote wrong choices to be plausible sentence continuations that do not fit the context."
        ],
        "flagged": false,
        "attempts": 1
      },
      "reaudit": {
        "num_choices": 4,
        "has_multiple_choices": true,
        "context_non_empty": true,
        "choices_non_empty": true,
        "valid_answer_index": true,
        "no_length_shortcut": true,
        "no_mojibake": true,
        "no_pii": true,
        "ctx_grammatical": true,
        "ctx_coherent": true,
        "ctx_fluent": true,
        "ctx_complete": null,
        "choice_grammatical": true,
        "choice_coherent": true,
        "choice_fluent": true,
        "choice_complete": true,
        "choice_defects": [3],
        "label_correct": true,
        "single_valid_answer": true,
        "distractors_discriminating": true
      }
    }
  ],
  "original_count": 1,
  "refined_count": 1,
  "cost_usd": 0.01,
  "input_tokens": 3588,
  "output_tokens": 14956
}

audit found the lowercase she, the length shortcut and two incoherent distractors, and routed the fix.
reaudit: ctx_grammatical and no_length_shortcut are now true. choice_defects: [3] remains: the new fourth choice read as incoherent. The reaudit is how you know.
cost_usd covers all three stages.

Options

One config per stage. On the HTTP API the three nest under config: {"audit": {"votes": 3}, "fix": {"max_attempts": 2}, "reaudit": {"votes": 1}}.

AuditorConfig

default:"AuditorConfig(votes=1)"

The audit before the fix. votes as on sdvm-audit-1.

FixerConfig

default:"FixerConfig()"

The fix stage. max_attempts as on sdvm-fix-1.

AuditorConfig

default:"AuditorConfig(votes=1)"

The audit after the fix, on the samples the fix changed. votes as above.

str | None

default:"None"

Dataset conventions, forwarded to every stage. See Dataset conventions.

Configs set on the client are defaults; each can be overridden on a call.

result = refinery.run(samples, audit=AuditorConfig(votes=5), fix=FixerConfig(max_attempts=3))

Get started

Sample types

Models

Guides

SDK

HTTP API

Pricing and limits

A real request and response

Options

​A real request and response

​Options

A real request and response

Options