Multiple choice

A multiple-choice item is a context, a list of choices, and the answer_index of the one correct choice. The same shape covers two different tasks, and the models read the context differently for each.

Shape

Completion stem
Question
Generic with style

The context is a sentence stem the correct choice continues, as in HellaSwag. The stem ends mid-sentence by design, so it is never judged for completeness.

from sdvm.types import MultipleChoiceCompletionSample

sample = MultipleChoiceCompletionSample(
    context="A woman applies mascara. she",
    choices=["applies it to her lashes.", "drives off.", "mixes a bowl.", "exits."],
    answer_index=0,
)

The context is a complete question the correct choice answers, as in MMLU. A truncated or malformed question is a defect, so completeness is judged.

from sdvm.types import MultipleChoiceQuestionAnswerSample

sample = MultipleChoiceQuestionAnswerSample(
    context="What is the capital of France?",
    choices=["Berlin", "Paris", "Madrid", "Rome"],
    answer_index=1,
)

MultipleChoiceSample takes a style of "continuation" (default) or "qa". Decoding a server response returns the specific class for the style, so a round-tripped sample carries the same guarantees as one you built directly.

from sdvm.types import MultipleChoiceSample

sample = MultipleChoiceSample(
    context="What is the capital of France?",
    choices=["Berlin", "Paris", "Madrid", "Rome"],
    answer_index=1,
    style="qa",
)

On the wire:

{
  "task_type": "multiple_choice",
  "style": "qa",
  "context": "What is the capital of France?",
  "choices": ["Berlin", "Paris", "Madrid", "Rome"],
  "answer_index": 1
}

The type refuses an empty context, fewer than two choices, an answer_index outside the choices, or an unknown style.

The distinct types exist so the guarantee is visible in the name. MultipleChoiceCompletionSample refuses style="qa" and vice versa, so you cannot accidentally get a completeness verdict on a stem.

Extra columns

Anything beyond the core fields goes in extra and comes back untouched:

MultipleChoiceQuestionAnswerSample(
    context="...", choices=[...], answer_index=2,
    extra={"subject": "astronomy", "source_id": "mmlu-1234"},
)

What the models return

sdvm-audit-1

What it returns for this sample.

sdvm-fix-1

What it changes, and what it never does.

Examples

hellaswag.py for stems and mmlu.py for questions, end to end.

Get started

Sample types

Models

Guides

SDK

HTTP API

Pricing and limits

Multiple choice

Shape

Extra columns

What the models return

sdvm-audit-1

sdvm-fix-1

Examples

​Shape

​Extra columns

​What the models return

sdvm-audit-1

sdvm-fix-1

Examples

Shape

Extra columns

What the models return