Create a score
Records one evaluator result on a trace or a thread of the caller’s workspace. The value is checked against the pinned evaluator version’s output type and passed is derived from its pass rule. Scores are immutable: a second score for the same evaluator version, target and revision (and author, for human scores) is a 409 score_exists. A score from a local code evaluator must carry source_sha256, the hash of the pinned version’s source (409 source_mismatch otherwise). An experiment_item target takes a local code evaluator the experiment pinned, on an item that is done (the experiment completes when its last owed local score arrives), or a human label (source: human) from any evaluator on a done item; human labels feed the judge-vs-human agreement check and never count toward the run.
curl --request POST \
--url https://api.veri.studio/v1/scores \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '
{
"evaluator": "<string>",
"target_type": "<string>",
"evaluator_version": 123,
"request_id": "<string>",
"deployment_id": "<string>",
"thread_id": "<string>",
"target_revision": "<string>",
"value": "<unknown>",
"reasoning": "<string>",
"status": "<string>",
"error": "<string>",
"source": "<string>",
"source_sha256": "<string>",
"experiment_id": "<string>",
"item_id": "<string>"
}
'import requests
url = "https://api.veri.studio/v1/scores"
payload = {
"evaluator": "<string>",
"target_type": "<string>",
"evaluator_version": 123,
"request_id": "<string>",
"deployment_id": "<string>",
"thread_id": "<string>",
"target_revision": "<string>",
"value": "<unknown>",
"reasoning": "<string>",
"status": "<string>",
"error": "<string>",
"source": "<string>",
"source_sha256": "<string>",
"experiment_id": "<string>",
"item_id": "<string>"
}
headers = {
"Authorization": "Bearer <token>",
"Content-Type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text)const options = {
method: 'POST',
headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
body: JSON.stringify({
evaluator: '<string>',
target_type: '<string>',
evaluator_version: 123,
request_id: '<string>',
deployment_id: '<string>',
thread_id: '<string>',
target_revision: '<string>',
value: '<unknown>',
reasoning: '<string>',
status: '<string>',
error: '<string>',
source: '<string>',
source_sha256: '<string>',
experiment_id: '<string>',
item_id: '<string>'
})
};
fetch('https://api.veri.studio/v1/scores', options)
.then(res => res.json())
.then(res => console.log(res))
.catch(err => console.error(err));{
"object": "<string>",
"id": "<string>",
"evaluator_id": "<string>",
"evaluator_version": 123,
"name": "<string>",
"target_type": "<string>",
"value": "<unknown>",
"source": "<string>",
"status": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"request_id": "<string>",
"deployment_id": "<string>",
"thread_id": "<string>",
"target_revision": "<string>",
"experiment_id": "<string>",
"item_id": "<string>",
"passed": true,
"reasoning": "<string>",
"error": "<string>",
"author": "<string>",
"model_version_id": "<string>",
"monitor_id": "<string>",
"confidence": 123,
"probabilities": {}
}Authorizations
API key with the vk_ prefix. Create one from the dashboard.
Body
Evaluator id (evl_...) or name.
trace | thread | experiment_item. A single_turn evaluator scores traces, a thread evaluator scores threads.
The version to score with; default the latest.
Trace targets: the request id (from GET /v1/deployments/{id}/requests or a trace's metadata.veri_request_id).
Thread targets: the deployment and thread id.
Thread targets: the last request id covered; default the thread's latest request at write time.
Required when status is ok: a boolean, one of the labels, or a number in [min, max], per the evaluator's output type. Must be absent otherwise.
ok (default) | error | skipped
Required for status error; optional reason for skipped.
api (default) | sdk | human. Human scores are unique per author.
Hex sha256 of the evaluator source that produced this score. Required
when the pinned version is a code evaluator with runtime: local
(the SDK harness returns it), and must match that version's source;
ignored for every other evaluator.
Experiment-item targets: the experiment and the item
(<row_id>.<trial>, or <row_id>.<trial>.t<turn> for one turn of a
replayed thread). Outside the runner, only a local code evaluator the
experiment pinned takes scores, on an item that is done (see
GET /v1/experiments/{id}/items?pending_local=true), plus human labels
(source: human, any evaluator, a done item) that never count
toward the run.
Response
The stored score
One score.
Always "score".
scr_...
The evaluator's name at write time.
trace | thread
boolean | label string | number; null for error and skipped scores.
api | sdk | human | experiment | monitor
ok | error | skipped
The scored request (trace targets). Trace id = veri-<request_id>.
The thread (thread targets; also stamped on trace targets whose request carried a thread id).
Thread targets: the last request id the score covers.
Experiment-item targets: the experiment.
Experiment-item targets: <row_id>.<trial>, or
<row_id>.<trial>.t<turn> for one turn of a replayed thread.
The evaluator version's pass rule applied to the value; null when the version has no pass rule or the score carries no value.
Why an error / skipped score has no value.
The user who wrote the score.
The model version that answered the scored request, when known.
The monitor that wrote the score (source monitor).
The judge's probability for its verdict, 0..1; null unless a decision-mode judge wrote the score.
The Jev judge's distribution behind the value: noul
{"true": p, "false": 1 - p}, choice {option: p}, score
{"<level>": p}. Null for every other score.
curl --request POST \
--url https://api.veri.studio/v1/scores \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '
{
"evaluator": "<string>",
"target_type": "<string>",
"evaluator_version": 123,
"request_id": "<string>",
"deployment_id": "<string>",
"thread_id": "<string>",
"target_revision": "<string>",
"value": "<unknown>",
"reasoning": "<string>",
"status": "<string>",
"error": "<string>",
"source": "<string>",
"source_sha256": "<string>",
"experiment_id": "<string>",
"item_id": "<string>"
}
'import requests
url = "https://api.veri.studio/v1/scores"
payload = {
"evaluator": "<string>",
"target_type": "<string>",
"evaluator_version": 123,
"request_id": "<string>",
"deployment_id": "<string>",
"thread_id": "<string>",
"target_revision": "<string>",
"value": "<unknown>",
"reasoning": "<string>",
"status": "<string>",
"error": "<string>",
"source": "<string>",
"source_sha256": "<string>",
"experiment_id": "<string>",
"item_id": "<string>"
}
headers = {
"Authorization": "Bearer <token>",
"Content-Type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text)const options = {
method: 'POST',
headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
body: JSON.stringify({
evaluator: '<string>',
target_type: '<string>',
evaluator_version: 123,
request_id: '<string>',
deployment_id: '<string>',
thread_id: '<string>',
target_revision: '<string>',
value: '<unknown>',
reasoning: '<string>',
status: '<string>',
error: '<string>',
source: '<string>',
source_sha256: '<string>',
experiment_id: '<string>',
item_id: '<string>'
})
};
fetch('https://api.veri.studio/v1/scores', options)
.then(res => res.json())
.then(res => console.log(res))
.catch(err => console.error(err));{
"object": "<string>",
"id": "<string>",
"evaluator_id": "<string>",
"evaluator_version": 123,
"name": "<string>",
"target_type": "<string>",
"value": "<unknown>",
"source": "<string>",
"status": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"request_id": "<string>",
"deployment_id": "<string>",
"thread_id": "<string>",
"target_revision": "<string>",
"experiment_id": "<string>",
"item_id": "<string>",
"passed": true,
"reasoning": "<string>",
"error": "<string>",
"author": "<string>",
"model_version_id": "<string>",
"monitor_id": "<string>",
"confidence": 123,
"probabilities": {}
}