Experiments
Diff an experiment against a baseline
Compares a completed experiment with a baseline experiment, per evaluator: the items that newly fail, newly pass or still fail, plus pass-rate, mean and error deltas. Two runs are comparable only on the same dataset snapshot with identical evaluator pins (versions and judge models) and bindings; otherwise the body is {comparable: false, reason}.
GET
/
v1
/
experiments
/
{experiment_id}
/
diff
/
{baseline_id}
Diff an experiment against a baseline
curl --request GET \
--url https://api.veri.studio/v1/experiments/{experiment_id}/diff/{baseline_id} \
--header 'Authorization: Bearer <token>'import requests
url = "https://api.veri.studio/v1/experiments/{experiment_id}/diff/{baseline_id}"
headers = {"Authorization": "Bearer <token>"}
response = requests.get(url, headers=headers)
print(response.text)const options = {method: 'GET', headers: {Authorization: 'Bearer <token>'}};
fetch('https://api.veri.studio/v1/experiments/{experiment_id}/diff/{baseline_id}', options)
.then(res => res.json())
.then(res => console.log(res))
.catch(err => console.error(err));{}Authorizations
API key with the vk_ prefix. Create one from the dashboard.
Path Parameters
The experiment (exp_...)
The baseline experiment (exp_...)
Response
{comparable: true, baseline_experiment_id, evaluators: [{evaluator_id, version, name, newly_failing, newly_passing, still_failing, not_comparable, pass_rate, mean, errors}]} or {comparable: false, baseline_experiment_id, reason}
The response is of type object.
Diff an experiment against a baseline
curl --request GET \
--url https://api.veri.studio/v1/experiments/{experiment_id}/diff/{baseline_id} \
--header 'Authorization: Bearer <token>'import requests
url = "https://api.veri.studio/v1/experiments/{experiment_id}/diff/{baseline_id}"
headers = {"Authorization": "Bearer <token>"}
response = requests.get(url, headers=headers)
print(response.text)const options = {method: 'GET', headers: {Authorization: 'Bearer <token>'}};
fetch('https://api.veri.studio/v1/experiments/{experiment_id}/diff/{baseline_id}', options)
.then(res => res.json())
.then(res => console.log(res))
.catch(err => console.error(err));{}