Core concepts
Jobs, statuses, the difference between finishing and passing, artifacts, and tenancy.
Five ideas cover almost everything you will meet in the API. The one that catches people out is the third.
Synchronous vs. asynchronous
Most work can be submitted either way, and the difference is only who does the waiting.
| Synchronous | Asynchronous | |
|---|---|---|
| Example | POST /sandbox/harbor/v2/run/datapoint | POST /sandbox/harbor/v2/jobs/execute-tasks |
| Returns | 200 with the complete result | 202 with a job_id |
| Good for | Short runs where holding a connection is fine | Anything real — long tasks, many agents, cancellation |
Both put the work on the same queue and the same worker runs it; the synchronous
route just waits for the verdict on your behalf. So the synchronous route is not a
way around a busy queue, and it has one failure mode worth planning for: if the run
outlasts the wait you get 504, the run keeps going, and the job id is in the
response headers. Poll it — do not resubmit, or you pay for the task twice.
A job and its lifecycle
An asynchronous submission creates a job you then poll.
status moves through:
| Status | Meaning |
|---|---|
queued | Accepted, waiting for capacity |
running | A worker is executing it |
succeeded | The run finished and produced at least one real result |
failed | Nothing usable came out, or the platform gave up on it |
cancelled | You asked it to stop |
The last three are terminal — stop polling. Rather than polling, you can follow
GET .../jobs/{job_id}/events as a server-sent event stream, and resume after a
dropped connection with Last-Event-ID.
status is not a verdict
This is the one that matters. A finished job carries two judgements, and treating them as one will eventually make you score a run that never happened.
statusanswers did the orchestration finish?outcome.valid_for_scoringanswers should you believe the numbers?
Gate your scoring on outcome.valid_for_scoring. A job can be succeeded
while its results are explicitly not trustworthy — for example, a verifier that
exited zero without collecting a single test has demonstrated nothing, so the
platform withholds the claim rather than granting it.
Rewards are never rewritten: whatever the run recorded stays in
agent_results[].result, including one the platform is declining to vouch for, so
an audit can always see what happened.
Full field-by-field breakdown: Run a benchmark.
Artifacts
A run produces files — logs, verifier reports, whatever the agent wrote. They are not returned inline. You list them, then download.
| Call | Gives you |
|---|---|
GET .../jobs/{job_id}/artifacts | An inventory of what exists |
GET .../jobs/{job_id}/artifacts/archive | A time-limited URL for everything as one archive |
GET .../jobs/{job_id}/artifacts/test-stdout/{path} | One verifier's captured output |
Artifacts expire. Download what you need to keep.
Tenancy: tenant and project
Every request resolves to a tenant:project pair, which is what the platform bills,
rate-limits and isolates against. Where it comes from depends on your credential —
a team token carries its own and cannot claim another.
The consequence you will actually notice: a job is visible only to a request with
the same tenancy it was submitted under. Asking for someone else's job returns
404, identically to a job that does not exist, because a 403 would confirm the
id is real.
How that resolves per credential: Authentication.
Correlation id
Send x-correlation-id on every request and the same id appears in logs, traces,
the worker, and LLM observability — one string to search when you need to explain
what a run did. Omit it and the gateway generates one and echoes it back.
Use one stable id per logical run, not per HTTP call: Observability.