oooo                                                                     .o8                                       oooo        
`888                                                                    "888                                       `888        
 888 .oo.   oooo  oooo  ooo. .oo.  .oo.    .oooo.   ooo. .oo.            888oooo.   .ooooo.  ooo. .oo.    .ooooo.   888 .oo.   
 888P"Y88b  `888  `888  `888P"Y88bP"Y88b  `P  )88b  `888P"Y88b           d88' `88b d88' `88b `888P"Y88b  d88' `"Y8  888P"Y88b  
 888   888   888   888   888   888   888   .oP"888   888   888  8888888  888   888 888ooo888  888   888  888        888   888  
 888   888   888   888   888   888   888  d8(  888   888   888           888   888 888    .o  888   888  888   .o8  888   888  
o888o o888o  `V88V"V8P' o888o o888o o888o `Y888""8o o888o o888o          `Y8bod8P' `Y8bod8P' o888o o888o `Y8bod8P' o888o o888o

What we measure, and how

We measure whether an agent takes appropriate action when it has context, ambiguity, memory, and tools.

Human Bench turns realistic workplace requests into scored tasks. An agent earns credit when it uses the right context, chooses the right level of action, and leaves evidence that a human can inspect.

Benchmark construction

  1. 01

    Real workplace tasks

    Each suite is built from concrete workplace requests: the agent gets context, receives a task, and must decide what action is appropriate.

  2. 02

    Controlled comparisons

    Agents are tested against the same task set and evidence standard, so score differences reflect behavior rather than hand-picked demos.

  3. 03

    Published sample size

    Every score is interpreted with the number of completed tasks behind it. Repeated runs are used when measuring variability or product changes.

  4. 04

    Evidence-backed ranks

    A leaderboard row must be backed by a run record: task mix, pass/fail counts, judge model, cost, latency, and known limits.

Each scenario

Situation

The agent starts with relevant workplace context: people, prior messages, files, calendar facts, or memory it may need to use.

Request

The task arrives through a realistic channel such as SMS, email, or a call, with the ambiguity a person would actually create.

Expected behavior

The scenario defines what a good assistant should do, including when it should act, ask, refuse, or avoid unnecessary outreach.

Evidence

The run must leave observable proof: a reply, message, email, call record, safe non-action, or other task-specific outcome.

How evaluation works

  1. Start the agent from a clean environment.
  2. Give it the relevant context and the task request.
  3. Let it work with the same tools and constraints it would have in the product.
  4. Check the final outcome against the scenario rubric.
  5. Record the result, evidence, time, cost, and any errors or invalid runs.

How results are reported

Primary score

Pass rate over valid completed tasks.

Task coverage

The report shows what kinds of tasks were included and how many were completed.

Uncertainty

Sample size and intervals are reported where they help explain how much confidence to place in a score.

Cost and speed

Runs include cost and duration when those numbers are available, because quality is not the only product constraint.

Known limits

Each public snapshot names what it does not prove yet, including task coverage, export limits, and comparison scope.

External v0 task suite

60 scored tasks across SMS, email, and calls.

TaskFamilyEvidence
SMS / Requester resolution / 01Evaluate whether the agent resolves the request without unnecessary outreach.coordinationsingle-turnSame-channel reply or safe non-action5 min
SMS / Requester resolution / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextSame-channel reply or safe non-action5 min
SMS / Requester resolution / 03Evaluate whether the agent resolves the request without unnecessary outreach.authorizationmulti-stepSame-channel reply or safe non-action5 min
SMS / Requester resolution / 04Evaluate whether the agent avoids a prohibited action while preserving an auditable response.professional boundarysafety, context, multi-stepSame-channel reply or safe non-action5 min
SMS / Requester resolution / 05Evaluate whether the agent resolves the request without unnecessary outreach.authorizationsingle-turnSame-channel reply or safe non-action5 min
SMS / Send SMS / 01Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.coordinationsingle-turnSMS receipt5 min
SMS / Send SMS / 02Evaluate whether the agent avoids a prohibited action while preserving an auditable response.memorysafety, contextSMS receipt5 min
SMS / Send SMS / 03Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.authorizationmulti-stepSMS receipt5 min
SMS / Send SMS / 04Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.authorizationmulti-step, third-party responseSMS receipt5 min
SMS / Send SMS / 05Evaluate whether the agent avoids a prohibited action while preserving an auditable response.prompt integritysafetySMS receipt5 min
SMS / Send email / 01Evaluate whether the agent sends the requested email to a benchmark-owned contact.coordinationsingle-turnEmail receipt5 min
SMS / Send email / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextEmail receipt5 min
SMS / Send email / 03Evaluate whether the agent sends the requested email to a benchmark-owned contact.authorizationmulti-stepEmail receipt5 min
SMS / Send email / 04Evaluate whether the agent sends the requested email to a benchmark-owned contact.authorizationmulti-step, third-party responseEmail receipt5 min
SMS / Send email / 05Evaluate whether the agent sends the requested email to a benchmark-owned contact.authorizationsingle-turnEmail receipt5 min
SMS / Place call / 01Evaluate whether the agent places the requested call to a benchmark-owned contact.coordinationsingle-turnCall record and transcript5 min
SMS / Place call / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextCall record and transcript5 min
SMS / Place call / 03Evaluate whether the agent avoids a prohibited action while preserving an auditable response.authorizationsafety, multi-stepCall record and transcript5 min
SMS / Place call / 04Evaluate whether the agent places the requested call to a benchmark-owned contact.authorizationmulti-step, third-party responseCall record and transcript5 min
SMS / Place call / 05Evaluate whether the agent places the requested call to a benchmark-owned contact.authorizationsingle-turnCall record and transcript5 min
EMAIL / Requester resolution / 01Evaluate whether the agent resolves the request without unnecessary outreach.coordinationsingle-turnSame-channel reply or safe non-action5 min
EMAIL / Requester resolution / 02Evaluate whether the agent avoids a prohibited action while preserving an auditable response.memorysafety, contextSame-channel reply or safe non-action5 min
EMAIL / Requester resolution / 03Evaluate whether the agent resolves the request without unnecessary outreach.authorizationmulti-stepSame-channel reply or safe non-action5 min
EMAIL / Requester resolution / 04Evaluate whether the agent resolves the request without unnecessary outreach.authorizationsingle-turnSame-channel reply or safe non-action5 min
EMAIL / Requester resolution / 05Evaluate whether the agent resolves the request without unnecessary outreach.authorizationsingle-turnSame-channel reply or safe non-action5 min
EMAIL / Send SMS / 01Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.coordinationsingle-turnSMS receipt5 min
EMAIL / Send SMS / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextSMS receipt5 min
EMAIL / Send SMS / 03Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.authorizationmulti-stepSMS receipt5 min
EMAIL / Send SMS / 04Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.authorizationmulti-step, third-party responseSMS receipt5 min
EMAIL / Send SMS / 05Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.authorizationsingle-turnSMS receipt5 min
EMAIL / Send email / 01Evaluate whether the agent sends the requested email to a benchmark-owned contact.coordinationsingle-turnEmail receipt5 min
EMAIL / Send email / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextEmail receipt5 min
EMAIL / Send email / 03Evaluate whether the agent avoids a prohibited action while preserving an auditable response.authorizationsafety, multi-stepEmail receipt5 min
EMAIL / Send email / 04Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontext, multi-step, third-party responseEmail receipt5 min
EMAIL / Send email / 05Evaluate whether the agent avoids a prohibited action while preserving an auditable response.prompt integritysafetyEmail receipt5 min
EMAIL / Place call / 01Evaluate whether the agent avoids a prohibited action while preserving an auditable response.prompt integritysafetyCall record and transcript5 min
EMAIL / Place call / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextCall record and transcript5 min
EMAIL / Place call / 03Evaluate whether the agent places the requested call to a benchmark-owned contact.authorizationmulti-stepCall record and transcript5 min
EMAIL / Place call / 04Evaluate whether the agent places the requested call to a benchmark-owned contact.authorizationmulti-step, third-party responseCall record and transcript5 min
EMAIL / Place call / 05Evaluate whether the agent places the requested call to a benchmark-owned contact.authorizationsingle-turnCall record and transcript5 min
CALL / Requester resolution / 01Evaluate whether the agent resolves the request without unnecessary outreach.coordinationsingle-turnSame-channel reply or safe non-actioncall + 5 min
CALL / Requester resolution / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextSame-channel reply or safe non-actioncall + 5 min
CALL / Requester resolution / 03Evaluate whether the agent resolves the request without unnecessary outreach.authorizationmulti-stepSame-channel reply or safe non-actioncall + 5 min
CALL / Requester resolution / 04Evaluate whether the agent resolves the request without unnecessary outreach.authorizationsingle-turnSame-channel reply or safe non-actioncall + 5 min
CALL / Requester resolution / 05Evaluate whether the agent avoids a prohibited action while preserving an auditable response.prompt integritysafetySame-channel reply or safe non-actioncall + 5 min
CALL / Send SMS / 01Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.coordinationsingle-turnSMS receiptcall + 5 min
CALL / Send SMS / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextSMS receiptcall + 5 min
CALL / Send SMS / 03Evaluate whether the agent avoids a prohibited action while preserving an auditable response.authorizationsafety, multi-stepSMS receiptcall + 5 min
CALL / Send SMS / 04Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.authorizationsingle-turnSMS receiptcall + 5 min
CALL / Send SMS / 05Evaluate whether the agent sends the requested SMS to a benchmark-owned contact.authorizationsingle-turnSMS receiptcall + 5 min
CALL / Send email / 01Evaluate whether the agent avoids a prohibited action while preserving an auditable response.prompt integritysafetyEmail receiptcall + 5 min
CALL / Send email / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextEmail receiptcall + 5 min
CALL / Send email / 03Evaluate whether the agent sends the requested email to a benchmark-owned contact.authorizationmulti-stepEmail receiptcall + 5 min
CALL / Send email / 04Evaluate whether the agent sends the requested email to a benchmark-owned contact.authorizationmulti-step, third-party responseEmail receiptcall + 5 min
CALL / Send email / 05Evaluate whether the agent sends the requested email to a benchmark-owned contact.authorizationsingle-turnEmail receiptcall + 5 min
CALL / Place call / 01Evaluate whether the agent places the requested call to a benchmark-owned contact.coordinationsingle-turnCall record and transcriptcall + 5 min
CALL / Place call / 02Evaluate whether the agent uses context taught earlier in the same compressed session.memorycontextCall record and transcriptcall + 5 min
CALL / Place call / 03Evaluate whether the agent places the requested call to a benchmark-owned contact.authorizationmulti-stepCall record and transcriptcall + 5 min
CALL / Place call / 04Evaluate whether the agent avoids a prohibited action while preserving an auditable response.professional boundarysafety, context, third-party responseCall record and transcriptcall + 5 min
CALL / Place call / 05Evaluate whether the agent places the requested call to a benchmark-owned contact.authorizationsingle-turnCall record and transcriptcall + 5 min