Technical overview
How this environment evaluates agents.
The benchmark tests laboratory medicine reasoning rather than short-vignette diagnosis. The agent has to understand that lab values change meaning under pregnancy, warfarin therapy, kidney disease, medication exposure, or multi-panel syndrome context.
Scenarios include hyperkalemia, acute myocardial infarction, severe anemia, pregnancy-adjusted hemoglobin, supratherapeutic warfarin INR, drug-induced hyperkalemia, disseminated intravascular coagulation, and tumor lysis syndrome.
The runtime uses an in-memory SQLite database with eight relational tables covering patients, medications, lab orders, lab results, previous results, diagnostic reports, critical alerts, and pending cases.
Rewards are deterministic, dense, scenario-specific, and bounded from 0.01 to 0.99, with score breakdowns that expose what the model inspected, missed, flagged, or diagnosed incorrectly.
A laboratory workflow, not a diagnosis quiz
The episode begins with a fresh in-memory SQLite LIMS and a hidden scenario selected from the requested difficulty tier. The agent discovers pending cases, identifies the relevant patient among distractors, gathers demographics and medication context, inspects orders and analyte-level results, checks previous values and context-sensitive reference ranges, files critical alerts when justified, and closes the case with an ICD-10 code, severity, and clinical notes.
Nine structured commands define the action space: list pending cases; retrieve demographics, medications, lab orders, lab results, and previous results; query reference ranges; flag a critical value; and submit the diagnostic report. The action contract is validated by an OpenEnv-compatible FastAPI runtime, while observations return deterministic JSON or structured text rather than exposing the answer key.
Eight scenarios across three reasoning tiers
Easy episodes cover potassium 7.2 mEq/L hyperkalemia, Troponin I 2.8 ng/mL acute myocardial infarction, and hemoglobin 6.2 g/dL severe anemia. Medium episodes require context: pregnancy-adjusted hemoglobin, supratherapeutic warfarin INR, and medication-associated hyperkalemia in chronic kidney disease. Hard episodes require multi-panel synthesis for disseminated intravascular coagulation and tumor lysis syndrome.
The benchmark is deliberately resistant to shortcuts. Sixteen distractor patients, neutral case descriptions, patient-specific ranges, medication dependencies, delta checks, false-positive penalties, and multi-panel syndromes prevent a model from earning full credit through one abnormal value or a guessed code.
Deterministic process rewards and inspectable outputs
Scenario-specific graders award credit for the investigative steps that make the diagnosis defensible and subtract for unsafe shortcuts. Relevant demographic, medication, result, history, and range lookups contribute alongside correct critical alerts, ICD-10 coding, and severity. Scores are bounded from 0.01 to 0.99 and the final observation explains the investigation checklist, submitted answer, expected answer, missed evidence, and penalties.
Each run can produce step logs, final rewards, score breakdowns, and trajectory JSON. That supports controlled model comparison, tool-policy analysis, process-supervision data, supervised or preference learning from trajectories, offline reinforcement learning, and clinical-agent failure analysis. The server is CPU-only and the package includes deterministic grader tests for all eight scenario variants.