Technical overview
How this environment evaluates agents.
Digital Hospital evaluates process, not only final medical answers. The agent must triage, inspect charts, use diagnostic tools, recognize critical findings, communicate when appropriate, and submit evidence-supported treatment plans.
Roles include emergency physician, intensivist, cardiologist, surgeon, nephrologist, senior hospitalist, neurologist, infectious disease physician, clinical pharmacist, clinical research expert, and hospital director.
The package includes patient records, hidden answer keys, role-specific MCQ banks, FastAPI runtime, tool handlers, graders, communication routes, trajectory collection scripts, Docker configuration, and inference runners.
The environment supports model evaluation, process-supervision datasets, offline RL experiments, multi-agent workflow research, director-level feedback, and reproducible failure analysis through deterministic hidden grading logic.
An eleven-role hospital operating environment
Digital Hospital evaluates nine specialist roles plus a Clinical Research Expert and Hospital Director. Specialist episodes combine role-specific medical knowledge checks with patient operations: triage the waiting room, select the urgent patient, read the chart, gather evidence through permitted tools, flag critical findings, inspect treatment options, and submit a diagnosis, severity, treatment IDs, and rationale.
The roles have different permissions and responsibilities. Emergency medicine, intensive care, cardiology, surgery, nephrology, hospital medicine, neurology, infectious disease, and pharmacy each receive five specialist cases. C01 handles evidence requests and research communication; D01 combines two direct cases with clinic-level performance review. Full runs proceed from S01 through S09, then C01 and D01 so later roles can consume structured context from earlier work.
A substantial, auditable logic surface
The downloadable package contains 47 patient records, 47 hidden grading keys, 550 role-specific MCQs with 2,200 answer options, 55 unique tools, and 13 Python runtime files totaling 7,436 non-comment lines. A conservative static audit counts 11,280 custom logic elements, including 201 functions, 969 decision nodes, 549 reward or grading references, and 3,834 scalar grading atoms inside the hidden case keys.
Patient-visible records contain demographics, histories, medications, vitals, labs, imaging, ECG findings, prior results, and treatment choices. Server-side keys specify diagnosis, ICD-10 code, severity, mandatory evidence tools, critical findings, correct treatment IDs and actions, partial-credit events, traps, score floors, and ceilings. The FastAPI state machine owns sessions, role permissions, cross-role communication, grading, and trajectory output.
Process supervision, anti-shortcut grading, and rollouts
Dense rewards capture process as well as outcome. Diagnostic calls receive diminishing returns, role-based access control blocks out-of-scope tools, active-patient locking prevents abandonment, step budgets constrain endless exploration, and hidden keys remain outside observations. Treatment submissions fall into comprehensive, inconclusive, expedited, or insufficient bands depending on evidence coverage, critical-finding recognition, diagnostic specificity, treatment alignment, and process safety.
Trajectories record the session, role, phase, step, observation, structured command, arguments, reward, completion state, errors, and contextual metadata. The included runners can execute isolated single-role checks or all eleven roles, producing logs, JSON results, Parquet trajectories, and aggregate summaries. This makes the environment useful for benchmark evaluation, SFT and preference-data generation, reward-weighted learning, offline policy improvement, and repeatable analysis of why a clinical agent failed.