REVENUE CYCLE MANAGEMENT / AGENT EVALUATIONS

rcm-benchmark-1

How well do AI agents handle the work of revenue cycle management?

Repeatable evaluations of tool use, workflow execution, and domain knowledge, run through the RCM agent harness.

Evaluated September 29–October 13, 2026 (UTC)Published by RCM APIDownload results JSON

Scenarios: 97
Workflows: 6
Attempts per scenario: 3
Attempts per run: 291

Performance and cost

Higher score · lower cost

Selected models · scroll horizontally on smaller screens

● Complete usage coverage○ Partial cost coverage *

Scores equally weight six workflows. Costs are estimates, not invoices. Models with unavailable pricing are omitted. Hollow points average fully priced attempts only. Complete usage coverage means all saved usage records are priced; Gemini includes no additional thinking-token estimate. All attempts remain in score denominators. Pricing assumptions and provider errors are disclosed below.

Workflow breakdown

Pass rate · selected models

Swipe or scroll to compare models. Category names stay visible.

Category	Claude Opus 5	Gemini 4 Argon	Claude Sonnet 5	GPT-5.6 Sol	Claude Sonnet 5.5	Gemini 3.8 Flash	Claude Opus 5.5	Claude Sonnet 4.6	DeepSeek V4.1 Flash	Claude Haiku 5.5	GLM-5.3	Grok 4.7	Claude Opus 4.6	GPT-6.1 Sol	GPT-5.6 Terra	GPT-6 Luna	GPT-5 mini	GPT-6 Sol	GPT-5.6 Luna	Claude Haiku 4.5
Overall weighted scoreSix equally weighted workflows	92.68% (Best in category, including ties)	91.49%	91.24%	91.21%	91.05%	90.23%	90.14%	89.82%	88.99%	88.38%	88.30%	88.16%	87.85%	87.37%	86.96%	85.81%	84.28%	81.53%	71.48%	70.23%
Agent run costEstimated USD · lower is better	$44.26	$25.11	$14.73	$20.28	$15.06	$5.81	$25.12	$14.27	$3.25Partial subtotal	$0.88	$6.49	$20.77	$24.03Partial subtotal	$6.22	$10.67	$0.56 (Best among displayed costs, including incomplete subtotals, including ties)	$2.46	$8.89	$0.96	$4.16
Claim submission20 scenarios	91.7%55/60 attempts passed	81.7%49/60 attempts passed	93.3% (Best in category, including ties)56/60 attempts passed	81.7%49/60 attempts passed	86.7%52/60 attempts passed	81.7%49/60 attempts passed	85.0%51/60 attempts passed	90.0%54/60 attempts passed	81.7%49/60 attempts passed	86.7%52/60 attempts passed	88.3%53/60 attempts passed	85.0%51/60 attempts passed	78.3%47/60 attempts passed	75.0%45/60 attempts passed	80.0%48/60 attempts passed	78.3%47/60 attempts passed	73.3%44/60 attempts passed	70.0%42/60 attempts passed	65.0%39/60 attempts passed	73.3%44/60 attempts passed
Encounter billing11 scenarios	87.9%29/33 attempts passed	78.8%26/33 attempts passed	87.9%29/33 attempts passed	93.9%31/33 attempts passed	93.9%31/33 attempts passed	81.8%27/33 attempts passed	100.0% (Best in category, including ties)33/33 attempts passed	93.9%31/33 attempts passed	97.0%32/33 attempts passed	84.8%28/33 attempts passed	87.9%29/33 attempts passed	93.9%31/33 attempts passed	100.0% (Best in category, including ties)33/33 attempts passed	90.9%30/33 attempts passed	87.9%29/33 attempts passed	84.8%28/33 attempts passed	87.9%29/33 attempts passed	78.8%26/33 attempts passed	66.7%22/33 attempts passed	81.8%27/33 attempts passed
Patient billing8 scenarios	100.0% (Best in category, including ties)24/24 attempts passed	100.0% (Best in category, including ties)24/24 attempts passed	79.2%19/24 attempts passed	95.8%23/24 attempts passed	100.0% (Best in category, including ties)24/24 attempts passed	100.0% (Best in category, including ties)24/24 attempts passed	100.0% (Best in category, including ties)24/24 attempts passed	87.5%21/24 attempts passed	95.8%23/24 attempts passed	95.8%23/24 attempts passed	87.5%21/24 attempts passed	87.5%21/24 attempts passed	87.5%21/24 attempts passed	95.8%23/24 attempts passed	87.5%21/24 attempts passed	100.0% (Best in category, including ties)24/24 attempts passed	87.5%21/24 attempts passed	100.0% (Best in category, including ties)24/24 attempts passed	79.2%19/24 attempts passed	50.0%12/24 attempts passed
Insurance updates5 scenarios	80.0%12/15 attempts passed	93.3% (Best in category, including ties)14/15 attempts passed	93.3% (Best in category, including ties)14/15 attempts passed	80.0%12/15 attempts passed	73.3%11/15 attempts passed	80.0%12/15 attempts passed	73.3%11/15 attempts passed	80.0%12/15 attempts passed	73.3%11/15 attempts passed	80.0%12/15 attempts passed	80.0%12/15 attempts passed	73.3%11/15 attempts passed	80.0%12/15 attempts passed	80.0%12/15 attempts passed	80.0%12/15 attempts passed	73.3%11/15 attempts passed	80.0%12/15 attempts passed	46.7%7/15 attempts passed	60.0%9/15 attempts passed	60.0%9/15 attempts passed
Eligibility checks5 scenarios	100.0% (Best in category, including ties)15/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	86.7%13/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	93.3%14/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	93.3%14/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	86.7%13/15 attempts passed	93.3%14/15 attempts passed	86.7%13/15 attempts passed	86.7%13/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed	73.3%11/15 attempts passed	100.0% (Best in category, including ties)15/15 attempts passed
RCM knowledge48 scenarios	96.5%139/144 attempts passed	95.1%137/144 attempts passed	93.8%135/144 attempts passed	95.8%138/144 attempts passed	92.4%133/144 attempts passed	97.9% (Best in category, including ties)141/144 attempts passed	95.8%138/144 attempts passed	87.5%126/144 attempts passed	86.1%124/144 attempts passed	89.6%129/144 attempts passed	86.1%124/144 attempts passed	95.8%138/144 attempts passed	81.3%117/144 attempts passed	95.8%138/144 attempts passed	93.1%134/144 attempts passed	91.7%132/144 attempts passed	90.3%130/144 attempts passed	93.8%135/144 attempts passed	84.7%122/144 attempts passed	56.3%81/144 attempts passed
Attempts passedOut of 291 attempts	274/291 (Best in category, including ties)	265/291	268/291	268/291	266/291	268/291	270/291	259/291	254/291	258/291	254/291	266/291	245/291	261/291	258/291	255/291	249/291	249/291	222/291	188/291
Agent cost / attemptEstimated USD · lower is better	$0.1521291/291 attempts fully priced	$0.0863291/291 attempts fully priced	$0.0506291/291 attempts fully priced	$0.0697291/291 attempts fully priced	$0.0518291/291 attempts fully priced	$0.0200291/291 attempts fully priced	$0.0863291/291 attempts fully priced	$0.0490291/291 attempts fully priced	$0.0112290/291 attempts fully priced	$0.0030291/291 attempts fully priced	$0.0223291/291 attempts fully priced	$0.0714291/291 attempts fully priced	$0.0829290/291 attempts fully priced	$0.0214291/291 attempts fully priced	$0.0367291/291 attempts fully priced	$0.0019 (Best among fully priced attempts, including ties)291/291 attempts fully priced	$0.0085291/291 attempts fully priced	$0.0305291/291 attempts fully priced	$0.0033291/291 attempts fully priced	$0.0143291/291 attempts fully priced
Input token costUSD / 1M tokens · see pricing notes	$5.50	$2.00	$2.00	$5.00	$2.00	$0.75	$4.00	$3.00	$0.30	$0.10 (Best token rate, including ties)	$1.40	$2.00	$5.00	$2.00	$2.50	$0.10 (Best token rate, including ties)	$0.25	$2.00	$0.20	$1.00
Cached input token costUSD / 1M tokens · see pricing notes	$0.55	$0.10	$0.20	$0.50	$0.20	$0.075	$0.20	$0.30	$0.03	$0.01 (Best token rate, including ties)	$0.14	$0.50	$0.50	$0.10	$0.25	$0.01 (Best token rate, including ties)	$0.025	$0.20	$0.02	$0.10
Output token costUSD / 1M tokens · see pricing notes	$27.50	$10.00	$10.00	$30.00	$10.00	$3.75	$20.00	$15.00	$1.20	$0.50 (Best token rate, including ties)	$4.40	$6.00	$25.00	$10.00	$15.00	$0.50 (Best token rate, including ties)	$2.00	$10.00	$1.20	$5.00

Cost per attempt averages fully priced, non-error attempts, including attempts that failed the eval. Run costs include recorded error spend and remain partial when coverage is incomplete. Judge, tool-internal model calls, and infrastructure are excluded. Estimates use the harness catalog and cache-rate assumptions, not provider invoices. DeepSeek V4.1 Flash and Opus 4.6 each have one provider error and 290/291 priced attempts; their run costs are partial subtotals. GPT-6 Sol and Luna estimates use all 291 usage records and assume OpenAI Standard short-context rates; request-level billing tiers are unavailable, and long-context, Fast, and regional premiums are excluded. Opus 5 uses all 291 usage records with published standard rates, an assumed 10% US Bedrock premium, and five-minute cache writes. Opus 5.5 uses all 291 usage records with standard direct Anthropic rates and five-minute cache writes. These assumptions are not invoice-verified. GPT-6.1 Sol uses all 291 usage records at Standard short-context rates verified September 29: $2 input, $0.10 cached input, $2.50 cache writes, and $10 output per million tokens. Its estimate excludes long-context and service-tier premiums. Gemini 3.8 Flash uses Google Standard paid ratesthrough December 31, 2026: $0.75 input, $0.075 cached input, and $3.75 output per million tokens. Its $5.81 recorded-usage estimate covers all 291 saved usage records ($0.01998 per attempt), with no additional thinking-token estimate. Claude Haiku 5.5 uses all 291 saved usage records at published Standard rates for prompts up to 100k tokens: $0.10 input, $0.01 cache reads, $0.125 five-minute cache writes, and $0.50 output per million tokens ($0.88 total; $0.00303 per attempt). Request-level context lengths were not retained; the estimate assumes the lower tier, while requests over 100k cost five times these rates. Gemini 4 Argon uses Vertex global Standard rates: $25.11 after promotional credits through November 30, 2026, or $50.21 at full rates. The promotion is delivered as 50% credits back; receipt is not invoice-verified. All 291 saved usage records are priced, with no additional thinking-token estimate. Unavailable costs are not zero.

BENCHMARK DESIGN

One harness. Six kinds of work.

Claim submission

Investigate, correct, and submit claims; handle approval and escalation.

20 scenarios · 16.67% of overall score

Encounter billing

Adjust balances, apply write-offs, and manage patient responsibility.

11 scenarios · 16.67% of overall score

Patient billing

Explain balances, apply credits, and handle refunds and permissions.

8 scenarios · 16.67% of overall score

Insurance updates

Update coverage, map payers, and relink insurance to claims.

5 scenarios · 16.67% of overall score

Eligibility checks

Check coverage, recheck updates, and handle inconclusive results.

5 scenarios · 16.67% of overall score

RCM knowledge

Answer revenue cycle management knowledge questions.

48 scenarios · 16.67% of overall score

What is evaluated?

Each scenario provides conversation context, a defined test environment, and expected behavior. Checks can cover required or forbidden tool calls, arguments, successful actions, response content, and resulting state. Cases use mocked tool responses or isolated backend environments.

How is the score calculated?

Each case runs three times. Pass/fail outcomes are averaged per case, then across cases within a workflow. The overall score is the mean of the six workflow scores. All attempts, including failures, stay in the denominator.

What can the results tell us?

Scores describe the evaluated agent configuration: model, prompts, tools, provider, and code revision. They do not establish standalone model quality or production success rates. Three attempts per case provide limited evidence about variability.

Which runs are shown?

This snapshot includes the 16 completed runs from the September 29 revised harness batch, the first GPT-6.1 Sol run, the October 5 Gemini 3.8 Flash run, the October 7 Claude Haiku 5.5 run, and the October 13 Gemini 4 Argon run, each with 97 cases and 291 attempts. Model/provider identity was checked across every trial artifact. Provider errors still count as failures. Cost coverage and reconstruction assumptions are disclosed per model. Gemini and Haiku 5.5 use later harness revisions. Five benchmark case files and the tool grader changed between the two Gemini runs. These runs are not a controlled same-revision comparison. Comparable token-level latency is not reported.

Reasoning settings

Medium: GPT-6.1 Sol, GPT-6 Sol, GPT-6 Luna, GPT-5.6 Sol, and GPT-5.6 Terra used an explicit medium reasoning effort through the OpenAI Responses API. Gemini 3.8 Flash and Gemini 4 Argon used medium reasoning through Google’s OpenAI-compatible endpoint, with default temperature and buffered responses.

Provider default: GPT-5 mini, Claude Opus 5.5, Claude Opus 5, Claude Opus 4.6, Claude Sonnet 5.5, Claude Sonnet 5, Claude Sonnet 4.6, Claude Haiku 5.5, Claude Haiku 4.5, Grok 4.7, DeepSeek V4.1 Flash, and GLM-5.3 used their provider defaults. The evaluated adapters did not explicitly set a reasoning effort or thinking budget for these models. This does not mean reasoning was disabled or that the provider offers no reasoning controls.

Tool compatibility exception: GPT-5.6 Luna used reasoning effort none with tools through Chat Completions. These results therefore do not represent a uniform medium-reasoning comparison across all models.

Run specifications

291/291 attempts recorded in each run

Gemini 4 ArgonSingle model; medium reasoning

All 291 trial artifacts identify gemini-4-argon on Gemini via Vertex AI global, with completed engine status and no recorded agent/provider errors. Original grading: 265/291 attempts passed across 97 cases, three trials each. Medium reasoning, default temperature, buffered responses. Recorded-usage cost is $25.1070706 after promotional credits, or $50.2141412 at full rates; no additional thinking-token estimate is included.

Provider configuration: Google Vertex AI (global)
Reasoning setting: Medium
Evaluation date: 2026-10-13
Run identifier: 38436156750
Code revision: 539012ace88a97da56eb50b0bcb815d538428874
Agent errors recorded: 0 / 291

Claude Haiku 5.5Single model; provider-default reasoning

All 291 trial artifacts identify claude-haiku-5-5 on direct Anthropic with completed engine status and no recorded engine/provider errors. Original grading: 258/291 attempts passed; assistant 169/195 and email 89/96. Provider-default reasoning with no explicit effort or thinking budget. All 291 saved usage records are priced at published Standard rates for prompts up to 100k tokens, assuming five-minute cache writes. Request-level context lengths were not retained, so long-context pricing is not verified.

Provider configuration: Anthropic (direct)
Reasoning setting: Provider default
Evaluation date: 2026-10-07
Run identifier: 37666178262
Code revision: 1d94adad588c5151607fdbc88b146e79f0273f2b
Agent errors recorded: 0 / 291

Gemini 3.8 FlashSingle model; medium reasoning

All 291 trial artifacts identify gemini-3.8-flash on gemini with completed engine status and no recorded engine/provider errors. Original grading: 268/291 attempts passed across 97 cases, three trials per case. Medium reasoning and default temperature. Mean agent execution time was 21.0 seconds per trial; this is not token-level latency, which the buffered adapter does not expose. Recorded-usage estimate is $5.8129509 at Google Standard paid rates effective through December 31, 2026. All 291 saved usage records are priced; no additional thinking-token estimate is included.

Provider configuration: Google Gemini (OpenAI-compatible endpoint)
Reasoning setting: Medium
Evaluation date: 2026-10-05
Run identifier: 37378208147
Code revision: afc7667901bb1243a2de3250f92f391ab5048a65
Agent errors recorded: 0 / 291

GPT-6.1 SolSingle model; medium reasoning

All 291 artifacts identify gpt-6.1-sol on OpenAI via Responses API with medium reasoning; 290 completed and one tool-limit stop, retained as a failure. Cost reconstructed from all 291 usage records at Standard short-context rates verified September 29; request-level billing tiers are unavailable.

Provider configuration: OpenAI
Reasoning setting: Medium
Evaluation date: 2026-09-29
Run identifier: 36604788518
Code revision: b27df812545618fbfe306505b725f39433479c75
Agent errors recorded: 0 / 291

Claude Opus 5Single model; provider-default reasoning

All 291 trial artifacts verify claude-opus-5 on bedrock. Eight concurrent trials. 0 engine/provider errors remain in the score. Standard Opus 5 pricing with an assumed 10% US Bedrock premium and five-minute cache writes; routing premium is not invoice-verified.

Provider configuration: AWS Bedrock (no fallback)
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502531672
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

Claude Sonnet 5Single model; provider-default reasoning

All 291 trial artifacts verify claude-sonnet-5 on bedrock. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: AWS Bedrock (no fallback)
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502537207
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

GPT-5.6 SolSingle model; medium reasoning

All 291 trial artifacts verify gpt-5.6-sol on openai. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: OpenAI
Reasoning setting: Medium
Evaluation date: 2026-09-29
Run identifier: 36502547281
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

Claude Sonnet 5.5Single model; provider-default reasoning

All 291 trial artifacts verify claude-sonnet-5-5 on anthropic. One concurrent trial. 0 engine/provider errors remain in the score.

Provider configuration: Anthropic
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502568619
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

Claude Opus 5.5Single model; provider-default reasoning

All 291 trial artifacts verify claude-opus-5-5 on anthropic, with 97 cases and no engine/provider errors. One concurrent trial. Cost estimated from all 291 usage records.

Provider configuration: Anthropic (direct)
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36518313926
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

Claude Sonnet 4.6Single model; provider-default reasoning

All 291 trial artifacts verify claude-sonnet-4-6 on bedrock. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: AWS Bedrock (no fallback)
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502539666
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

DeepSeek V4.1 FlashSingle model; provider-default reasoning

All 291 trial artifacts verify deepseek-ai/DeepSeek-V4.1-Flash on baseten. Eight concurrent trials. One engine/provider error is included in the score.

Provider configuration: Baseten
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502560191
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 1 / 291

GLM-5.3Single model; provider-default reasoning

All 291 trial artifacts verify zai-org/GLM-5.3 on baseten. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: Baseten
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502562802
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

Grok 4.7Single model; provider-default reasoning

All 291 trial artifacts verify grok-4.7 on xai. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: xAI
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502565865
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

Claude Opus 4.6Single model; provider-default reasoning

All 291 trial artifacts verify claude-opus-4-6 on bedrock. Eight concurrent trials. One engine/provider error is included in the score.

Provider configuration: AWS Bedrock (no fallback)
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502534368
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 1 / 291

GPT-5.6 TerraSingle model; medium reasoning

All 291 trial artifacts verify gpt-5.6-terra on openai. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: OpenAI
Reasoning setting: Medium
Evaluation date: 2026-09-29
Run identifier: 36502555491
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

GPT-6 LunaSingle model; medium reasoning

All 291 trial artifacts verify gpt-6-luna on openai. Eight concurrent trials. 0 engine/provider errors remain in the score. Standard short-context OpenAI pricing; per-request tiers unavailable; excludes long-context, Fast and regional premiums.

Provider configuration: OpenAI
Reasoning setting: Medium
Evaluation date: 2026-09-29
Run identifier: 36502549945
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

GPT-5 miniSingle model; provider-default reasoning

All 291 trial artifacts verify gpt-5-mini on openai. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: OpenAI
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502557823
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

GPT-6 SolSingle model; medium reasoning

All 291 trial artifacts verify gpt-6-sol on openai. Eight concurrent trials. 0 engine/provider errors remain in the score. Standard short-context OpenAI pricing; per-request tiers unavailable; excludes long-context, Fast and regional premiums.

Provider configuration: OpenAI
Reasoning setting: Medium
Evaluation date: 2026-09-29
Run identifier: 36502544877
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

GPT-5.6 LunaSingle model; reasoning disabled for tool compatibility

All 291 trial artifacts verify gpt-5.6-luna on openai. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: OpenAI
Reasoning setting: None (tool compatibility)
Evaluation date: 2026-09-29
Run identifier: 36502552471
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

Claude Haiku 4.5Single model; provider-default reasoning

All 291 trial artifacts verify claude-haiku-4-5 on bedrock. Eight concurrent trials. 0 engine/provider errors remain in the score.

Provider configuration: AWS Bedrock (no fallback)
Reasoning setting: Provider default
Evaluation date: 2026-09-29
Run identifier: 36502542258
Code revision: 2a90be400a5a88c88766e2f986cd3f0f82d52c94
Agent errors recorded: 0 / 291

Source: archived benchmark reports and agent outputs. Only aggregate results are published; source runs and case transcripts are not public. Runner-level completeness does not imply the absence of agent or provider errors.