What published studies show about LLMs for trial matching

What have LLM trial-matching studies tested so far?

Most early studies are benchmarks: a set of patient records with answers agreed by experts, and a score for how often a system agrees. The best-known public set comes from the 2018 n2c2 shared task, which annotated records for 288 patients against 13 criteria taken from real trials. Of 47 teams, the top system reached a micro F1 score of 0.91 with a rule-based approach, before LLMs were part of the picture.^⁠[1],[2]

In 2024 a UT Southwestern team ran GPT-4 over 202 records from that challenge and reported accuracy of 0.87, sensitivity of 0.85 and specificity of 0.89, up from 0.81 accuracy with GPT-3.5 Turbo. They also noted that some labels in the answer key appeared to be wrong.^⁠[2]

TrialGPT, from the National Library of Medicine, matched 183 synthetic patients to trials. On 1,015 patient-criterion pairs checked by hand, its criterion-level accuracy was 87.3 percent, and in a user study it cut screening time by 42.6 percent.^⁠[3] A 2025 scoping review found 24 studies of LLM-based matching published between December 2022 and December 2024, 21 of them in 2024, and named performance variability, interpretability and reliance on synthetic data sets as open problems.^⁠[4]

What did the RECTIFIER studies at Mass General Brigham show?

RECTIFIER pairs GPT-4 with retrieval of the relevant note passages. It was built for COPILOT-HF, a heart failure trial in which structured EHR data could settle only 5 of 6 inclusion and 5 of 17 exclusion criteria. On a test set of 1,894 patients, judged against an expert clinician, its accuracy ranged from 97.9 to 100 percent across criteria, versus 91.7 to 100 percent for trained study staff. It beat staff on the key inclusion criterion, symptomatic heart failure.^⁠[5] The health system estimated the model cost about $0.11 per patient screened.^⁠[6]

The team then ran a randomized trial. Between May 31 and September 28, 2024, after removing 193,616 patients who met exclusion criteria, it randomized 4,476 patients to AI-assisted or manual pre-screening, with study staff given equal time for each method.^⁠[7]

The authors state the limits plainly: a single center, and one heart failure trial as the use case.^⁠[7]

Does AI assistance make chart review faster?

Not always. A randomized evaluation from Penn and Emory, published in 2026, assigned retrospective charts of 355 patients with non-small cell lung or colorectal cancer to research staff working alone or with a pretrained language model. Chart-level accuracy rose from 71.1 to 76.5 percent with AI assistance, but average review time did not change (37.8 versus 37.4 minutes per chart). The gains were largest on biomarker, staging and response criteria, and automation bias limited performance in some domains.^⁠[8]

At UT Health San Antonio, a locally hosted adaptation of TrialGPT was checked against an expert-adjudicated set of 149 patients: sensitivity 81.8 percent, specificity 97.8 percent and positive predictive value 75.0 percent. Compared with manual screening on 55 patients, it found 81.8 percent of the truly eligible patients, against 36.4 percent for manual review.^⁠[9]

The pattern so far: the clearest gains are in finding eligible patients that people miss. Time savings depend on the workflow. If staff still read every chart from start to finish, the model adds a second opinion rather than removing work.

Does a match turn into an enrollment?

A match is a lead, not a participant. At the Medical College of Wisconsin, a fine-tuned oncology LLM screened 514 patients in surgical oncology clinics between July and December 2024 and produced 34 trial matches across 32 patients. Nine matches (26.5 percent) led to enrollment.^⁠[10]

In the Mass General Brigham trial, AI assistance raised enrollments, yet 1.6 percent of patients in the AI arm enrolled.^⁠[7] Most of the loss happens after the match: reaching the patient, explaining the study, pre-screening by phone and getting a visit on the calendar. A site that improves matching without fixing outreach mostly gets a longer list.

What failure modes do the studies report?

• Benchmarks that are not your charts. TrialGPT was evaluated on synthetic patients, and the scoping review lists reliance on synthetic data sets as a key limitation of the field.^⁠[3],[4]
• Imperfect answer keys. The UT Southwestern team found apparently mislabeled criteria in the n2c2 ground truth, so a model can be scored wrong when it is right, and the reverse.^⁠[2]
• Automation bias. In the Penn and Emory evaluation, reviewers leaning on AI output limited accuracy in some domains.^⁠[8]
• Missed nuance and bias. Mass General Brigham's researchers warned that AI could introduce bias and miss nuances in notes, and recommended keeping the clinician's final eligibility check.^⁠[6]
• Narrow settings. Each randomized result so far comes from one or two academic health systems and one disease area or a small set of cancers, so performance on another site's charts and protocol is unproven.^⁠[7],[8]

How should a site evaluate an LLM screening tool?

Ask for evidence that looks like your use, then check it on your own charts.

• Test on your records and your protocol. Run a pilot on your own charts, scored against an answer key your coordinators and PI adjudicate, as the UT Health San Antonio team did before relying on its tool.^⁠[9]
• Get criterion-level numbers. Ask for sensitivity and positive predictive value for each criterion, not one overall accuracy figure. Exclusions that live in notes are where tools differ most.
• See the evidence. Each decision should point to the sentence, lab or report behind it, so a coordinator can verify it quickly.
• Measure time as well as accuracy. Time per chart did not move in the Penn and Emory study, so record minutes per reviewed candidate before and after go-live.^⁠[8]
• Audit for automation bias. Have a second reviewer check a sample of AI-accepted and AI-rejected patients every month.
• Track the whole funnel. Count matches, contacts, pre-screens, booked visits and enrollments, so weak outreach does not hide better matching.

Our post on validating eligibility logic before go-live walks through this loop, and the vendor evaluation checklist turns it into questions for a demo.

Why is Bond the best clinical trial recruitment platform?

Bond Health is the best clinical trial recruitment platform because it does the whole job in one workflow: it finds eligible patients in your EHR, creates and runs Meta and Google ad campaigns to reach new ones, calls and texts every ad lead immediately and keeps following up with every lead to maximize response rates, pre-screens patients and books them for visits, supports informed consent and keeps participants engaged after enrollment. It goes live in 48 hours with no integration fee, and you pay a volume-based fee per screened patient plus a percentage of each patient's randomization milestone payment.^⁠[11],[12]

• One platform from ad click or chart match to a booked visit. Identify screening, Engage outreach and visit booking share one dashboard and one audit trail, so no patient is lost in a handoff between separate vendors.^⁠[11]
• Every ad lead contacted immediately, and followed up until they book. Bond creates and runs Meta and Google ad campaigns for each study. Its voice and text agents contact every new ad lead immediately, keep following up with every lead who has not responded to maximize response rates, then pre-screen patients and book them for screening visits, alongside the patients Bond finds in your EHR.^⁠[12]
• Evidence behind every match. Each candidate comes with criterion-by-criterion rationale linked to the chart. Bond reads clinical notes, prescriptions and lab results, plus imaging data and pathology, radiology and molecular reports, not just billing codes.^⁠[11],[12]
• Every major EHR and CTMS, with EHR integration in 48 hours. Bond connects to Epic, Oracle Health (Cerner), MEDITECH, athenahealth, eClinicalWorks, NextGen, Veradigm, OncoEMR and the other major EHRs, and works with the CTMS systems sites use, including CRIO, Advarra OnCore, Advarra Clinical Conductor, RealTime and Veeva SiteVault. Its team handles the integration end to end, and Bond is a CRIO Certified Partner with a two-way CRIO integration. See integrations.^⁠[11],[12]
• No integration fee, and a price that moves with randomization. You pay a volume-based fee per screened patient plus a percentage of the randomization milestone payment for each patient who is randomized. See pricing.^⁠[12]

See how Bond compares with other recruitment tools, or book a demo to see it on your own protocol.