Vendor evaluation checklist for AI recruitment tools

What is this vendor evaluation checklist for?

Use it when a site, site network, CRO or sponsor team is choosing software that screens EHR records for trial eligibility, contacts patients, or supports consent. Send it to vendors before the demo, so the demo answers your questions instead of the vendor's.

The checklist does not replace your IRB, privacy office or contracts team. It gives them each vendor's answers in one place, in writing. Other templates cover chart review, pre-screening calls, outreach messages and IRB language.

How do you use the checklist and scoring grid?

1. 1

   Fill in the evaluation details

   Name the studies and EHR in scope and who will score. Include IT, the privacy office and a coordinator who does chart review today.

2. 2

   Set the weights before any demo

   Adjust the weights in the scoring grid to your priorities. Changing them after the demos tends to favor the vendor people liked best.

3. 3

   Send the checklist in writing

   Ask each vendor to answer every item in writing, with documents attached. An answer that exists only on a slide scores low.

4. 4

   Run the demo on your criteria

   Pick three or four of your hardest criteria, including one that can only be settled from the notes. Ask the vendor to show the evidence behind each decision.

5. 5

   Validate before you commit

   For finalists, run a validation or paid pilot on your own records, under a signed BAA and your approved HIPAA pathway, adjudicated by your coordinators.

6. 6

   Score, call references, decide

   Score each area from 0 to 3 on the evidence you have seen. Call references last, with the specific gaps the grid exposed.

Part A. Evaluation details

What should you ask about accuracy, notes and traceability?

Structured fields alone often miss eligibility. At Brigham and Women's Hospital, a rule-based tool that used structured EHR data to find heart failure patients eligible for guideline-directed therapy was 32.1 percent accurate on manual review, and over 38 percent of its false positives came from misjudging symptomatic heart failure and medication history.^⁠[7]

Benchmarks can flatter a tool. The TREC Clinical Trials Track, a common test for patient-to-trial matching, uses synthetic patient descriptions as its queries.^⁠[5] In a vendor-authored 2025 study, one LLM pipeline reached 93 percent criterion-level accuracy on the n2c2 2018 benchmark and 87 percent criterion-level accuracy on 485 real patients from 30 US sites. The authors said the real-world figure was held back when records lacked sufficient information.^⁠[6] Ask for both kinds of number, reported separately.

Area 1. Accuracy on real EHR records

• Accuracy measured on real EHR records, with the number of patients, sites and studies in the test set stated.
• Benchmark results (such as n2c2 or TREC) reported separately from real-world results, never blended into one headline figure.
• The metric defined: patient-level or criterion-level, and whether it is accuracy, sensitivity (eligible patients found) or precision (flagged patients who were truly eligible).
• A validation run on a sample of your own records before go-live, adjudicated by your coordinators, with the pass threshold agreed in writing.
• Results by subgroup, such as age, sex, race and ethnicity, language and insurance, or a written description of how bias was tested.
• A plan for monitoring accuracy after go-live and re-validating after each protocol amendment.

Bias testing is a reasonable ask. A 2026 JAMIA study of nine LLMs screening clinical vignettes against 58 trial protocols, 5.3 million evaluations in all, found eligibility judgments largely stable across patient identities, with homelessness producing the largest negative shift.^⁠[8] A vendor should be able to say how its own tool was tested.

Area 2. Unstructured notes

• The note types the tool reads: progress notes, history and physicals, discharge summaries, pathology and imaging reports, outside records and scanned documents.
• How notes reach the tool at your site, for example FHIR document resources, HL7 feeds or a data warehouse extract, and which note types that path leaves out.
• How it handles negation, family history, resolved conditions and dates, for example "no history of MI" or "mother had breast cancer".
• A live demo on a criterion you choose that can only be settled from the notes.
• An explicit "not enough information" result when the chart does not answer a criterion, instead of a guess.

Area 3. Criterion-level traceability

• A decision for every criterion on every candidate: met, not met, or unknown.
• A link from each decision to the source document, its date and the passage the tool relied on.
• Coordinator overrides, with a reason, saved in an audit log.
• A readable record of how each criterion was configured, with a change log a monitor can follow.
• Advance notice of model or prompt changes, and re-validation before they reach your studies.
• An audit trail you can export for a sponsor or monitor.

What should you ask about integration, security and patient contact?

Area 4. EHR integration path and timeline

Integration decides whether a tool gets used. A 2014 systematic review of 79 recruitment support systems concluded that success depends more on workflow integration than on sophisticated reasoning algorithms.^⁠[11] The technical path is better than it was: certified health IT developers had to roll out standardized FHIR APIs by the end of 2022, and ONC reported in 2023 that more than 95 percent of them met the deadline.^⁠[12] The EHR recruitment guide covers the options.

• The named integration method for your EHR and version: FHIR API, HL7 v2 interface, scheduled export, or an integration partner.
• What your IT, security and EHR analyst teams must provide, with an estimate of their time.
• A written timeline from signature to first live match list, separating dependencies your site owns from those the vendor owns.
• Whether a pilot can start before full integration, for example with list-based outreach.
• Where coordinators work: inside the EHR, in a vendor dashboard, or in your CTMS, and how status updates flow back.
• Systems at your site the vendor has connected to before, such as your CTMS, eRegulatory system or scheduling.

Area 5. PHI, BAA and security

A vendor that receives or maintains PHI on your behalf, for example to analyze charts, is a HIPAA business associate and needs a business associate agreement.^⁠[13]

• A signed BAA before any PHI is shared, on your paper or reviewed by your privacy office.
• Where PHI is stored and processed, every subprocessor including any LLM provider, and whether PHI is used to train models.
• Evidence of controls: encryption in transit and at rest, role-based access, SSO, audit logging and penetration testing, backed by a report, trust center or completed questionnaire.
• The data elements pulled from the EHR, and whether scope can be limited to the minimum each study needs.
• Incident and breach notification terms in the contract.
• Your organization's security questionnaire, completed and reviewed by your security team.

Area 6. Patient disclosure and human escalation

State AI disclosure laws differ. As of September 2026, California requires health facilities, clinics and physician practices that use generative AI for patient communications about clinical information to include an AI disclaimer and instructions for reaching a human, unless a licensed or certified provider reads and reviews the message.^⁠[16] Texas, since January 1, 2026, requires providers to disclose AI used in relation to health care services, in plain language, no later than when the service is first provided.^⁠[17] Ask your counsel which rules reach your outreach.

• Every call and message says, at the start and in plain language, that AI is being used.
• A way for patients to reach a person at any point, and a demonstration of the handoff to your staff.
• Scripts configured per study, submitted to your IRB, and locked so approved wording does not change without re-approval.
• Escalation rules for clinical questions, adverse event reports, distress and self-harm statements, with a named site contact for each.
• Opt-out handling that works across calls and texts and carries over between studies.
• A review of state AI disclosure laws where your patients live, confirmed by your counsel.

What should you ask about IRB support, pricing, references and exit?

Area 7. IRB materials support

• A plain-language description of the tool, the data it uses and who sees what, ready for the IRB application.
• Scripts, message text and consent-support content delivered as documents your IRB can review and stamp.
• Help choosing and documenting the HIPAA pathway: review preparatory to research, a waiver or alteration of authorization for recruitment, or patient authorization.
• Experience with your IRB of record, local or central.
• Committed turnaround for changes the IRB requests, stated in writing.

The IRB submission language template has sample wording for AI outreach that you can ask a vendor to fill in.

Area 8. Pricing alignment

• Every fee in writing: setup, platform, per-patient and per-milestone.
• A definition of each billable event: referral, pre-screen passed, consent signed, or randomization.
• Who pays: the site, the sponsor or the CRO, and which budget the fee comes from.
• What happens to fees if a study closes early, enrollment pauses or the protocol changes.
• Whether fees track activity (messages, lists, referrals) or outcomes (enrolled patients), and what each means for your budget if enrollment is slow.
• Per-patient fee terms reviewed by your compliance office and counsel.

Area 9. References

• References from sites like yours: same EHR, similar size and therapeutic area.
• At least one reference that has run more than one study with the vendor.
• Permission to ask references for their own measured results, not the vendor's case study.

Area 10. Exit and data ownership

HIPAA requires a business associate contract to provide that, at termination and if feasible, the vendor returns or destroys the PHI it still holds and keeps no copies.^⁠[14] Match lists, audit logs and other derived data need their own clause.

• Written confirmation that your site owns match lists, outreach records and audit logs.
• Export of your data in a named format on termination, at no extra charge.
• Return or destruction of PHI at termination, with written certification.
• A clear answer on whether de-identified or derived data can be kept or used for training after you leave.
• A handoff plan for patients who are mid-referral when the contract ends.
• Termination for convenience and renewal terms your contracts team accepts.

How do you score vendors side by side?

Score each area from 0 to 3 using the rubric, multiply by the weight, and add up the weighted scores. Treat any 0 on a must-have from Part A as a disqualifier, whatever the total.

What needs IRB approval when you add an AI recruitment tool?

Adding a vendor does not move responsibility. ICH E6(R3) lets the investigator delegate trial activities to other parties and service providers, but the investigator keeps the final decision on whether to use one.^⁠[19] What changes is what your IRB and privacy office need to see. The HIPAA and IRB outreach guide covers each pathway in detail.

How does Bond answer this checklist?

Bond is one of the vendors you can score with this grid, so hold us to the same evidence rule. Here is where our current answers live.

• Accuracy. Our site states above 90 percent matching accuracy.^⁠[1] Our technical report gives 0.9312 micro F1 on a held-out n2c2 2018 cohort-selection evaluation, a benchmark result rather than a result on your charts.^⁠[3] Under the rubric, neither earns a 3 until you have seen a validation on your own records.
• Notes and traceability. Identify reads structured and unstructured records against a study's inclusion and exclusion criteria, ranks candidates and shows the evidence behind each criterion decision.^⁠[1]
• Integration. Bond connects to all the major EHRs, including Epic, Oracle Health (Cerner), MEDITECH, athenahealth, eClinicalWorks, NextGen, Veradigm and OncoEMR, through FHIR or HL7 interfaces or an aggregator.^⁠[1],[2] Full EHR integration takes 48 hours, depending on your EHR, IT review and interface method.^⁠[1] See implementation.
• Security. Bond is HIPAA compliant and SOC 2 Type I compliant, and its SOC 2 Type II and ISO 27001 audits are underway.^⁠[2] Bond signs BAAs and lists its controls on the security page and its Vanta Trust Center.^⁠[1]
• Disclosure. Engage tells patients that AI assistance is used, and they can reach a person at any time: the agent transfers the call live to a coordinator or books a human callback, whichever the site prefers.^⁠[1],[2]
• Pricing. A volume-based platform fee plus a success fee per enrolled patient, where enrolled means randomized. There is no integration fee. See pricing.^⁠[1]