Clinical Operations guide
AI for clinical operations: patient matching, site selection and risk-based monitoring
How AI and LLMs help clinical operations: patient-trial matching, eligibility criteria, fair site selection, enrollment forecasts and risk-based monitoring.
AI helps clinical operations in five places: matching patients to trials, refining eligibility criteria, choosing sites, forecasting enrollment and dropout, and risk-based monitoring. The strongest published evidence is for LLM patient matching: TrialGPT reached 87.3% criterion-level accuracy, close to experts, and cut screening time by 42.6% in a pilot. To use an LLM for matching, split the protocol into single criteria, ask the model for a verdict and cited reason per criterion, and keep a clinician's review before any patient is contacted.
Where AI helps in clinical operations
Most trial delays come from a few decisions: who is eligible, which sites to open, how fast they enroll, and where data quality is slipping. AI helps with each, but the evidence base differs by task.
| Task | What AI does | Strength of published evidence |
|---|---|---|
| Patient-to-trial matching | Reads patient notes and trial criteria and judges eligibility per criterion | Strong: peer-reviewed LLM studies with expert comparison |
| Eligibility criteria | Tests how each criterion changes the eligible pool and outcomes | Moderate: oncology studies on real-world data |
| Site selection | Ranks sites by expected enrollment, and optionally by diversity | Moderate: retrospective studies on historical trials |
| Enrollment and dropout forecasting | Predicts enrollment per site and attrition by arm | Early: few peer-reviewed studies |
| Risk-based monitoring | Flags outliers and key risk indicators across sites | Driven by regulation (ICH E6(R3)); method papers vary |
The sections below cover each task, then what to ask vendors.
How to use LLMs to match patients to clinical trials
Large language models can read free-text notes, which is where most eligibility facts live. The best-studied design splits the problem into three steps.
- Retrieve candidate trials or patients. Filter a large pool down to a short list before any detailed reading.
- Judge each criterion separately. Ask for a verdict (met, not met, not enough information) and the sentence in the record that supports it.
- Aggregate and rank. Combine criterion verdicts into a trial-level score and send the ranked list to a coordinator for review.
TrialGPT (Jin, Wang et al., Nature Communications, 2024) follows this design. On 183 synthetic patients with over 75,000 trial annotations, its retrieval step recalled over 90% of relevant trials while considering less than 6% of the collection. Its criterion-level accuracy was 87.3% on 1,015 patient-criterion pairs; expert accuracy ranged from 88.7% to 90.0%. In a pilot user study it cut screening time by 42.6% (full text).
Other work points the same way. RECTIFIER (Unlu et al.) tested GPT-4 with retrieval-augmented generation on clinical notes to screen patients for a heart failure trial. Sensitivity and specificity for overall eligibility were 92.3% and 93.9%, against 90.1% and 83.6% for trained study staff (medRxiv preprint, 2024). A JAMIA 2024 study fine-tuned open-source LLaMA models on synthetic data and reported performance parity with GPT-3.5 for eligibility judgments, which matters when notes cannot be sent to an external API. Keiji AI researchers have also trained a foundation model on clinical trial data for search, design and recruitment tasks (Panacea, medRxiv preprint, 2024).
Know the limits. The TrialGPT authors did not test on full EHR data, lab values or imaging, and state that matching should not be fully automatic. The RECTIFIER authors recommend final clinician review before any patient is contacted. A practical setup:
- Run the model where the records already sit, or use an open model you can host.
- Require a cited sentence for every criterion verdict, and treat "not enough information" as a valid answer.
- Measure accuracy on a sample of your own charts against a coordinator's review before using it at scale.
- Log every verdict and reviewer decision, so the screening log can be audited.
Refining eligibility criteria with data
Restrictive criteria shrink the eligible pool and can make trial populations less like the patients who will use the drug. Data can show which criteria matter.
Trial Pathfinder (Liu et al., Nature, 2021) emulated completed advanced non-small cell lung cancer trials on EHR data from 61,094 patients. Many common criteria, including several lab-value exclusions, had little effect on the trial hazard ratios. Broadening restrictive criteria more than doubled the eligible pool on average.
Two complementary approaches are now common:
- Compare with registered trials. Find comparable trials in registries such as ClinicalTrials.gov, compare their criteria, and justify each departure.
- Test against patient data. Run draft criteria on real-world or institutional data and look at the attrition at each criterion.
Language models can also draft criteria from comparable trials. AutoTrial (Wang, Xiao and Sun, EMNLP 2023), from Keiji AI researchers, prompts a language model to generate eligibility criteria for a new trial design.
Either way, the output should be an editable draft with a reason for each criterion, not a final protocol. Medical and regulatory review stays with the study team.
Site selection, including fairness and diversity
Site selection usually ranks sites by past enrollment. Machine learning can add real-world patient data. A PLOS ONE 2024 study (Hulstaert et al.) combined indication-level recruitment history with real-world data and ranked sites better than common industry baselines in inflammatory bowel disease and multiple myeloma.
Enrollment speed is not the only goal. FDA's draft guidance on diversity action plans (June 2024) describes enrollment goals by race, ethnicity, sex and age group, with a rationale. Site choice is one of the main levers for meeting them.
FRAMM (Theodorou et al., Patterns, 2024) treats this as a ranking problem with two objectives: enrollment and diversity. It also handles sites with missing data. Evaluated on 4,392 trials matched to 33,323 sites, it reached higher diversity at the same enrollment level than the baseline methods it was compared with. The authors present it as a proof of concept.
What to expect from a tool:
- A ranked list with the expected enrollment and the expected patient mix for each site.
- A visible trade-off setting between speed and diversity, recorded with the result.
- Clear handling of sites with incomplete history, rather than silently dropping them.
Forecasting enrollment and dropout
Leadership wants to know when enrollment will finish and which sites or arms are at risk. Forecasts combine site activation dates, enrollment rate per site and observed dropout.
The peer-reviewed evidence here is thinner than for matching. Site-level models such as the PLOS ONE 2024 study predict recruitment per site; much other work is still in preprints. Treat any forecast as a range, and check it against actual enrollment every month.
- Ask for the rate per site and the main bottleneck, not just a single finish date.
- Ask how the model updates as real enrollment comes in.
- Split dropout by treatment arm, since an imbalance can bias the analysis.
Risk-based monitoring under ICH E6(R3)
Good clinical practice now expects monitoring to follow risk. ICH E6(R3) principles and Annex 1 came into effect in the EU on 23 July 2025; the guideline "encourages a risk-based and proportionate approach" (EMA). FDA issued E6(R3) as final guidance in September 2025, citing quality by design and risk-based quality management (FDA). FDA's 2023 questions-and-answers guidance on risk-based monitoring covers planning a monitoring approach, writing the monitoring plan, and acting on results.
AI fits in the central-monitoring layer:
- Key risk indicators per site, with thresholds agreed in advance.
- Outlier and data-quality checks on adverse events, labs and visit data as data arrive.
- Targeted visit agendas and query recommendations for monitors.
The monitor makes the decision. Signals should come with the analysis behind them, and disagreement between methods should be shown, not averaged away.
What TrialMind does for clinical operations
TrialMind is Keiji AI's platform of AI agents for clinical research. For clinical development and operations it covers five workflows: site selection, patient matching, eligibility criteria, enrollment and dropout forecasting, and data and medical monitoring.
- Site selection: a ranked site list that balances expected enrollment and patient diversity, with the trade-off recorded. On NCT03100149, a Phase 2 Parkinson's study, a fairness-aware run ranked 26,287 candidate sites against a 1,000-patient target and shortlisted five under a five-site cap.
- Patient matching: eligibility criteria converted into rules and run on the customer's patient data, with counts at each step. Screening against the TROPION-Lung17 protocol narrowed 1,001 patients to 164 candidates (16.4%); prior treatment was the largest single exclusion.
- Eligibility criteria: comparable trials grouped by topic, then an editable draft of inclusion and exclusion criteria. At a global biopharmaceutical company, drafting criteria for a new protocol went from months to days.
- Enrollment and dropout: enrollment rate per site and dropout by arm, with the main bottleneck identified.
- Monitoring: risk-based monitoring and key risk indicators per ICH E6(R2), with site risk scores, targeted visit agendas and query recommendations. This workflow is newer and is co-designed with medical monitors.
TrialMind deploys into the customer's environment, and raw patient records are not moved. Related research includes FRAMM for fair site selection and TrialGPT, a National Institutes of Health project on patient matching that Keiji AI's founders co-authored.
Buyer checklist
- Whose data? Does the tool bring a network of patients and sites, or run on data you hold? Both are valid; they answer different questions.
- Where does data go? Ask whether patient-level data leaves your environment, and get it in the contract.
- Per-criterion evidence. For matching, each verdict should cite the record it relies on.
- Your own accuracy check. Ask to measure accuracy on a sample of your charts before rollout.
- Diversity trade-off. For site selection, ask how diversity is weighted and whether the setting is recorded.
- Published methods. Ask for peer-reviewed papers, not only case-study percentages.
- Audit trail. Every screening decision, reviewer override and monitoring signal should be logged.
- Regulatory fit. Ask how monitoring outputs map to your risk-based quality management plan under ICH E6(R3).
Frequently asked questions
- Can large language models reliably match patients to trial eligibility criteria?
- In peer-reviewed tests, close to experts but not equal: TrialGPT reached 87.3% criterion-level accuracy against 88.7% to 90.0% for experts. Use LLMs to prioritize and explain, with a clinician confirming eligibility.
- What is the best way to screen patients against a protocol with an LLM?
- Split the protocol into single criteria, ask for a verdict and a cited sentence for each, and aggregate the results into a ranked list. Validate on your own charts and keep human review before contact.
- How does AI support fair site selection?
- It ranks sites on expected enrollment and expected patient diversity together, and makes the trade-off explicit. FRAMM (Patterns, 2024) is one published method.
- Which guidance covers risk-based monitoring?
- ICH E6(R3), in effect in the EU since July 2025 and issued by FDA as final guidance in September 2025, plus FDA's 2023 questions-and-answers guidance on risk-based monitoring.
- Does patient data need to leave the hospital or sponsor for AI matching?
- Not necessarily. Federated networks and tools deployed in the customer's environment, such as TrialMind, run where the data already lives.
See TrialMind on your own question
A 30-minute walkthrough on a study like yours, with sources and code you can check.
Sources · checked 2026-10-21
Product details come from each company’s public pages. Spotted something out of date? Tell us and we will correct it.
- Jin, Wang et al. Matching patients to clinical trials with large language models. Nature Communications, 2024
- TrialGPT full text (PMC)
- Unlu et al. RAG-enabled GPT-4 for clinical trial screening (RECTIFIER). medRxiv preprint, 2024
- Nievas et al. Distilling large language models for matching patients to clinical trials. JAMIA, 2024
- Lin et al. Panacea: A foundation model for clinical trial search, summarization, design, and recruitment. medRxiv preprint, 2024
- Wang, Xiao, Sun. AutoTrial: Prompting Language Models for Clinical Trial Design. EMNLP, 2023
- Liu et al. Evaluating eligibility criteria of oncology trials using real-world data and AI. Nature, 2021
- Hulstaert et al. Enhancing site selection strategies in clinical trial recruitment using real-world data modeling. PLOS ONE, 2024
- Theodorou et al. FRAMM: Fair ranking with missing modalities for clinical trial site selection. Patterns, 2024
- FDA. Diversity Action Plans to Improve Enrollment of Participants from Underrepresented Populations (draft guidance, June 2024)
- EMA. ICH E6 Good clinical practice scientific guideline
- FDA. E6(R3) Good Clinical Practice (final guidance, September 2025)
- FDA. A Risk-Based Approach to Monitoring of Clinical Investigations: Questions and Answers (2023)