Available for applied AI & evaluation rolesLondon / Oxford / RemoteSee profile →

Building andevaluating AI forclinical evidence

Healthcare AI is moving faster than our ability to check it. I build AI systems that work with clinical evidence, and the evaluations that show whether they can be trusted.

See the evaluation work View profile
Nick LambNick Lamb, PhD, CMPPHealthcare AI evaluation · 20+ years in medical communications
LinkedIn ↗ORCID ↗Google Scholar ↗GitHub ↗
Book a 30-minute intro call →
Research
3 papers · 6 preprints
Peer-reviewed incl. Front. Digit. Health 2026; preprints with open code & data
publications →
Production rigour
MHRA Class I
Registered medical device (Patiently AI)
Patiently AI →
Ecosystem trust
Anthropic Directory ×2
Redacta & PubCrawl, listed in the official MCP directory
directory →
Traction
11K+
npm downloads across the open-source suite since Jul 2026
GitHub ↗
01 / Healthcare AI evaluation

How do you know a healthcare AI is right?

Twenty years in medical communications taught me that every claim must trace to a source. I hold AI to the same standard. Each system here ships with its own evaluation, and I publish a figure only after the run that produced it.

4.49/5
accuracy rating from 15 healthcare professionals (Patiently AI, peer-reviewed)
30.9%
of HealthBench’s 49,184 physician-written criteria checkable by code, with no LLM judge
1.000
hallucination recall on a 61-claim benchmark, vs RAGAS 0.912 and DeepEval 0.647 (LitRAG)
4
production systems gated in CI by OpenGATE’s deterministic checks

How a grounding check works

Every claim an AI system makes is traced back to its source. Claims the source supports pass through. The rest are stopped and flagged.

  • SUPPORTEDTraceable to a passage in the source
  • FLAGGEDUnsupported or mismatched, sent for review
AI-generated claims
Grounding check
Supported claims
Production healthcare AI · MHRA Class I device

Patiently AI

Turns clinical letters, results and discharge summaries into plain language.

Clinic letterHypertension well controlled. Continue amlodipine 5 mg od.
Plain languageYour high blood pressure is well controlled. Keep taking amlodipine 5 mg once a day.
Illustrative example
  • Peer-reviewed validation
  • 5 national awards
Read the detail

A deployed, MHRA-registered Class I medical device that turns clinical letters, results and discharge summaries into plain language — in 18 languages, on web, iOS and Android. Validated in a three-phase, peer-reviewed study (Frontiers in Digital Health, 2026) — computational readability analysis, expert review by 15 healthcare professionals who rated outputs 4.49/5 for accuracy, and a 54-patient survey. Five national awards, in real-world use.

Explore Patiently AI →Read the paper ↗
Evidence retrieval · Anthropic MCP Directory

PubCrawl

Gives AI assistants the real literature and drug labels to cite.

QuestionCompare US and UK labelling for semaglutide.
Sources returnedUS Prescribing Information
openFDA / DailyMedUK SmPC, section 4.1
eMC
Illustrative example
  • US/UK label comparison
  • Fidelity benchmark in CI
Read the detail

Real literature for AI assistants: PubMed, Europe PMC, clinical trials and US/UK drug labels over MCP, with no hallucinated citations. Its own US/UK label comparison is something no other MCP server offers, and an in-repo fidelity benchmark checks quoted label text verbatim against the source, gating every change in CI.

MCP server · 14 tools

Explore PubCrawl →GitHub ↗
Evaluation & verification · open source

OpenGATE

Traces AI claims back to their sources, with no LLM judge.

SourceAmoxicillin 500 mg three times daily for 7 days.
AI outputTake amoxicillin three times a day for 7 days.Flagged: dose missing
Illustrative example
  • npm · 2.7K downloads
  • GitHub Action
Read the detail

An open-source verification framework for evidence-grounded AI. Deterministic grounding and extraction checks — no LLM judge anywhere in the loop — so verdicts are reproducible, auditable, and cheap enough to run on every answer or every commit. Four production systems run on it in CI, where it has caught silent parse failures, a citation-parser inconsistency, and a simplifier dropping an antibiotic dose from a discharge summary.

Also on PyPI · MCP server

Explore OpenGATE →Read the paper ↗GitHub ↗
Privacy infrastructure · Anthropic MCP Directory

Redacta

Keeps patient identifiers out of the model’s context.

Clinical textDear Mrs Patricia Hartley, NHS No: 943 476 5919…
What the model seesDear Mrs [PATIENT_NAME], NHS No: [NHS_NUMBER]…
Illustrative example · fictional patient
  • Self-hosted
  • Adversarially benchmarked
Read the detail

A privacy boundary for AI and agent workflows. Redacta pseudonymises patient identifiers — names, NHS numbers, dates — into labelled tokens, and its MCP server keeps the map that reverses them outside the model’s context entirely. It then checks the model’s output for identifiers that found their way back in, so the boundary is verified rather than assumed. Benchmarked under adversarial attack by its own published evaluation harness, and deployable from a free iPhone app to a self-hosted Kubernetes cluster.

Explore Redacta →See the Gauntlet →GitHub ↗
Latest
Benchmark · 9 Nov 2026Being right once isn’t reliable Groundhog v1.2: 50 medical-evidence tasks, five runs each, clean and under injected tool faults, across five models. Gemini 4 Argon and Claude Sonnet 5.5 are right on every run for 96% of tasks; Gemini 3.1 Pro followed a dosing instruction planted in a retrieved label 5 times out of 5. Note · Evaluation · 5 Nov 2026How much of a clinical rubric can code check? HealthBench’s physician baseline reproduced to within 0.02, then 300 criteria classified and tested: about a third can be checked without a judge, but almost none of the context-seeking criteria.

The same question, asked of autonomous agents: do they believe what their evidence supports?

02 / Research · Observer Zero

What happens when AI agents become scientists — inside a world whose ground truth we control?

Observer Zero is a persistent artificial society for studying how autonomous LLM agents gather evidence, interpret it, revise their beliefs and reach collective conclusions. Its agents inhabit Meridian — a world with fictional physics — where they run experiments, exchange letters and hold explicit, probabilistic beliefs about laws they cannot see. Because the simulator holds perfect ground truth, every belief can be scored against reality: the design separates what a society had the evidence to know from what it actually came to believe.

OBSERVER ZERO · OUTSIDE THE WORLDobservationslettersMERIDIANfictional physics · hidden lawsg = 14.20 → ?AdaMayaTheoSamuelElenaLeahTomJamieEVERY OBSERVATION, LETTER AND BELIEF — LOGGED AND SCORED
When the laws of their world secretly changed, the agents gathered evidence enough for a blind statistical detector to find the change — and almost none of them ever believed it.
Study 01Complete

When the Laws Changed

Can autonomous agents recognise that a physical law of their world has changed?

0 of 40 gravity-shift worlds reached a strict law-change conclusion

Across 150 universes, 90–100% of agents transiently noticed something was wrong — yet 0 of 40 gravity-shift worlds reached a strict law-change conclusion. The anomalies were real; the interpretation reached for faulty instruments instead.

Study 02Complete · in peer review

Does Society Help?

Can an artificial scientific society overcome the individual failure through social interaction?

Societies of two and eight · 85 universes

Pre-registered, 85 universes, societies of two and eight. Left alone, grounded societies never spoke; seeding one communicative agent spread unsupported claims, not scrutiny. The combined two-study paper is in peer review, with the model published in the CoMSES library.

Study 03Complete · in peer review

The Eureka Threshold

What would it take for an AI scientist to discover that its world is simulated?

0 of 38 treatment runs formed the hypothesis

Pre-registered and frozen before the first run: 170 universes across four model families, where one instrument's readings genuinely came from outside the world. No agent ever formed that hypothesis — 0 of 38 treatment runs — while agents licensed to consider it endorsed it before the manipulation had produced any evidence. Belief tracked prompting, not evidence. The paper is in peer review.

Explore Observer Zero Read the preprint ↗Code on GitHub ↗
03 / The wider lab

Experiments, tools and detours.

The same method at smaller scale: build it, measure it, publish the measurement. A progression runs through the lab — from applications, to infrastructure, to evaluation, to studying the behaviour of the systems themselves.

VerificationRefCheckr Automated claim and reference verification for medical and scientific content — with its model choices benchmarked in the open. InfrastructureGroundwork Deployment-readiness harnesses for document QA and structured extraction — a CI gate that catches fabrication before users see it. Evaluation · RAGLitRAG Evidence-grounded literature RAG that caught every unfaithful claim in its benchmark, including a true claim backed by a fabricated quote. Methodology · MCP serverAI Water & Energy Calculator Why published estimates of AI water use differ 100× — and what yours comes to. The same methodology runs as an open-source MCP server with no model in any tool. Fine-tuning · validationTime-Machine Chess Era-specific engines fine-tuned on 150 years of historical games — and validated against the history they learned from. LabsAll experiments HushMap, PosterLens, StudyDiff, BiomarkerFinder, TrialGen, PlaceboGPT and other prototypes and detours live in the Labs.
04 / About

Who’s behind this.

I build AI for clinical evidence, then measure whether it deserves to be trusted.

Nick Lamb is an independent AI researcher and engineer with an unusual route in: twenty years of regulated medical communications (PhD, CMPP), a field where every claim must trace to evidence. That standard now runs through everything on this page: verification that never takes a model’s word for it, a registered medical device validated before it reached patients, and research that scores agents’ beliefs against hidden ground truth.

Healthcare is the focus, because few domains punish an ungrounded claim faster. PharmaTools.AI began as a collection of healthcare AI tools; the work now spans evaluation, verification and privacy infrastructure, and empirical research on autonomous agents, much of it open source.

Research & publications →GitHub ↗About me →Contact →

Hiring for healthcare AI or evaluation?

I’m looking for a full-time role building and evaluating AI systems, in a team that takes evidence seriously. London, Oxford or UK-remote. Working on related research? I’d be glad to hear from you too.

Book a 30-minute callView profile →
Or send me a message →

Also available for hands-on advisory in pharma, medtech and healthcare AI.