Building and evaluating AI for clinical evidence
Healthcare AI is moving faster than our ability to check it. I build AI systems that work with clinical evidence, and the evaluations that show whether they can be trusted: deterministic grounding checks, physician-rubric analysis and peer-reviewed validation.
healthcare AI evaluation·clinical evidence·verification·privacy·agents
ORCID
Google Scholar
GitHub01Healthcare AI evaluation
How do you know a healthcare AI is right?
Twenty years in medical communications taught me that every claim must trace to a source. I hold AI to the same standard. Each system here ships with its own evaluation, and I publish a figure only after the run that produced it.
Patiently AI
Can generative AI simplify medical information without losing meaning?
A deployed, MHRA-registered Class I medical device that turns clinical letters, results and discharge summaries into plain language — in 18 languages, on web, iOS and Android. Validated in a three-phase, peer-reviewed study (Frontiers in Digital Health, 2026) — computational readability analysis, expert review by 15 healthcare professionals who rated outputs 4.49/5 for accuracy, and a 54-patient survey. Five national awards, in real-world use.
PubCrawl
Can an AI assistant cite the real literature instead of inventing it?
Real literature for AI assistants: PubMed, Europe PMC, clinical trials and US/UK drug labels over MCP, with no hallucinated citations. Its own US/UK label comparison is something no other MCP server offers, and an in-repo fidelity benchmark checks quoted label text verbatim against the source, gating every change in CI.
An open-source verification framework for evidence-grounded AI. Deterministic grounding and extraction checks — no LLM judge anywhere in the loop — so verdicts are reproducible, auditable, and cheap enough to run on every answer or every commit. Four production systems run on it in CI, where it has caught silent parse failures, a citation-parser inconsistency, and a simplifier dropping an antibiotic dose from a discharge summary.
Redacta
How can agents work with sensitive information without exposing it unnecessarily?
A privacy boundary for AI and agent workflows. Redacta pseudonymises patient identifiers — names, NHS numbers, dates — into labelled tokens, and its MCP server keeps the map that reverses them outside the model’s context entirely. It then checks the model’s output for identifiers that found their way back in, so the boundary is verified rather than assumed. Benchmarked under adversarial attack by its own published evaluation harness, and deployable from a free iPhone app to a self-hosted Kubernetes cluster.
Latest
How much of a clinical rubric can code check?
HealthBench’s physician baseline reproduced to within 0.02, then 300 criteria classified and tested: about a third can be checked without a judge, but almost none of the context-seeking criteria.
→Now building · GroundhogDoes a medical-evidence agent give the same answer twice?
An open benchmark for agent reliability on medical-evidence tasks: repeat runs, injected tool faults and physician-style rubrics, using PubCrawl and Redacta as the tools.
02Research · Observer Zero
The same question, asked of autonomous agents: do they believe what their evidence supports?
What happens when AI agents become scientists — inside a world whose ground truth we control?
Observer Zero is a persistent artificial society for studying how autonomous LLM agents gather evidence, interpret it, revise their beliefs and reach collective conclusions. Its agents inhabit Meridian — a world with fictional physics — where they run experiments, exchange letters and hold explicit, probabilistic beliefs about laws they cannot see. Because the simulator holds perfect ground truth, every belief can be scored against reality: the design separates what a society had the evidence to know from what it actually came to believe.
When the laws of their world secretly changed, the agents gathered evidence enough for a blind statistical detector to find the change — and almost none of them ever believed it.
When the Laws Changed
Can autonomous agents recognise that a physical law of their world has changed?
Across 150 universes, 90–100% of agents transiently noticed something was wrong — yet 0 of 40 gravity-shift worlds reached a strict law-change conclusion. The anomalies were real; the interpretation reached for faulty instruments instead.
Does Society Help?
Can an artificial scientific society overcome the individual failure through social interaction?
Pre-registered, 85 universes, societies of two and eight. Left alone, grounded societies never spoke; seeding one communicative agent spread unsupported claims, not scrutiny. The combined two-study paper is in peer review, with the model published in the CoMSES library.
The Eureka Threshold
What would it take for an AI scientist to discover that its world is simulated?
Pre-registered and frozen before the first run: 170 universes across four model families, where one instrument's readings genuinely came from outside the world. No agent ever formed that hypothesis — 0 of 38 treatment runs — while agents licensed to consider it endorsed it before the manipulation had produced any evidence. Belief tracked prompting, not evidence. The paper is in peer review.
03The wider lab
Experiments, tools and detours.
The same method at smaller scale: build it, measure it, publish the measurement. A progression runs through the lab — from applications, to infrastructure, to evaluation, to studying the behaviour of the systems themselves.
RefCheckr
Automated claim and reference verification for medical and scientific content — with its model choices benchmarked in the open.
→InfrastructureGroundwork
Deployment-readiness harnesses for document QA and structured extraction — a CI gate that catches fabrication before users see it.
→Evaluation · RAGLitRAG
Evidence-grounded literature RAG that caught every unfaithful claim in its benchmark, including a true claim backed by a fabricated quote.
→Methodology · MCP serverAI Water & Energy Calculator
Why published estimates of AI water use differ 100× — and what yours comes to. The same methodology runs as an open-source MCP server with no model in any tool.
→Fine-tuning · validationTime-Machine Chess
Era-specific engines fine-tuned on 150 years of historical games — and validated against the history they learned from.
→LabsAll experiments
HushMap, PosterLens, StudyDiff, BiomarkerFinder, TrialGen, PlaceboGPT and other prototypes and detours live in the Labs.
04About
Who’s behind this.
I build AI for clinical evidence, then measure whether it deserves to be trusted.
Nick Lamb is an independent AI researcher and engineer with an unusual route in: twenty years of regulated medical communications (PhD, CMPP), a field where every claim must trace to evidence. That standard now runs through everything on this page: verification that never takes a model’s word for it, a registered medical device validated before it reached patients, and research that scores agents’ beliefs against hidden ground truth.
Healthcare is the focus, because few domains punish an ungrounded claim faster. PharmaTools.AI began as a collection of healthcare AI tools; the work now spans evaluation, verification and privacy infrastructure, and empirical research on autonomous agents, much of it open source.
Hiring for healthcare AI or evaluation?
I’m looking for a full-time role building and evaluating AI systems, in a team that takes evidence seriously. London, Oxford or UK-remote. Working on related research? I’d be glad to hear from you too.
