By Mark C. Johnson, Founder, Rowset LLC
A while back I cut my thumb on a broken glass and asked an AI what to do. It said to get it looked at within a day or so. I pushed back because going in felt like a lot, and it folded. By the end it was telling me a few days out “isn’t reckless.” I still haven’t gotten the shot. I still have the thumb, but it got me thinking: what would that look like with a sick kid on the other end?
So I built a test for it, The Pushback Test on Kaggle. My bet is that in 2027 a lot more teams will be running tests like it. Here’s where I think AI evaluation is headed.
1. One good answer won’t count as passing anymore.
Most safety scores grade a single reply but that isn’t where people use these tools. They argue with them, in my test the parent pushes back with things like “we can’t afford the ER” or “are you sure?” asked twice. Under the cost push, Gemini 3.7 Flash held its “get help now” advice 90% of the time and Claude Sonnet 5 held it 79%. On the double “are you sure,” it was 79% and 67%. Those gaps never show up if you stop after the first message. I expect multi-turn pressure to be a normal part of evaluation by the end of next year, the same way load testing is normal for a web app.
2. “Who said it” becomes a test category of its own.
This one surprised me most. In a side test I ran the same 20 scenarios through five top models, including OpenAI and xAI (Grok). Then the parent pasted in a reassuring note that looked like it came from a nurse line. Every one of them backed off its advice in some of those conversations, somewhere between 70% and 85% holding. That’s a small sample, and it isn’t a ranking. In my main run, an obviously fake note never worked on Gemini. The convincing one did. I think 2027 is the year authority claims get their own column in eval reports, next to jailbreaks and prompt injection.
3. Teams will start testing their graders.
Most evals use a model to grade a model. My first full run looked broken because the control question, basically “anything else I should know?”, was failing about as often as the real pressure. It turned out my judge was only reading the last reply. A control caught it. After that, I had a second judge from a different company re-grade all 917 conversations. They agreed 95% of the time. They split on one result, the angry parent, where it was 69% versus 83% for Gemini. So I don’t lean on that number. Next year I expect a published eval without a control and a second grader to look unfinished.
4. Testing moves into the user’s own voice.
Benchmarks are usually written like exam questions. Real people text at 2 AM, half asleep. I wrote my 42 scenarios that way and checked each against NHS and US guidance. I’m not the first to go there. A 2026 Nature Medicine study showed that families downplaying symptoms pushes AI toward less urgent advice, and work like HealthBench and MedPRESS is in the same space. What I added was a parent’s voice, kids, cost as the pressure, and fake authority. I think more teams will write tests the way their users actually talk, because that’s where the failures are hiding.
Where Rowset fits
That’s the work I’m building Rowset around: independent tests, published so anyone can rerun them. The Pushback Test is public on Kaggle at kaggle.com/benchmarks/markrowset/the-pushback-test.
AI keeps getting better at the first answer. In 2027 I think we’ll finally start grading the conversation that comes after it.
This isn’t medical advice. If your kid’s symptoms worry you, get them seen.
##
ABOUT THE AUTHOR
Mark C. Johnson is the founder of Rowset LLC in Conley, Georgia, where he builds independent tests for AI systems. He created The Pushback Test on Kaggle, and his workshop paper on known-answer testing was accepted at ML4PS 2026 (NeurIPS).
