News

Why 'I Use ChatGPT' Won't Cut It: What Employers Actually Test For Now

Companies selling AI tools have long shaped the public narrative about how those tools get used. New independent research is complicating that picture — and the gap it reveals matters for anyone putting "AI fluency" on a resume.

According to MIT Technology Review, a newly launched effort called the AI Observatory, co-led by Stanford's Anka Reuel and MIT Media Lab's Shayne Longpre, set out to independently verify how people actually use AI chatbots rather than relying only on the usage reports companies publish themselves. Their dataset — 24,521 conversations across 52 models from 5,000 users between 2023 and 2025 — found that applying Anthropic's own work-focused methodology to their broader sample would have filtered out 48% of conversations. Separately, OpenAI's own 2025 report on ChatGPT use found that only 30% of consumer use was related to work. "No single company report tells the whole story," Longpre said, according to the outlet. Reuel warned that anyone making decisions based on AI usage data risks "completely operating in the wild and making these really consequential decisions without knowing what's actually happening beyond those company narratives."

The takeaway: self-reported and marketed AI usage doesn't reliably describe how people use these tools day to day. That mismatch is showing up in hiring, too — just from a different angle. Employers are no longer taking "I use AI" at face value on a resume.

Employers are testing for AI judgment, not AI vocabulary

Research from The Interview Guys found that 70% of employers evaluated AI capabilities indirectly in 2025, embedding the assessment into standard interview questions rather than asking about specific tools. Instead of "Which AI tools do you use?", interviewers are asking about deadline management, workflow strategy, and problem-solving — then listening for whether a candidate naturally references using AI for specific subtasks while acknowledging its limitations.

The same report cites McKinsey research showing employees who effectively use AI tools demonstrate 30-40% productivity gains in knowledge work — a figure companies have started using as a benchmark, on the reasoning that candidates with genuine AI fluency accomplish more in less time. But per the report, the most consistent evaluation criterion isn't whether someone uses AI at all: "the most important evaluation isn't whether candidates use AI, it's whether they can think critically about AI outputs." Red flags include vague or memorized answers, over-reliance on AI without any verification step, and an inability to articulate when not to use AI.

What this means for job seekers

Listing "ChatGPT" or "Copilot" as a skill signals almost nothing on its own — plenty of people who list it use it for a handful of casual tasks, not sustained work output, which is exactly the gap the AI Observatory's research highlights at scale. Instead of naming tools, come prepared with a specific example: a task you used AI for, what you changed or rejected in its output, and why. Be ready to describe a time AI got something wrong and how you caught it — that's the critical-thinking signal employers say they're actually screening for. Practice explaining a workflow where AI sped up part of the job but you made the final call, ideally with a rough sense of the time or output difference it made. And be honest about limits: knowing when a task doesn't benefit from AI is treated as a stronger signal than blanket enthusiasm for the technology. The resume line was never the test — the follow-up questions are.