I have a problem. When I write, I edit as I go. Every sentence gets rewritten three times before I move to the next one. It's a terrible habit and it absolutely kills my productivity. I've known about it for years and I still do it.

Product management involves a lot of writing. Idea briefs, discovery documents, analysis, stakeholder communications. Each one requires pulling together scattered knowledge, structuring an argument, and getting it down on paper. For someone who can't stop fiddling with sentences, this is not ideal.

So when AI tools started promising to transform knowledge work, I was interested. Then I read Arvind Narayanan's AI Snake Oil and became considerably less interested.

The evidence is not great

The prevailing narrative right now is that AI is transforming everything. Replace your team with AI agents. Automate your entire workflow. The future is here.

But what does the evidence actually say?

Narayanan's framework is useful here. He splits AI capabilities into three buckets: perception tasks (image recognition, speech), narrow prediction with clear feedback loops (ad targeting, recommendations), and prediction of complex social outcomes (hiring, recidivism, job performance). AI is genuinely strong in the first bucket, decent in the second, and largely snake oil in the third. The problem is that marketing claims rise steadily across all three, while actual reliability drops off a cliff.

A 2026 BCG field experiment with 758 consultants put this in sharp relief. When consultants used AI tools, correctness fell from 84% to as low as 60%. But here's the thing: subjective quality ratings went up by over 25%. The outputs looked better, read better, and were less accurate. That is, almost by definition, snake oil.

It's not just that study. A large-scale BBC/EBU evaluation of AI assistants answering news questions found that 45% of answers contained at least one significant flaw. A meta-analysis of 445 AI benchmarks found that the tests used to measure AI capability systematically overstate it.

There's also an interesting study from Anthropic (worth noting that the source, Drew Bent, is Anthropic's Education Lead, so weight accordingly). They found that students using AI tools finished coding assignments faster but scored 17% worse on a subsequent assessment without AI. However, students who used AI in an inquiry-based way, probing and questioning rather than just extracting answers, performed just as well.

That distinction matters.

So I should have stopped there

But I noticed a pattern across the research. Transactional AI use, where you delegate the thinking and accept the output, consistently fails. Inquiry-based use, where you come with the problem, iterate, and keep your judgment in the loop, doesn't.

I tested this myself. I tried using AI as a judge in a triage process, getting it to assess whether incoming work would affect a downstream team. It missed the nuance nearly every time. Worse, it wasn't helpful for the team's own understanding of the work. We dropped it quickly. Looking back, this is exactly what Narayanan's framework predicts: asking AI to exercise judgment on ambiguous, context-heavy work is the failure mode.

So I stopped asking AI to think for me. Instead, I started asking it to help me think faster.

The distinction is this: I use AI to retrieve, structure, and iterate on knowledge I already have - not to generate new knowledge. When the "correctness floor" is set by what I already know, the failure mode changes entirely.

Where it actually helps

Here's where AI genuinely speeds up my PM work, broken into four categories.

Research and retrieval

I've built up a knowledge base of my work: learnings, documents, research, decisions. I also use a sparkfile system to capture notes, half-formed ideas, and observations as they happen. AI then helps me groom and analyse these notes, surfacing patterns I might not have connected on my own. (The sparkfile system is a whole topic in itself, one I'll write about separately if there's interest.)

Previously, when I needed to pull together information for a new document, I'd spend twenty minutes per question hunting through my own files. If I had half a dozen questions to answer (you can do the maths), that's a significant chunk of time.

Now I query my own knowledge base. The AI isn't adding information. It's finding what I already stored and synthesising it. The answers come from my material, not the model's training data.

Structuring and planning

Before writing an idea brief, I'll iterate dozens of times on the approach. Not the prose. What structure should the document take? What evidence supports each section? What angles haven't I considered?

This directly addresses my editing problem. Instead of writing and rewriting the same paragraph, I'm iterating on the blueprint. By the time I actually write, the structure is solid. I've separated thinking from editing, which for someone with my particular affliction is genuinely transformative. (I realise the irony of using the word "transformative" in a post about AI snake oil. It stays.)

Data analysis

I pull information from workflow tools and make sense of patterns, disambiguating between what's launched (dcumentation) and what's in progress (my knowledge base). Identifying trends across workstreams. The AI is processing my data, not generating its own claims about the world.

Building workflow tools

I used to write Python scripts for data analysis and process automation. I still design the workflows, define the controls, and own the process logic. But AI has massively accelerated how quickly I can build and iterate on these tools. The result is better accuracy and faster review times for downstream teams, built on processes I've designed and maintain.

This is the same principle at a different layer. I design the thinking - AI accelerates the execution.

The principles

Looking across these use cases, a few patterns emerge:

  • AI retrieves, you judge. Use it to surface and synthesise what you already know. Not to know things for you.
  • Iterate on structure, not prose. Let AI help you think about what to write before you write it.
  • Curate the inputs. AI is only as good as the knowledge base you've built. The PM work of capturing, organising, and storing context is what makes AI retrieval useful in the first place.
  • Keep the correctness floor with you. If the AI gets something wrong, you should be able to catch it because you already know the domain.
  • Design the process, accelerate the execution. Especially for tooling. Own the workflow logic.

Where does this leave us?

I'm not anti-AI, nor pro-AI. I'm pro-thinking.

The snake oil problem is real. Most people using AI for knowledge work are probably making their outputs look better while making them less accurate. The BCG study should be required reading for anyone deploying AI tools in their organisation.

But if you're deliberate about where AI operates, on your knowledge rather than instead of it, it genuinely speeds up work that would otherwise be bottlenecked by retrieval, structuring, and implementation.

The question for you: where is your correctness floor? Where do you already have the knowledge but lack the time to organise it? That's where AI helps. Everything else is probably snake oil.

I mentioned my sparkfile and productivity system above. If you'd like me to write a deeper post on how I capture, organise, and retrieve knowledge as a PM, let me know in the comments.