Self-assessment of 8 AI PM skills: from knowing to doing, with evals, agents, RAG and costs

The article, published on Medium by Shailesh Sharma, proposes a 10-minute self-assessment for product managers aiming for AI PM roles. His central thesis is that most candidates lack practice rather than knowledge: they have read about AI but have done little with it. According to the author, a certificate doesn't reveal that gap, but an interview exposes it in about two minutes.

The piece opens with a fictional example: a PM with five years of experience at a Bengaluru lending app who has completed three AI courses and can explain RAG and agents. In the interview, she is asked how she knew an AI feature was good enough to ship. She talks about A/B tests, but the interviewer was asking about evals, which she has never written.

How to score yourself. Each skill is rated from 1 to 5: 1, I've never done it; 2, I know the idea but haven't applied it; 3, I've done it once with help; 4, I do it on my own at work; 5, I could teach it. The rule is to score what you have done, not what you have read. A 4 requires work you can show; if you only watched a video, it's a 2.

Three skills your current role already trains. The author warns that because they feel familiar, PMs tend to overrate them. The first is choosing problems where AI fits. The test is being able to name an AI idea you dropped because of data, cost or the risk of error; what matters is knowing when not to use AI. The second is AI metrics and experiments: for a specific feature, naming a quality metric, an adoption metric, a business metric and a guardrail, and knowing when an offline evaluation is enough and when an A/B test is needed. The third is shipping and communicating AI work: explaining quality, risk and cost to an executive in five minutes and having a plan for when the model fails in production, with a fallback and someone who can roll it back. In the example, the PM scores 3.3, 3.7 and 4.3 respectively.

Three skills that AI work adds. This, according to the author, is where the practice gap lies. The first is defining quality and writing evals: building a set with common cases, edge cases and "never" cases (responses that must never be given) and making a ship/no-ship decision based on the results. He argues that even PMs who know AI tend to score lower here, because evals are written, not read. The second is RAG and retrieval: faced with a wrong answer from a RAG bot, being able to tell whether retrieval brought back the wrong chunk or the model misread the right one, because that determines what the team fixes. The third is agents and tools: having built or specified an agent that uses at least one tool and deciding where it needs a human check before acting. In the example: 1.7, 2.3 and 2.0.

Two skills that decide whether the feature survives the budget review. The author says many AI features work in the demo and die in a meeting with finance. The first is cost and latency: calculating cost per query from tokens and the price per million tokens, and naming three ways to reduce it along with the quality risk of each. He illustrates with a lending assistant handling 2 lakh (200,000) queries a day: a difference of 50 paise per query adds up to 30 lakh rupees a month. The second is AI strategy and pricing: knowing where the moat lies beyond the model (data, workflow or distribution) and how per-user cost shapes pricing. In the example, both score 2.7.

Why the average is misleading. Adding everything up, the example PM scores 2.8 out of 5, which looks acceptable. But her three best scores correspond to what her previous role trained (communicating, metrics, choosing problems), and her lowest to evals, agents and RAG—that is, the building skills. This, the author says, is the profile that tends to struggle in interviews: strong at talking about AI, weak at doing it. He recommends ignoring the average, ranking the eight skills from lowest to highest and focusing on the bottom two.

What to do this week. Don't try to fix all eight—fix two and re-score after four to six weeks. If the gap is evals: pick an AI feature (your own or one you use daily, such as search in a quick-commerce app), write 20 test cases (common, edge and "never"), define what a good answer must include, run them, mark pass or fail and decide whether you would ship. According to the author, that takes you from a 1 to a 3 and gives you a real answer to the interview question. If the gap is agents: use Claude to build a single-tool agent (for example, one that reads a CSV of support tickets and drafts a weekly summary), note every step it took and decide what it must never do without asking. That last part, he says, is what interviewers care about: anyone can get an agent working, but the PM decides where it must stop. If your two lowest scores don't go up, he concludes, you went back to reading instead of building.

Editor's note on the nature of the text. The article has a clear promotional component: it offers a free scoring sheet with 24 statements and promotes the author's paid "AI PM Builder Cohort," along with his other courses. The example scores are illustrative, and the article provides no empirical data to back its claims about interviews. Even so, the framework is practical: it offers concrete, verifiable criteria for distinguishing knowledge from experience, and its emphasis on evals, RAG, agents and unit economics matches what AI product roles typically demand today.