Product Craft
Writing PRDs in the age of LLMs
Listen — audio summary in Nehal Vyas's voice
Transcript
Hey everyone! My latest blog post is about writing Product Requirement Documents, or PRDs, for LLM-powered features. Because AI behavior is probabilistic, your PRD needs to define the outcome and how you'll measure it. It's essentially a spec and an evaluation harness in one. My key takeaways: First, lead with outcomes, not prompts. Prompts are implementation details; anchor your document on user accomplishment and quality. Second, enumerate edge cases as concrete test cases. Ambiguous or adversarial inputs reveal AI's weaknesses. List these with expected handling; they become your core eval set. Third, define your evaluation criteria before you build. If you can't measure success, like with task success rate or human-rated quality, you’re not ready. This keeps the team honest. Ultimately, a PRD with its own eval harness makes AI's fuzziness steerable. The spec proves the product. Read the full article at hinehal.com.
For deterministic software, a PRD describes behavior you can guarantee. For AI features, behavior is probabilistic, so the same document has to do double duty: it defines the outcome and it defines how you'll know the outcome was met. The best PRDs I write for LLM features read partly like specs and partly like evaluation harnesses.
Lead with outcomes, not prompts
It's tempting to over-specify the prompt. Resist it. The prompt is an implementation detail that will change ten times before launch. What shouldn't change is the outcome: what should the user be able to accomplish, at what quality bar, and what does unacceptable look like? Anchor the document there.
Enumerate edge cases as test cases
Ambiguous inputs, adversarial inputs, empty states, and out-of-scope requests are where AI features embarrass themselves. I list these not as prose but as concrete examples with expected handling. That list becomes the eval set engineering runs against every model or prompt change.
Define eval criteria before you build
If you can't describe how you'd measure success, you're not ready to build. Whether it's task success rate, human-rated quality, or deflection, naming the metric up front keeps the team honest and turns "it feels better" into something you can defend to stakeholders.
A PRD that ships with its own eval harness turns the fuzziness of AI into something a team can steer. That's the shift: the spec no longer just describes the product, it proves it.