We Do Not Have Ground Truth – Can We Still Ship an AI Feature?

In the rapidly evolving landscape of AI-powered B2B SaaS products, one question frequently haunts product teams:

"We don't have ground truth for this feature—can we still ship it?"

This conundrum rings especially true when building on commoditized large language models (LLMs) like Anthropic’s Claude Opus 4.7. Without a golden standard or an oracle to declare the perfect answer, how do we design, measure, and ship AI features responsibly? What techniques help us balance ship risk, user trust, and measurable quality?

Drawing on 12 years of product management experience, shipping internal AI agents across support, risk, and developer tools, and incorporating insights from frameworks like PM Toolkit, this article unpacks pragmatic patterns and principles to guide your journey. We’ll cover why you must build eval first, embrace workflow-first thinking, carefully choose reasoning models to manage hallucination risk, and systematically use feature flags and kill switches to mitigate ship risk.

The Challenge: We Cannot Measure “Correctness” in the Traditional Sense

Unlike traditional software where outputs are deterministic and correctness is binary, AI features often operate in ambiguous domains:

Generating free-form text summaries or code Answering open-ended questions without fixed answers Recommending actions in complex workflows

In these cases, ground truth is elusive or entirely absent. We cannot simply compare output to a fixed golden set and https://pmtoolkit.ai/learn/ai-modern-pm/ai-literacy-for-pms compute an accuracy percentage.

Yet, we cannot let this lack of “correctness” stop us from delivering value. The question then becomes, how do we define quality, trust, and success metrics when outputs vary, are subjective, or grounded in external documents or user context?

Workflow-First Thinking: Trust as the Real Moat

Before we jump to evaluating models, we must zoom out and ask:

What does the user do today? What pain points does AI help alleviate or accelerate? How does the AI output fit seamlessly into the user's workflow?

Workflow-first thinking is the foundation of sustainable AI products. It prioritizes designing around user jobs and trust mechanisms rather than chasing raw model performance metrics.

Trust is the moat. Users may tolerate occasional hallucinations or “off” outputs if:

The AI suggestions are surfaced clearly as suggestions, not commands Users maintain control and can easily edit or override outputs Features elevate productivity rather than block workflows Errors gracefully degrade with fallbacks or transparent failure modes

For example, consider a compliance monitoring agent integrated into a support workflow. It does not have a singular golden verdict but highlights risky responses probabilistically, aiding human review. Here, trust emerges from transparency and actionability, not absolute correctness.

Eval Design as Product Specification: Define What "Good Enough" Means

When you cannot measure correctness conventionally, eval design becomes your product specification.

Rather than “ground truth,” build an evaluation framework that tests targeted behaviors reflecting user value and safety. Treat your eval cases like carefully crafted bug reports with expected output patterns, some fuzziness allowed but firm guardrails.

Steps to Build a Robust Eval Framework

Identify key user intents and edge cases: Map the most common tasks and error modes Capture representative examples: Use real production data with all personal info scrubbed Define measurable success criteria: Can be binary pass/fail or graded—Does the output prioritize relevant info? Avoid hallucination? Follow compliance rules? Automate evals where possible: Create spreadsheet-based or scriptable tests, making it easy for engineers and PMs to run routinely Continuously maintain and expand evals: Tracking retry rate and failure modes helps illuminate regressions after model or prompt changes

This structured approach ensures you ship to a known quality bar, avoiding ship-on-vibes or vague "accuracy improved" claims devoid of golden sets, a pitfall that all too often leads to silent regressions and eroding trust.

Reasoning Model Tradeoffs: Balancing Power and Hallucination Risk

Reasoning-capable LLMs like Claude Opus 4.7 bring immense power but also increased risk of hallucination, especially if used as open-ended Q&A or decision engines without grounding or retrieval retrieval.

Hand-wavy “reasoning is better” claims must be interrogated. For grounded Q&A, simpler retrieval-augmented generation (RAG) models might yield more reliable answers. For complex reasoning or summarization, token limits and latency grow, and hallucination grows harder to detect and mitigate.

Understanding the tradeoffs is critical:

Model Type Strength Risk When to Use Retrieval-augmented base model Grounded answers, lower hallucinations Limited reasoning depth Fact lookup, grounded Q&A Reasoning model (e.g., Claude Opus 4.7) Complex synthesis, multi-step logic Higher hallucination, harder eval Summarization, ideation, internal agent reasoning

This decision affects your ship risk profile and eval design deeply. For example, in critical compliance workflows, hallucination risk might be unacceptable, so anchoring to data sources and more conservative models prevails.

Feature Flags and Kill Switch: Essential Tools to Manage Ship Risk

Even with solid eval frameworks and workflow-first design, unforeseen issues happen post-launch. So, incorporate infrastructure-level safety nets:

Feature flags: Gradually roll out AI features to subsets of users, allowing real-world monitoring to catch unseen failure modes Kill switches: Emergency turn-offs that can disable AI features instantaneously if they cause regressions or harmful outputs

These tools preserve customer trust and prevent costly downtime or harm. Teams deploying Anthropic’s new models, for example, often combine feature flags with proactive eval dashboards tracking regression metrics to decide on ramp-up or rollbacks.

Summary: How to Ship AI Features Without Ground Truth

Start by understanding your user’s existing workflow and pain points. Trust and usability are your competitive moat, not raw model score improvements. Design your eval framework first. Treat eval cases as living product specs that measure meaningful success criteria, even when you cannot measure strict correctness. Choose your models wisely. Understand reasoning model tradeoffs with hallucination risk, and prioritize grounded outputs for high-stakes tasks. Use feature flags and kill switches rigorously. Gradual rollouts and immediate fallback prevent negative impact and allow iterative learning. Measure retry rates and failure modes continuously. Keep a sticky note with retry rates visible to ensure regressions don’t slip through.

Many companies attempt to ship wrapped “AI features” without rigorous product design or testing—this leads to silent regressions, eroded trust, and wasted developer cycles. By applying these best practices from frameworks like PM Toolkit and harnessing advanced, but cautiously used models like Anthropic’s Claude Opus 4.7, you can responsibly ship transformative AI functionality even in the absence of a golden ground truth.

Ultimately, the ability to ship with uncertain correctness is not magic—it’s a discipline blending workflow empathy, smart eval design, model understanding, and thoughtful rollout strategies. Follow this path, and your users—and your team—win.