You ask Amplitude Global Agent why activation fell. It returns a plausible explanation, but you still can’t tell which events it examined, whether the comparison was valid, or what your product team should do next.
The fix is to treat the prompt as an analysis specification. Define the decision, provide the relevant analytics context, constrain unsupported conclusions, and make the agent show its work. You will get an answer that is easier to verify and more useful in a product review.
Start with the decision, not a broad request for insights
Requests such as “analyze activation” leave several decisions unresolved. The agent must guess what activation means, which users belong in the analysis, which period matters, and what kind of answer you expect. Even a polished response may answer the wrong question.
Before writing the prompt, complete this sentence: “After reading the answer, we need to decide whether to…” Your ending might be “change the onboarding sequence,” “investigate a recent release,” or “prioritize one segment for discovery.” That decision gives the analysis a destination.
Then assign a role that matches the work. “You are a product analyst investigating activation performance” is more useful than “You are a helpful assistant.” Add the audience as well. An executive needs the size and business relevance of a change; a product trio also needs the affected steps, segments, and follow-up questions.
A strong opening contains three elements:
- Role: the analytical perspective the agent should take.
- Decision: what the team will choose or investigate after reading the result.
- Success criteria: what the answer must establish before it is useful.
For example: “You are a product analyst helping the onboarding team decide whether to redesign a weak activation step. Identify the largest meaningful drop-off, show which defined segment is most affected, and separate measured findings from possible explanations.”
Give the agent a compact analytics contract
The most reliable prompt names the data the agent may use. Include the relevant event names, property names, segment definitions, filters, and timeframe. If activation has an internal definition, write it out rather than relying on the agent to infer it.
This is a retrieval-first approach: put authoritative definitions, dashboard context, and prior query logic into the request before asking for interpretation. Concrete grounding reduces room for invented assumptions and makes repeated analyses easier to compare. A structured prompt can also specify the role, business objective, allowed data, and output fields.
| Prompt element | What to provide | What it prevents |
|---|---|---|
| Metric definition | The exact event sequence or outcome that counts | A different interpretation of activation or retention |
| Population | Included users or accounts and explicit exclusions | Comparisons across unlike populations |
| Segments | Named properties and the values to compare | Arbitrary segmentation |
| Timeframe | The analysis period and comparison period | Hidden or inconsistent date choices |
| Evidence boundary | The events, properties, definitions, and dashboards allowed | Unsupported claims presented as measured facts |
| Output contract | Required sections, fields, ordering, and length | A long narrative that cannot be reviewed quickly |
Do not dump every available definition into the context. Include only what the question requires. More context is useful when it resolves ambiguity; irrelevant context competes for attention and makes the prompt harder for a teammate to audit.
Use a reusable prompt that exposes uncertainty
You can adapt the following structure for activation, retention, anomaly investigation, or another behavioral analysis:
- Role and audience: “Act as a product analyst. Write for the product manager and analytics lead responsible for [area].”
- Decision: “Help us decide whether to [decision].”
- Question: “Determine [specific analytical question].”
- Definitions: “For this analysis, [metric] means [explicit event or outcome definition].”
- Data context: “Use these events: [names]. Use these properties: [names]. Compare these segments: [definitions]. Analyze [timeframe] against [comparison period]. Apply [filters and exclusions].”
- Constraints: “Use only the supplied Amplitude analytics events, properties, and definitions. Do not treat an unmeasured explanation as a finding.”
- Output: “Return the metric result, segment comparison, timeframe, evidence, interpretation, confidence or limitation, and recommended next check.”
- Fallback: “If the available data cannot answer the question, state what is missing and provide the smallest follow-up query needed.”
The fallback matters. Without it, the agent has an incentive to complete the requested narrative even when the evidence is incomplete. A useful failure is specific: it identifies a missing event, undefined property, absent comparison, or ambiguous metric. Your team can fix that. A confident guess is harder to detect.
Ask for measured findings, interpretations, and recommendations as separate fields. A measured drop-off is evidence. A claim that users were confused is an interpretation unless the supplied data establishes it. A recommendation to inspect session replay or conduct customer interviews is a next step, not proof of the cause. Keeping those layers separate makes the result safer to use in prioritization.
Turn prompt quality into a small product evaluation
Do not judge a prompt by whether one response sounds intelligent. Save the prompt version, input context, and output. Then test it against a question whose answer your team already knows. This gives you a reference point for accuracy before you use the template on an ambiguous problem.
Score each version on three dimensions:
- Accuracy: Did the answer use the supplied definitions, filters, segments, and timeframe correctly?
- Clarity: Can a reviewer distinguish evidence, interpretation, limitations, and next steps?
- Actionability: Does the result support the stated decision or name the next query required?
Change one meaningful element at a time. You might compare a broad objective with a decision-specific objective, a narrative response with a fixed output contract, or an unrestricted answer with an explicit evidence boundary. Run the same test question through each variant. Otherwise, you will not know which change improved the result.
Commit to two or three prompt iterations for one critical workflow. Review the failures, tighten the ambiguous instruction, and keep the better-performing version. Within a sprint, that process can produce a reusable template for a recurring analysis such as activation, retention, or anomaly detection.
Store winning prompts with their required inputs and known limitations. A template without those notes becomes cargo cult: teammates copy the wording but omit the definitions that made it work. Treat the prompt, context requirements, evaluation question, and scoring criteria as one asset.
Key takeaways
- State the product decision before requesting analysis.
- Define the metric, population, segments, filters, and timeframe explicitly.
- Restrict conclusions to the analytics evidence you supplied.
- Separate measured findings from interpretations and recommended actions.
- Require a specific fallback when the data is insufficient.
- Version and score prompts for accuracy, clarity, and actionability.
Start with the recurring Amplitude question that currently creates the most debate. Write its decision, definitions, evidence boundary, and output contract. Run two or three scored iterations, then give the winning template to another product manager. If they can obtain a defensible answer without you translating the prompt, it is ready to become part of the team’s operating system.
Leave a Reply