Tag: deployment frequency

  • Auditable AI Code Review: A Practical Operating Model

    Auditable AI Code Review: A Practical Operating Model

    You are not deciding whether an AI model can find bugs in a pull request. You are deciding whether an automated reviewer can participate in a production control without leaving your team unable to explain, challenge, or reverse its decision.

    If the only evidence behind an approval is a bot comment that says the change looks safe, keep the system advisory. An auditable AI reviewer needs a bounded mandate, a deterministic approval policy, traceable evidence, and a feedback loop tied to production outcomes. Build those controls first, and faster review becomes a consequence rather than a gamble.

    Start with a decision contract, not a model prompt

    An approval is a policy decision. The model can supply findings, evidence, and a recommendation, but it should not define the conditions under which its own recommendation becomes authoritative.

    Write a decision contract before selecting a model or tuning a prompt. It should answer five questions:

    • What may the system decide? Typical outcomes are approve, request changes, provide non-blocking comments, or escalate to a person.
    • Which changes are eligible? Eligibility should be determined by explicit repository, path, change-type, test, ownership, and reversibility rules.
    • Which checks are mandatory? An eligible pull request should not be approved if a required review lens failed to run, returned incomplete evidence, or produced an unresolved blocking finding.
    • When must the system abstain? Missing context, conflicting findings, unavailable tools, excessive scope, low-confidence evidence, and protected code paths should cause escalation rather than optimistic approval.
    • Who owns the result? Name the engineer accountable for the change, the owner of the review policy, and the person or group authorized to change the automation boundary.

    The core approval rule can be expressed plainly: the change is eligible, every mandatory check completed, no blocking issue remains, the evidence record is complete, and no human-review requirement was triggered. Encode that rule in a controller your team can inspect and test. Do not bury it inside natural-language instructions to the model.

    This separation gives you a clean control plane. Review agents analyze the change. A policy engine evaluates their structured results. A narrowly permissioned service performs the approved action. The model never gets to reinterpret the boundary at the moment it encounters a difficult pull request.

    Auditability does not require a future model run to reproduce the same words. Model endpoints, retrieved context, and dependencies can change. It requires the original decision to remain reconstructable from preserved inputs, outputs, policies, tool results, and versions. A skeptical engineer should be able to determine why the pull request was approved without trusting the personality or reputation of the bot.

    Split the review into specialist checks with explicit evidence

    A single prompt asking whether a pull request is safe compresses several different judgments into one opaque answer. Decompose the review so that each judgment has a clear purpose, input set, output schema, and failure mode.

    A practical review pipeline can include these specialist lenses:

    • Problem-definition quality: Is the requested behavior specific enough to review, and are the acceptance conditions testable?
    • Intent alignment: Does the diff implement the stated change without silently expanding or contradicting it?
    • Scope and dependency impact: Which callers, data flows, interfaces, jobs, or services can the change affect outside the edited files?
    • Logical correctness: Do the changed execution paths handle expected states, boundary conditions, and failure paths?
    • Test adequacy: Do the tests exercise the behavior that changed, and did the required checks actually run against the reviewed commit?
    • Security and privacy: Does the change alter trust boundaries, permissions, authentication, secrets, sensitive data handling, or externally controlled inputs?
    • Local engineering guidance: Does the implementation comply with versioned repository conventions, architectural constraints, and known anti-patterns?
    • Deployment and recovery: Can the change be observed, disabled, or rolled back without creating a second unsafe operation?

    Every specialist should return the same minimum structure: a check identifier, pass/fail/escalate status, a concise claim, evidence tied to files or tool output, the applicable rule version, severity, and a recommended action. A finding such as a possible regression is not auditable. A finding that identifies the affected path, explains the conflicting behavior, points to the relevant code, and names the violated policy is.

    Run independent checks before aggregation, and preserve every result even when the final decision is approval. The aggregator may deduplicate findings, but it should not erase dissent. If the intent checker says the change is aligned while the execution-path checker finds a contradiction, route the conflict to a person.

    Review context must extend beyond the visible diff. A seemingly harmless one-line copy change was once found to contradict validation behavior elsewhere in the codebase. That is the kind of defect a diff-only reviewer is structurally unlikely to see. Give relevant checks controlled access to callers, validators, schemas, tests, ownership metadata, and versioned internal guidance, then record exactly which context each check used.

    More context is not automatically better. Retrieval should be targeted and attributable. When a finding depends on an internal rule, capture the rule identifier and version. When it depends on a test, capture the command, commit, status, and output reference. When it depends on inferred execution flow, record the relevant path so a maintainer can inspect it.

    Treat pull-request text, code comments, test fixtures, generated files, and documentation on the changed branch as untrusted data. They can contain instructions designed to redirect an agent. Load approval policy from a protected service or the trusted base branch, not from files the pull request can rewrite. Run proposed code in an isolated environment, mediate tool calls through an allowlist, and keep approval or merge credentials outside the model’s reach. The policy controller should translate a valid decision into an action; the model should never hold the credential that performs it.

    Set the automation boundary with hard eligibility gates

    Do not begin by assigning every pull request a risk score and approving anything below a convenient number. A composite score can hide a disqualifying condition: a tiny authorization change may receive a low size score even though its blast radius is high. Apply hard gates first. Use scoring only to route changes that remain eligible after those gates.

    Common reasons to require human review include:

    • Authentication, authorization, permissions, cryptography, secrets, or trust-boundary changes.
    • Payments, billing, entitlements, destructive data operations, or irreversible migrations.
    • Public API contracts, shared schemas, release infrastructure, or broadly consumed dependencies.
    • A pull request that changes its own review policy, test requirements, ownership rules, or deployment controls.
    • Missing required tests, failed or stale CI results, unavailable analysis tools, or a mismatch between the reviewed commit and the tested commit.
    • Changes spanning too many concerns, components, or execution paths for the approved review envelope.
    • An active incident, an unclear rollback path, or a direct request for human review.

    There is no universal line-count threshold for a small pull request. Derive limits from your architecture and incident history, then version them. A change to a central permission function may be riskier than a much larger isolated test refactor. Scope should include dependency reach and behavior change, not just added and deleted lines.

    A staged authority model keeps the boundary legible:

    ModeWhat the AI reviewer may doWho decides the mergeAppropriate use
    ShadowProduce a private decision record without affecting the pull requestHuman reviewerBaseline evaluation and policy tuning
    AdvisoryPost evidence-backed, non-blocking findingsHuman reviewerMeasuring usefulness and false alarms in normal work
    BlockingRequest changes for narrow, testable policy violationsHuman reviewer after resolutionStable rules with clear evidence and an appeal path
    Bounded approvalApprove only changes that pass every eligibility and review conditionPolicy controller within its delegated scopeValidated low-risk change classes with complete audit records
    Mandatory escalationSummarize evidence and route the changeNamed human ownerSensitive paths, conflicting findings, missing evidence, or any requested human review

    Do not turn bounded approval into an auto-approval quota. Coverage is a result of demonstrated safety, not a target that should pressure teams to weaken eligibility rules.

    One high-frequency engineering environment reports that more than 93% of pull requests across two main codebases are agent-driven and more than 19% are approved without a human reviewer. Its reported median merge time fell from 75.8 minutes with human review to 14.6 minutes with AI approval, while downtime from breaking changes declined 35% as deployments doubled. Those organization-level results show that bounded automation can coexist with high deployment frequency and improving safety outcomes. They do not prove that AI approval caused the downtime reduction, and they should not be imported as another team’s launch threshold.

    Keep the escape hatch explicit. Any engineer should be able to request a human review without defending the choice. The accountable engineer should still watch the change in production and be ready to roll it back. Automated approval changes who performs a review step; it does not transfer ownership of the production outcome to a model.

    Preserve the evidence, then earn autonomy through evaluation

    Build a decision record that survives model and policy changes

    Create an append-only decision event for every review attempt, including abstentions and failed runs. At minimum, retain:

    • Repository, pull-request identifier, base commit, reviewed head commit, author, accountable owner, and timestamps.
    • The pull-request description and acceptance criteria as they existed when the decision was made.
    • Eligibility rules, protected-path rules, ownership data, prompt-template identifiers, and policy versions.
    • Model provider and model identifier, relevant runtime settings, retrieval configuration, and tool versions.
    • The context each specialist received, including immutable references or preserved snapshots for mutable material.
    • Structured specialist outputs, supporting evidence, tool invocations, CI results, conflicts, and failures.
    • The deterministic rule evaluation that produced approve, block, comment, or escalate.
    • Subsequent human overrides, appeals, edits, approvals, merges, rollbacks, hotfixes, and linked incidents.

    Store concise decision rationale and inspectable evidence, not hidden chain-of-thought. An auditor needs to know which claim was made, what supported it, which rule applied, and how the controller reached the outcome. Private internal reasoning is neither necessary nor a reliable substitute for those artifacts.

    Apply the same security discipline to review logs that you apply to source code. Minimize captured secrets and personal data, control access, define retention, and log policy changes. If a model or retrieval service cannot handle the code under your data-governance requirements, that repository is not eligible for the workflow.

    Evaluate decisions, not polished comments

    A review can sound thoughtful and still approve the wrong change. Build an evaluation set around decisions and evidence rather than writing quality.

    1. Assemble representative cases. Include clean pull requests, valuable historical human findings, escaped defects, incident-causing changes, incomplete requirements, sensitive paths, cross-component changes, and attempts to manipulate the reviewer through repository content.
    2. Label the expected control outcome. For each case, identify whether the correct action is approve, request changes, or escalate. Record the evidence that an acceptable review must surface.
    3. Separate clear cases from disputed ones. Known incident causes and explicit policy violations can provide strong labels. Ambiguous architectural judgments need maintainer adjudication, and disagreement should remain visible rather than being forced into false certainty.
    4. Freeze a holdout set. Use one portion to improve prompts, retrieval, and policy. Keep another portion unseen until release evaluation so repeated tuning does not create a misleading score.
    5. Compare equivalent cohorts. Evaluate AI and human review on the same risk classes and change types. Comparing AI-approved low-risk changes with all human-reviewed pull requests confounds reviewer quality with task difficulty.

    Track metrics that expose different failure modes:

    • Decision accuracy: How often did the system choose the expected approve, block, or escalate outcome?
    • False auto-approval rate: How often did it approve a labeled case that should have been blocked or escalated? Break this out by severity and risk class.
    • Blocking precision: Of the findings that stopped a change, how many maintainers judged valid and actionable?
    • Known-defect recall: Which seeded or historically verified defects did the review catch? Label this carefully; it is not recall over every defect that might exist.
    • Evidence completeness: Can every decision be traced to required checks, immutable inputs, policy versions, and supporting artifacts?
    • Abstention and override rates: Where is the system uncertain, and where do engineers reverse it? Investigate patterns by repository and change class.
    • Delivery performance: Measure review latency and merge time, but only alongside quality metrics.
    • Production outcomes: Track rollbacks, hotfixes, escaped defects, incidents, downtime, and customer impact for comparable risk cohorts.

    Comment helpfulness is useful feedback, but it is not a safety metric. Engineers may like a concise reviewer that misses a critical defect, or dislike a strict reviewer that correctly blocks an unsafe change. Keep usefulness, correctness, and production impact as separate measures.

    Roll out by change class and turn escapes into regression tests

    Move from shadow mode to advisory comments, then to narrow blocking rules, and only then to bounded approval. Start with one repository and one low-risk, reversible change class. Write the exit criteria before the pilot begins, including acceptable false-approval and false-block rates, required audit completeness, escalation behavior, and production guardrails.

    Canary each expansion. Maintain a kill switch that disables new automated approvals without removing the accumulated evidence. If a required service, model, retrieval index, test runner, or policy store is unavailable, fail closed and return the pull request to the human path.

    When an approved change causes a production problem, diagnose the control layer that failed:

    • Was the change wrongly eligible?
    • Did retrieval omit relevant code or guidance?
    • Did a specialist miss or misclassify the defect?
    • Did the aggregator suppress a conflict?
    • Did the policy permit approval despite the evidence?
    • Did CI test a different commit or an incomplete environment?
    • Did production monitoring fail to surface the effect promptly?

    Add the case to the regression suite, version the corrective policy or guidance, rerun the holdout evaluation, and preserve the relationship between the incident and the updated control. That is eval-driven development applied to governance: every escape should make a specific layer harder to fail in the same way again.

    Key takeaways

    • AI output is an input to approval, not the approval policy itself.
    • Use deterministic eligibility gates before any model-based risk judgment.
    • Decompose review into specialist checks that return claims, evidence, rule versions, and explicit pass/fail/escalate states.
    • Keep policy and credentials outside the pull request and outside the model’s control.
    • Preserve enough evidence to reconstruct the original decision even when the model, repository, or internal guidance later changes.
    • Expand autonomy only when evaluation and comparable production cohorts support it; never optimize for auto-approval coverage by itself.

    Your first useful milestone is not an AI-approved pull request. It is a shadow decision that a maintainer can reconstruct, dispute, and improve. Once that record is dependable, grant the smallest reversible slice of authority, watch what reaches production, and make every expansion earn its place.

    References

  • How We Built PR Review Bots In‑House for a Fraction of the Cost—and How You Can Too

    How We Built PR Review Bots In‑House for a Fraction of the Cost—and How You Can Too

    PR review bots are all the rage, but they cost a premium. We built our own for cheap that work just as well, if not better. Here's how.

    As a VP of Product Management, I care deeply about the velocity and quality of our software delivery. The decision to build our own pull request (PR) review agents came from a simple calculus: we needed tighter control over developer experience, CI/CD integration, and cost—without sacrificing accuracy or reliability. The result was a pragmatic system that accelerates reviews, improves code quality, and pays for itself through faster feedback loops.

    Before we wrote a line of code, we defined success. Our objectives were to shorten review cycles, reduce back-and-forth on style and test coverage, and surface risks earlier—measured against DORA metrics like lead time and deployment frequency. That focus aligned the team, guided our build vs buy decision, and anchored scope to the highest-impact use cases.

    We started rules-first, AI-optional. The initial release enforced guardrails that are universally valuable: linting and formatting checks, required test coverage thresholds, commit message standards, ownership validation (CODEOWNERS), and basic security scans. These automated gates eliminated predictable review friction, freeing engineers to focus on logic and architecture rather than style debates.

    Then we layered intelligence where it mattered. We added lightweight, explainable checks for common code smells and dependency risks, plus optional natural-language summaries that turn large diffs into concise context. Where appropriate, we introduced agentic AI workflows to triage PRs by risk, draft review comments, and suggest missing tests—always keeping humans in the loop. This hybrid approach kept costs low and outcomes high.

    Integration with our CI/CD pipeline was non-negotiable. We wired GitHub/GitLab webhooks to a stateless service that queued work, executed checks in containerized workers, and posted results back as status checks and review comments. Caching, parallelization, and smart diff-scoping ensured we only computed what changed, keeping the experience snappy even on large repos.

    Adoption hinged on developer experience. We made the bot’s feedback fast, specific, and actionable, with clear remediation steps and links to documentation. Feature flags allowed teams to opt into new checks gradually. ChatOps commands enabled quick overrides for emergencies, while policy-as-code kept rules visible, versioned, and auditable.

    We treated this like any product: eval-driven development for accuracy, ongoing telemetry for false-positive rates, and explicit SLAs for response times. We instrumented outcomes end-to-end—tracking PR cycle time, comment-to-merge ratios, and rework—so we could prove the ROI and tune the system without guesswork.

    The outcome: a reliable PR review companion that runs on a shoestring budget, integrates cleanly with our workflows, and measurably improves engineering throughput. If you’re weighing build vs buy, start small with rules that deliver immediate value, then layer intelligence where it earns its keep. With a clear product strategy, you can stand up capable PR review bots quickly—and scale them as your needs grow.

    If you’re ready to try this yourself, begin with your top three friction points in code reviews, wire them into your CI/CD checks, and pilot with a single team. Iterate weekly, measure relentlessly, and let your developers be your strongest signal. You’ll be surprised how far a pragmatic, product-led approach can take you.


    Inspired by this post on Amplitude – Perspectives.


    Book a consult png image
  • A Product Leadership System for Faster, Clearer Execution

    A Product Leadership System for Faster, Clearer Execution

    Your roadmap is full, every function has a planning ritual, and experienced people are working hard. Yet decisions still wait, priorities keep reopening, and substantial work reaches customers later than anyone expected. Adding another process layer will not solve that problem.

    You need an execution system: explicit ownership, small batches, a dependable decision cadence, direct customer feedback, and a scorecard that distinguishes progress from activity. When those elements reinforce one another, your teams can move faster without lowering the quality bar or routing every judgment through you.

    Give each team an operating contract, not just a roadmap

    A roadmap identifies intended destinations. It rarely tells a team how to make the decisions required to reach them. That gap is where autonomy turns into ambiguity: product believes it owns the sequence, engineering waits for scope to stabilize, design explores a wider problem, and an executive assumes a requested feature is already committed.

    Before an initiative becomes active work, give the team a short operating contract. It should fit on one page and answer these questions:

    • Whose problem are you solving, and in what specific scenario does it occur?
    • What observable customer or business outcome should change if the work succeeds?
    • Who is accountable for the initiative and its sequence?
    • Which constraints are fixed, and which assumptions remain open?
    • What is explicitly outside the current scope?
    • What is the smallest end-to-end slice that can produce useful evidence?
    • What evidence will support the next decision?
    • When will that decision be made, and who has the right to make it?

    The owner is not the person who approves every task. The owner keeps the problem, outcome, sequence, and unresolved decisions coherent. Engineering, design, research, and product still make solution decisions together inside the stated boundaries.

    This contract also protects the team from executive requests that arrive as solutions without context. When someone asks for a feature, do not turn the request directly into a backlog item. Translate it into a problem entry first: the affected customer, the workflow that breaks, the evidence behind the request, the relevant constraint, and the result the requester expects. A commercially important request can remain urgent after that translation, but the team can now evaluate it rather than merely obey it.

    Set escalation boundaries at the same time. A team should escalate when decision rights are unclear, two constraints conflict, a priority change affects another team, or the work crosses an agreed risk boundary. It should not need escalation merely because a solution choice is consequential. If every consequential choice travels upward, the team is not autonomous; it is a queue feeding a senior leader.

    Finally, maintain one prioritized backlog for the team. Separate executive, product, engineering, and sales backlogs create hidden competition. The operating contract establishes the logic, and the single backlog makes the resulting sequence visible.

    Run a weekly loop around decisions and customer learning

    Many product cadences organize meetings while leaving decisions to happen unpredictably. A useful cadence does the opposite. Every recurring touchpoint should help the organization choose, learn, or remove a constraint.

    A workable leadership week looks like this:

    • Monday: Confirm the few priorities that matter, identify decisions that could block progress, and resolve changes in sequencing. Do not reread the entire roadmap.
    • Midweek: Review selected product requirements, design flows, research findings, and engineering readiness. Concentrate on ambiguity, batch size, and untested assumptions.
    • Thursday: Spend time with customers and partners. Put working slices in front of them when possible, and bring the resulting evidence back to the team.
    • Friday: Write down what changed in your understanding. Update the backlog, decision log, and operating contracts where the evidence warrants it.

    The sequence matters. Monday establishes intent. Midweek exposes execution risk while there is still time to change course. Customer contact tests the team’s reasoning. Friday turns scattered observations into organizational memory. Without the synthesis step, customer conversations can become interesting anecdotes that never alter a decision.

    Make the weekly demo the heartbeat of the team. A good demo starts with the user scenario and the intended outcome, shows the smallest working behavior, states what the team learned, and ends with the next decision. A tour of completed tickets is not a substitute. For platform or infrastructure work, demonstrate working behavior, operational evidence, or a retired technical risk rather than manufacturing a customer-facing screen.

    When a team repeatedly has nothing meaningful to demonstrate, inspect the system before questioning effort. The batch may be too large. A dependency may lack an owner. Decisions may be waiting in an approval queue. The team may be building several disconnected components before completing one testable path. The correction is usually to narrow the slice, clarify the decision, or remove the dependency.

    A thin slice is not an arbitrary reduction in scope. It must preserve one coherent scenario, reach a state where someone can evaluate it, and create evidence for a consequential next choice. Backend, frontend, and enablement tasks can all be necessary, but completing them separately does not create a feedback loop.

    Put product and revenue in the same operating loop

    Product and revenue drift apart when they maintain different versions of the customer. Product sees research themes and usage behavior. Revenue sees active deals, objections, urgency, and willingness to pay. Neither view is sufficient on its own.

    Use one customer narrative, one shared pipeline of problems worth solving, and one scorecard. Review them together every week. Each proposed problem should carry the customer segment, affected workflow, available evidence, commercial context, expected outcome, and complexity the solution could add.

    Then make the sequencing decision explicit:

    • Solve now: The problem is important enough, supported well enough, and compatible with the current strategy.
    • Stage for scale: The need is credible, but the team must first validate the pattern, build a reusable foundation, or resolve a dependency.
    • Do not add: The request is too narrow, conflicts with the product direction, or creates complexity that its value does not justify.
    • Sunset: Existing functionality consumes attention without contributing enough customer value or strategic leverage.

    This turns product-versus-sales conflict into a visible portfolio decision. Revenue contributes evidence and urgency. Product protects coherence and long-term defensibility. Both functions see why an item moved, waited, or stopped.

    Measure outcomes, flow, and quality as separate signals

    A team can ship frequently without improving a customer outcome. It can also improve an outcome temporarily while accumulating quality problems that make the pace unsustainable. Your scorecard needs to keep those conditions separate.

    For each important bet, review three signal groups:

    • Outcome: The observable customer or business result the team is trying to change, supported by current evidence rather than a list of releases.
    • Flow: Deployment frequency and the age or state of the current thin slice. These signals reveal whether value and learning can move through the system.
    • Quality: Change failure rate and the recurring friction exposed in customer feedback, support conversations, or postmortems.

    Use the scorecard to direct attention, not to automate judgment. If deployment frequency is healthy but the intended outcome is not moving, inspect the hypothesis, target customer, and value proposition. More releases may simply deliver the wrong idea faster. If deployment frequency falls, examine batch size, dependencies, and delayed decisions. If change failure rate worsens, narrow the slice and strengthen readiness or recovery before asking the team to accelerate.

    Do not rank unlike teams by raw deployment counts. Use trends within the relevant product and technical context. The point is to find constraints and make decisions, not to turn a diagnostic signal into a performance contest.

    Write outcome-focused OKRs with enough precision to guide a trade-off. A useful structure is: for a named user and scenario, improve an observable result from its current baseline toward an agreed target by the review point, without damaging a stated guardrail. Establish the baseline before debating the target. If the team cannot observe the result, say that plainly and make instrumentation or customer evidence part of the initial slice.

    Feature count, roadmap completion, tickets closed, and activity volume can help with local planning. They are not proof of customer value. Treat them as operational context, not as the headline definition of success.

    Keep the executive view compact. Each team should be able to present its intended outcome, current evidence, deployment-frequency trend, change-failure trend, most important customer learning, and next unresolved decision. If a metric never changes a question, a priority, or an intervention, remove it from the review.

    Stay close to the work without taking the work away

    Product leaders lose judgment when they only consume summaries. They become bottlenecks when they join every working session. The useful middle ground is deliberate sampling: inspect enough real work to calibrate your view, then give feedback that strengthens the team’s next decision.

    Each week, sample a rotating set of artifacts such as a product requirements document, a design flow, customer research notes, a postmortem, or a customer thread. You are not trying to approve every artifact. You are checking whether the operating system is producing clear thinking.

    Use questions that reveal decision quality:

    • Does the requirement name a user scenario and a problem, or does it begin with a predetermined feature?
    • Does the design expose a complete path that can be tested, or only polished fragments?
    • Do the research notes separate what customers did and said from the team’s interpretation?
    • Does the postmortem change an operating mechanism, or merely remind people to be careful?
    • Does the customer thread reveal a pattern, an important exception, or one loud request?
    • Can the team state the next decision this artifact is meant to support?

    Feedback should create motion. Name the user scenario, identify the friction or ambiguity, state the decision principle, propose a smaller testable slice when appropriate, and clarify the next decision. A vague comment such as make this more strategic forces the team to guess what you mean and then wait for another review.

    I use a simple leadership boundary: push hard on problem clarity, sequencing, and the quality bar; leave room on solution design and implementation. That boundary keeps accountability with leadership without converting senior judgment into remote-control product management.

    Exemplars make this boundary easier to scale. Keep a small, current library of strong problem statements, concise narrative memos, useful research syntheses, clear acceptance criteria, and honest postmortems. Show why each example is effective. Teams learn a quality bar faster from visible work than from an expanding rulebook.

    Create short paths for decisions and uncomfortable information

    Open office hours give anyone a direct route for a difficult escalation, unfinished design, customer insight, or cross-team conflict. Run them as a decision forum, not an extra status meeting. Capture the decision, owner, rationale, and follow-up so people who were not present can still act consistently.

    Keep weekly one-to-ones with your leaders as well. Office hours expose work across the organization; one-to-ones develop judgment, surface recurring constraints, and help a leader notice when someone is absorbing ambiguity on behalf of the system.

    Fast feedback from leadership matters because waiting expands batches. When teams expect a long approval cycle, they tend to gather more material and seek approval for more decisions at once. Publish clear decision rights and a dependable response path. If you do not need to make the decision, say so immediately and return it to the named owner.

    Spend unscripted time with individual contributors, too. Formal reporting lines filter information. Direct exposure to the people building, researching, designing, supporting, and selling the product helps you hear where the written process and actual work have diverged.

    Install the system without reorganizing first

    You do not need a company-wide transformation program to test this operating model. Start with one important initiative that is moving slowly or generating repeated disagreement. Keep the current reporting structure and change the mechanics around the work.

    1. Capture the current friction. Identify where the initiative waits, where priorities conflict, which decisions keep reopening, and where work returns for avoidable clarification.
    2. Write the operating contract. Name the problem, outcome, owner, constraints, non-goals, initial thin slice, required evidence, and next decision.
    3. Collapse the work into one sequence. Bring product, engineering, executive, and commercial requests into one prioritized backlog. Preserve their context rather than preserving separate queues.
    4. Run the weekly loop. Set priorities on Monday, inspect selected artifacts midweek, expose work and assumptions to customers, and synthesize the learning on Friday.
    5. Publish the compact scorecard. Show the intended outcome, deployment frequency, change failure rate, newest customer evidence, and next decision. Do not wait for a perfect dashboard.
    6. Inspect the mechanism after a full loop. Remove one gate that added waiting without adding learning, divide one oversized batch, and clarify one decision right that caused an escalation.

    During the review, ask concrete questions: What waited for a decision? What was redone because the original problem was unclear? Which customer signal changed the plan? Which metric caused an intervention? Which request arrived without enough context? Where did leadership provide useful boundaries, and where did it take ownership away from the team?

    Expand the model only after the team can explain how it changed actual work. Copying the ceremonies without the decision rights, customer exposure, and scorecard will create more meetings, not a stronger execution system.

    Key takeaways

    • Start each important initiative with a one-page operating contract that connects a real customer problem to an owner, outcome, constraints, thin slice, and next decision.
    • Protect autonomy with explicit boundaries. Escalate conflicting priorities and constraints, not every consequential solution choice.
    • Organize the week around decisions, working evidence, customer contact, and written synthesis rather than status reporting.
    • Read outcomes, deployment frequency, and change failure rate together. No single signal can tell you whether the team is delivering sustainable value.
    • Sample real artifacts and give specific feedback, while leaving solution and implementation ownership with the team.
    • Give product and revenue one customer narrative, one problem pipeline, and one scorecard so trade-offs become visible sequencing decisions.

    At your next Monday priority review, choose one live initiative and write its operating contract before discussing another roadmap change. The missing answers will show you where execution is actually slowing down. Fix that mechanism, run the loop, and let evidence determine where the system should expand next.

    References

    • Shivam.Consulting Blog — The CPO Playbook I Wish I’d Had: Ditch Bad Wisdom, Ship Faster, and Lead with Clarity
  • Ship MVPs in Days, Not Months: My Proven Prompt Prototyping Playbook for Product Teams

    Ship MVPs in Days, Not Months: My Proven Prompt Prototyping Playbook for Product Teams

    Most MVPs take too long, cost too much, and still miss the mark. Over the past year, I’ve shifted my team to a prototyping prompts approach that lets us validate problem-solution fit in days, not months. The result is faster learning loops, clearer tradeoffs, and a dramatically higher hit rate on features that actually move the needle.

    When I say prototyping prompts, I mean structured, layered instructions that guide gen ai systems to produce the right artifacts at the right fidelity. Instead of jumping straight to code, we generate concise problem briefs, user stories, interaction flows, low-fidelity UI descriptions, and test plans. Each pass is constrained by acceptance criteria and business outcomes, which keeps the work grounded in value rather than output.

    Here’s the playbook my product trios use to go from idea to a testable MVP in 48–72 hours. First, we anchor on outcomes vs output OKRs and clarify the customer job-to-be-done using evidence from customer interviews and support data. This is classic continuous discovery, but we compress it by focusing on the single riskiest assumption to de-risk this week.

    Second, we build a prompt scaffold. We specify the role, constraints, target users, success metrics, and the exact output format we expect. We also define evaluation upfront, borrowing from eval-driven development. For example, before any generation, we list the acceptance tests that a good solution must pass, including edge cases and compliance considerations. This discipline keeps hallucinations in check and improves repeatability.

    Third, we spin up multiple prototypes in parallel. One prompt generates a lean product brief; another outlines user flows; a third proposes UI states and error handling. If we’re exploring voice, we add prompt engineering for voice to script dialogs and repair strategies. For data-heavy features, we call out retrieval-first pipeline patterns so the model references source-of-truth data rather than guessing.

    Fourth, we validate with real users using the lightest-weight experiment possible. Fake-door tests, concierge workflows, and guided click-throughs let us measure intent before we invest. Where we can, we run quick A/B testing and size the effort using minimum detectable effect (MDE) so we don’t over- or under-sample. The point isn’t perfection; it’s fast, directional signal to inform the next iteration.

    Fifth, we instrument and ship behind feature flags. We track activation, task completion, and time-to-value from day one. On the delivery side, we watch DORA metrics and deployment frequency to ensure we’re learning continuously rather than batching big bets. This bridges discovery and delivery so roadmaps reflect real-world feedback, not assumptions.

    One recent example: we needed to evaluate a voice AI agent for appointment scheduling. In 72 hours, prompts produced the problem brief, dialog flows, error recovery strategies, and a sandbox to simulate inbound requests across three user personas. We exposed a thin slice to a pilot cohort, captured call outcomes, and iterated the repair prompts twice before writing any production code. The pilot converted at a higher rate than our control flow and gave us the confidence to invest in full integration.

    This approach only works if we treat governance as a first-class concern. We bake in privacy-by-design, clear data governance boundaries, and AI risk management from the start. Prompts include guardrails on personally identifiable information, explicit constraints on data use, and links to approved sources. We also maintain a prompt repository with versioning and automated evaluations so changes are observable and reversible.

    Practically, strong prompt scaffolds share three traits. They’re specific about context and constraints, they define success in measurable terms, and they separate concerns by artifact type. I’ll often ask for three variants with different tradeoffs, then run a quick synthesis prompt that highlights points of parity and differentiation. This gives the team structured options rather than a single, brittle path.

    If you’re starting from zero, begin with one high-leverage workflow. Write a crisp outcome statement, draft your acceptance tests, and create a prompt that outputs a one-page brief, three user flows, and the top five risks with mitigations. Validate with five users in 48 hours, then decide: double down, pivot, or park. Rinse and repeat, and your product roadmapping and sprint planning will shift from speculation to evidence.

    The bottom line is simple. Prototyping prompts won’t replace product judgment, but they will accelerate it. By turning ideas into testable artifacts in hours, you minimize waste, maximize learning, and ship better MVPs—fast.


    Inspired by this post on Product School.


    Book a consult png image
  • The Safety of Speed: 180 Deploys a Day, 12‑Minute Releases, 99.8%+ Availability

    The Safety of Speed: 180 Deploys a Day, 12‑Minute Releases, 99.8%+ Availability

    “Speed is not the enemy of safety; it is the prerequisite for it.” I live by this principle. In our organization, the average time from merging code to it being used by customers in production is just 12 minutes, and that short window is fundamental to how we build, ship, and learn.

    In January 2026, we are averaging 180 ships per workday – roughly 20 deployments every hour. Conventional wisdom suggests that to increase stability, you must slow down. I believe the opposite. Speed is not the enemy of safety; it is the prerequisite for it. Accumulating code creates risk; shipping small batches minimizes it. Shipping is our company’s heartbeat.

    Maintaining this frequency while targeting 99.8+% availability has required over a decade of focused investment in systems, principles, and processes. We protect the integrity of our systems through three layers of defense: an automated pipeline that is simple, reliable, and removes the need for manual intervention, a shipping workflow that promotes ownership and uses guardrails as accelerants, and a recovery model that optimizes for mitigating inevitable failures. Here’s how we’ve built each layer so that velocity is our greatest source of stability.

    While our platform consists of various services and frontend applications, I’ll focus here on our Ruby on Rails monolith. It is our core application and the one we deploy most frequently; we also deploy it to three different data‑hosting regions with independent pipelines. Our other services follow similar pipeline principles and safeguards, but the Rails monolith is the clearest example of how we ship at scale.

    The automated pipeline is designed to move code from merge to production as fast as possible while enforcing strict safety checks. It is fully automated, and the vast majority of releases require no human intervention—critical for CI/CD at high deployment frequency.

    Once an engineer merges code to GitHub, two things happen immediately. First, the build: we compile the Rails application and its dependencies into a deployable asset (a slug) in about four minutes. Second, parallel CI: our test suite runs alongside the build; through extensive optimization, parallelization, and test selection, the vast majority of CI builds finish in under five minutes.

    As soon as the slug is built, it’s deployed to a pre‑production environment. CI does not block the progression of the slug to pre‑production. Deploying to pre‑production takes around two minutes. This environment serves no customer traffic, but it is connected to our production datastores, mirrors our production infrastructure variants (e.g., web serving, asynchronous worker), and is configured so that requests exercise the pre‑release code and workers.

    Immediately after deployment, we run and await several automated approval gates. We verify that the application boots cleanly on hosts (boot test), confirm the parallel test suite passed (CI check), and execute functional synthetics using Datadog Synthetics on critical flows—such as loading or editing a Fin workflow. If any gate fails, the release is halted and does not go to production.

    Once approved, we promote the code to thousands of large virtual machines. A deployment orchestrator triggers these deployments simultaneously, while a decentralized, staggered rollout avoids changing the state of the entire fleet at the same millisecond. Within each machine, a rolling restart mechanism removes a process with old code from the serving path, lets it drain gracefully, and replaces it with a fresh process running the new code. From the moment a deployment starts, first requests are served by new code within roughly two minutes, and the vast majority of the global fleet updates transparently within six minutes. When restarts trigger on every machine, production unblocks so the next deployment can begin.

    We treat a stalled pipeline as a high‑priority incident. If the automated system rejects three consecutive release attempts, it pages an on‑call engineer. These are pre‑production blocks, but if the shipping lane stops moving, changes pile up—and our stability relies on building and shipping in small steps. The on‑call’s job is to restore flow so that tiny, safe, frequent updates continue to keep risk low.

    Our shipping workflow is built on extreme ownership: tools assist, but the engineer is accountable for quality and the decision to merge. I insist that you are present when you ship. The practical benefit of a 12‑minute deployment cycle is that engineers remain in the zone, focused on the problem they just solved, and ready to validate behavior as it goes live.

    Stylized rocket launch piercing dramatic cumulus clouds at sunrise, glowing vapor trail symbolizing fast yet controlled delivery; overlaid headline text reads 'The safety of speed' in the sky.
    A rocket lifts into a luminous sky, a metaphor for shipping code fast without breaking things, where precision, automation, and guardrails power 180 safe deployments a day.

    To support this, our deployment system sends Slack notifications the moment code is submitted and as it advances through stages, embeds direct observability links to relevant dashboards and logs in every PR and message, and prompts verification so engineers actively watch the dials and test features in production. It is not acceptable to rely on green builds. You’re expected to watch your change go live and if you’re not prepared to rollback, you’re not prepared to ship. We maintain a no‑blame culture: quick rollbacks and immediate reverts are signs of vigilance and ownership, not failure.

    We make extensive use of feature flags to turn deployment into a non‑event. By decoupling deployment (moving code to servers) from release (turning features on), we shrink the blast radius of change. Flags can be enabled for all customers, a specific subset, or disabled for everyone in under 60 seconds through our backend UI. Engineers can group flags into beta features and run phased rollouts; we also ensure flags work consistently across non‑monolith applications. In the past three months, we created over 560 flags—and we actively manage them to avoid permanent complexity.

    For complex refactors—especially when behavior should not change—we leverage GitHub Scientist, an open‑source experimentation library. It runs candidate logic (new code) in parallel with existing logic (old code) in production, instruments both paths for result and timing comparisons, and keeps existing behavior user‑visible. That means we can iterate on and validate new code under real load without risking the experience, then switch seamlessly when confident.

    When engineers need to go deeper before merging, they can generate a slug and deploy it to a virtual machine, detaching a running production host from the serving path and connecting for manual testing. They can also put a pre‑release slug on a serving machine that handles a small percentage of jobs or web requests. Single‑host validation lets us slice observability to those hosts, compare against the main release, and make low‑level changes safer. Staging is a simulation; production is reality. Testing on a single production host validates assumptions with real‑world data without risking the fleet.

    Our recovery model starts from a simple principle: stop monitoring systems; start monitoring outcomes. Traditional monitoring tells you if a server is healthy; we care whether customers are healthy. We rely on heartbeat metrics—vital signs that represent the core value our product provides—such as the rate at which messages and comments are created.

    Unlike standard uptime checks, heartbeat metrics are binary in spirit. If message send rates dip below baseline, it does not matter if infrastructure dashboards are green. Down is down, and if customers can’t do their job, uptime percentages are irrelevant. By tracking real‑world success rates as a high‑level signal, we catch subtle degradations that traditional alerting either misses or over‑alerts on.

    Because we ship in small, incremental steps and maintain previous releases on our virtual machines, our Time to Recover (TTR) is generally very fast. If a heartbeat metric drops or a critical anomaly is detected right after a ship, the system can trigger an automatic rollback, reverting to the release that was running 20 minutes ago—often restoring service before an engineer responds. For complex issues, engineers can initiate a manual rollback through our deployment UI; doing so also locks the production pipeline to prevent further releases while we investigate and remove problematic code.

    Resumption of service is not the end. Every incident prompts an incident review, and we don’t just fix the bug. We ask, “How did the machine allow this to happen?” Then we harden the system so it cannot happen again. This loop—fast shipping, fast recovery, rigorous learning—compounds resilience over time.

    This operating model aligns to DORA metrics: high deployment frequency, short lead time for changes, low change failure rate, and rapid time to restore service. It’s a CI/CD and SRE‑informed approach that converts speed into a defensive advantage rather than a liability.

    Shipping 180 times a day isn’t a vanity metric; it’s a deliberate choice to protect the customer experience. With a 12‑minute window from code to customer, the feedback loop is tight and engineers retain context—and accountability—for the immediate impact of their work. Maintaining this pace requires more than fast CI; it requires judgment, extreme ownership, disciplined use of feature flags, and a recovery model that monitors outcomes. We rely on human expertise, augmented by these layers of defense, to catch issues before they turn into customer pain. We don’t ship fast despite our need for stability; we ship fast to stay in control of change.


    Inspired by this post on The Intercom Blog.


    Book a consult png image
  • How Fast Is Fast Enough? Turn Deployment Frequency into a Durable Competitive Advantage

    How Fast Is Fast Enough? Turn Deployment Frequency into a Durable Competitive Advantage

    Every product leader I know wrestles with the same question: how fast is fast enough when it comes to shipping? Over the years, I’ve learned that deployment frequency isn’t just a DevOps vanity metric—it’s a direct lever on customer value, risk, and competitive advantage.

    When I talk about deployment frequency, I mean how often a team puts code into production, per service or product, in a given time period. It sits alongside lead time for changes, change failure rate, and mean time to recovery (MTTR) as part of the DORA metrics—together, they tell a coherent story about delivery performance and reliability.

    If you’re looking for a compass, here’s how I calibrate expectations. Elite teams deploy on demand—often multiple times per day—because they’ve engineered safety into their CI/CD pipeline and decoupled deploy from release. High-performing teams comfortably ship daily to weekly. Medium performers land in the weekly-to-monthly range. These bands aren’t moral judgments; they’re context-aware guideposts. The goal isn’t to copy someone else’s speed, but to reach the fastest sustainable cadence your business, architecture, and risk profile can support.

    So what does “fast enough” look like in practice? It depends on your product’s blast radius, regulatory constraints, and architecture. Microservice-heavy platforms with strong automated testing, feature flags, and progressive delivery generally sustain higher cadences with lower risk. Monoliths and highly coupled systems can still move quickly, but they need disciplined trunk-based development, robust test pyramids, and strong release controls to avoid brittle deployments.

    At HighLevel, we’ve moved products from a cautious weekly train to safe daily (and eventually on-demand) deploys without increasing incident volume. The breakthrough wasn’t a single tool—it was a system: smaller batch sizes, automated tests that actually fail when they should, immutable artifacts, canary releases, and feature flags that decouple deployment from exposure. The result was faster learning loops, fewer late surprises, and more predictable delivery.

    If you’re not measuring deployment frequency yet, start simple. Instrument your CI/CD pipeline or GitOps tooling to count production deployments by service each day. Normalize for rollbacks and re-deploys to avoid inflating the metric. Visualize by team and product area so you can spot bottlenecks and trend improvements over time. Pair it with change failure rate and MTTR to ensure you’re not trading speed for stability.

    Once you’ve got a baseline, focus on the levers that actually move the needle. Reduce batch size by merging smaller, well-scoped changes. Embrace trunk-based development to minimize long-lived branches. Accelerate feedback with fast, reliable unit and integration tests, contract testing for services, and ephemeral environments for preview. Use feature flags to control exposure, and progressive delivery (canary, blue-green) to verify in production safely. Automate change approvals where policy allows, and replace heavyweight gates with observable, auditable pipelines.

    Watch out for common anti-patterns. Batching several unrelated features into a single deploy increases risk and slows learning. Heroic “release nights” mask systemic issues. Friday deploy bans are a smell; if you can’t safely deploy on Friday, you can’t safely deploy any day—invest in recovery speed and blast-radius controls instead. And never treat deployment frequency as a target in isolation; it’s only healthy when reliability improves or holds steady.

    For strategy alignment, I tie deployment goals to outcomes, not outputs. If your objective is time-to-value or activation improvement, a higher cadence of small, measurable changes aligns perfectly. If your objective is stability for a major seasonal event, slow the cadence temporarily and increase release controls. The point is to let business outcomes set the tempo while engineering creates the conditions for safe speed.

    Here’s a pragmatic 30-day plan I’ve used with teams: Week 1, baseline deployment frequency and map your current release process end-to-end. Week 2, choose two services and cut batch size in half while enabling feature flags for new code paths. Week 3, refactor the pipeline for faster test feedback and add canary or blue-green for one critical service. Week 4, publish a dashboard that shows deployment frequency alongside change failure rate and MTTR, and run a retrospective to decide the next bottleneck to remove.

    Culturally, celebrate small, frequent, reversible changes. Reward teams for boring deploys, rapid recovery, and high-quality instrumentation. Build psychological safety around rollback and kill switches—confidence breeds cadence.

    Track deployment frequency, optimize it, and watch delivery speed turn into a competitive edge. Explore how in this article!

    Fast enough isn’t a number you copy; it’s a capability you build. When deployment frequency rises in tandem with reliability, you unlock faster learning, happier customers, and a durable advantage in your market.


    Inspired by this post on Product School.


    Book a consult png image