algorithms.live
    AI-ENGINEERING
    intermediate

    Prompt Management & LLMOps

    AI Engineer path · lesson 95 of 124
    The idea
    A prompt is code that runs inside the model. Ship it like code: PRs, tests (evals), a staging environment, a canary to a small share of users, and a rollback button. The alternative — editing prompts in production consoles — is how outages happen.
    How to use
    Pick a version, promote to canary, and watch the live quality bar update. Try promoting v4 and see the regression gate fire.
    Versions
    v1 — original
    eval 72%$900ms
    v2 — added examples
    eval 84%$$1050ms
    v3 — schema + citation
    eval 91%$$1080ms
    PROD
    v4 — shorter, sharper
    eval 68%$720ms
    Regression gate
    baseline = 80% gate | live eval shown in color
    91%
    live eval
    Try promoting v4
    v4 is shorter and cheaper but its eval is 0.68 — below the 0.80 baseline. The gate blocks the promotion; that is exactly what "CI for prompts" is meant to do.

    Key insight

    Treat prompts like code: version, review, evaluate, promote through environments, and gate on regression. The reason production incidents come from "someone edited a prompt in the console" is that consoles have no gate.

    Ship a prompt change safely

    Sequence

    Order the lifecycle of a prompt edit.

    Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

    1. 1Drop a step here
    2. 2Drop a step here
    3. 3Drop a step here
    4. 4Drop a step here
    5. 5Drop a step here
    6. 6Drop a step here
    6 pieces left to place

    Test your knowledge

    20 questions
    Question 1 of 20
    Score: 0

    What is the single strongest reason NOT to edit prompts directly in a production admin console?

    The short version

    A prompt is code that runs inside the model. Ship it like code: PRs, tests (evals), a staging environment, a canary to a small share of users, and a rollback button. The alternative — editing prompts in production consoles — is how outages happen.

    In more depth

    Prompt management & LLMOps

    A prompt is code that runs inside a model. Ship it with the same discipline.

    Registry + versioning

    A prompt registry (Langfuse, LangSmith, PromptLayer, or a home-grown store) is a versioned artifact store. Apps fetch prompt_v_current("customer_support_answer") at boot and cache; promoting a version is a registry mutation, not a code deploy.

    Eval gate

    Every promotion runs a labeled eval suite. Below the baseline score, promotion is blocked. Per-slice reporting (segment, template, question type) is essential — averages hide regressions.

    Canary / A-B rollout

    Canary a small fraction of live traffic to the new version, watch live KPIs (latency, cost, thumbs-down, error rate, cascade escalation rate), and complete rollout only if the canary is winning. A-B for decision-making between two variants.

    Feedback loop

    Failing eval samples and thumbs-down production traces flow back into the labeled dataset. Each version is measured against the failure modes of the last.

    Composes with everything else

    Observability emits the prompt version on every generation span — traces become attributable to a rollout.
    Prompt caching works because promotions do not change the byte-exact prefix of every call.
    Evals feed the gate.

    Interview Questions

    Anthropic
    Databricks
    Langfuse
    OpenAI
    Anthropic
    Meta
    Databricks
    Cohere
    Microsoft
    Anthropic
    OpenAI

    References & Further Reading

    Prompt Management and Prompt Engineering — TutorialYouTube
    video
    Open Source Prompt ManagementLangfuse Docs
    docs
    Get started with Prompt ManagementLangfuse Docs
    docs
    phoenixtype/ai-engineering-interview-prepGitHub
    docs
    All AI Engineering topics

    Love this visualization?

    Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.

    100% free • No credit card required