Agent Deployment Strategies: Canary, Blue-Green, and Progressive Rollouts for Agent Pipelines

July 24, 2026 · Advanced

30-Second Takeaway

The core problem: Traditional software deployment assumes “same input → same output”—agents don’t work that way. Model updates introduce non-deterministic behavioral drift, tool API changes cause silent interface breakage, and model upgrades can double per-task token consumption without triggering a single traditional alert. These three risks are systematic challenges every production agent system must confront.
Three-layer deployment strategy: Canary release (5–10% traffic for semantic correctness validation) → Blue-green deployment (zero-downtime environment switching with session state and tool registry protection) → Progressive rollout (10%→25%→50%→100% traffic ladder, each stage gated by semantic quality, cost regression detection, and latency regression). These are not mutually exclusive options—they are three lines of defense in a single deployment safety net.
Agent-specific deployment thresholds: Generic DevOps deployment cares about HTTP status codes and CPU utilization. Agent deployment cares about entirely different health signals—output semantic quality scores, tool call success rates, per-task token consumption growth, and hallucination rate deltas. These are the real vital signs of an agent deployment.
What you’ll be able to do after reading: Design a complete deployment pipeline from canary to full rollout for your agent system—with a semantic equivalence evaluator, tool call smoke tests, cost regression detectors, and automated rollback triggers. Every strategy comes with reference YAML configuration templates you can adapt directly.

§1 Introduction: Why AI Agents Need Specialized Deployment Strategies

One night in April 2026, an e-commerce platform’s customer-service agent migrated from gpt-4-turbo to gpt-4o. The deployment team followed the standard Kubernetes rolling-update workflow—all pod health checks passed, HTTP probes returned 200, CPU and memory looked normal. Thirty minutes later, the operations team was flooded with complaints: the agent had started refusing refund requests, claiming “no eligible items found in your order”—when in fact every order met the refund policy.

What went wrong? The new model differed subtly from the old one in how it reasoned about the return window: the old model computed delivery date + 7 days, the new model incorrectly used order date + 7 days. This difference was completely invisible to traditional HTTP health checks. Every request returned 200 OK; response times were actually faster. Semantic correctness had collapsed without a single alert firing. This is the core dilemma of agent deployment: traditional software deployment assumes “same input → same output.” Agents do not satisfy that assumption.

The deployment equation has changed: Traditional: deploy(new_version) → verify(HTTP 200, CPU < 80%, latency < 500ms) → success. Agent system: deploy(new_model/tool/system_prompt) → verify(HTTP 200 and semantic score ≥ 0.95 and tool call success rate ≥ 99% and per-task token growth < 15% and token throughput ≥ baseline) → success. The latter has 3–4 additional verification axes, and each axis demands dedicated evaluation infrastructure.

Three Unique Risks in Agent Deployment

Traditional software deployment risk centers on a single dimension—code regression: does the new version introduce bugs? Is rollback straightforward? Agent systems layer three new dimensions on top of code—model reasoning, tool execution, and token economics—each introducing risks that traditional deployment pipelines cannot detect:

Behavioral Drift: Model updates—even unannounced minor version bumps from your provider—can change an agent’s reasoning path. Same prompt, same temperature parameter; from claude-sonnet-20250219 to claude-sonnet-20250601, the model’s “understanding” of the same refund policy may shift in subtle but fatal ways. This drift doesn’t produce error logs like a code bug—it happens silently and can only be captured through semantic-level comparative evaluation.
Tool API Incompatibility: An agent isn’t just an HTTP service—it’s a composite system of LLM inference engine + tool registry + prompt templates + context management strategy. A new agent version may use new tool definitions, updated function signatures, or different parameter schemas. If the new agent calls search_documents(query, top_k=10) but the tool endpoint only accepts search(query, limit), the agent’s function_call will produce perfectly valid JSON—then get rejected with a 400 Bad Request. The agent enters a retry loop, burning tokens on every attempt until the context window overflows with error messages.
Cost Regression: This is a risk category unique to agent deployment. A new model may be “better”—higher output quality, more accurate reasoning—while consuming far more tokens. If migrating from gpt-4o-mini to gpt-4o pushes average tokens per customer conversation from 1,200 to 3,800, and your system handles 5,000 conversations daily, your monthly API bill could jump from $5,000 to $16,000. Cost regression triggers no traditional monitoring alert—no error logs, no latency spikes, no crashes—only a larger invoice at month-end.

Three-Layer Deployment Strategy: Canary → Blue-Green → Progressive

These three risks cannot be addressed by a single deployment pattern. Behavioral drift requires small-traffic semantic validation (testing the agent’s actual reasoning quality against a trickle of real requests)—exactly what canary releases are designed for. Interface breakage requires full parallel-environment comparison (running old and new versions against the same tool set and comparing outputs side by side)—blue-green deployments provide this capability. Cost regression demands gradual traffic amplification with continuous metric observation—progressive rollouts are purpose-built for this traffic ladder.

The table below maps each strategy to the risk it addresses:

Deployment Strategy	Primary Protection	Traffic Share	Decision Trigger
Canary Release	Behavioral drift detection	5–10%	Semantic score below threshold → auto-rollback
Blue-Green Deployment	Interface breakage guard + state consistency	0%→100% (switch)	Environment smoke tests pass → route switch
Progressive Rollout	Cost regression + latency regression monitoring	10%→25%→50%→100%	Each stage’s metric thresholds hold → advance to next stage

These three strategies are not mutually exclusive—they form a deployment safety net, from tight to loose. Canary sits at the front line, using minimal traffic to detect the most dangerous semantic degradation. Blue-green sits in the middle, ensuring the new version’s complete environment can be verified and reverted at any point. Progressive rollout sits at the final ring, using a stepped traffic ladder to surface cost and performance issues that only emerge at scale. For rollback mechanism design after a deployment failure, see Agent Rollback Design—this article focuses on the deployment safeguards before rollback becomes necessary.

§2 Canary Deployment: Minimum-Viable Agent Behavior Validation

Canary deployment takes its name from the practice of coal miners carrying canaries underground—the bird is more sensitive to toxic gas and shows symptoms before humans do, giving miners time to evacuate. In agent deployment, the “toxic gas” is semantic degradation: route 5–10% of real traffic to the new agent version, using those requests as an early-warning system to detect reasoning-quality collapse before it reaches all users.

The key difference between an agent canary and a microservice canary is the verification target. A microservice canary verifies “is the JSON structure correct?” An agent canary verifies “is the JSON content semantically correct?” The former is structural validation—completable in under a millisecond. The latter is semantic evaluation—requiring an evaluator model to perform pairwise comparison, potentially taking seconds per sample.

Canary ≠ phased rollout: A phased rollout expands scope by user dimension (“10% of users see the new version”). A canary release samples by request (“randomly route 10% of requests to the new version”). In the agent context, canary is better for behavioral consistency verification—consecutive requests from the same user may be split across old and new versions, enabling direct output comparison. A phased rollout is better for user experience verification—each user consistently sees one version. This article’s canary strategy refers to the former.

Agent-Specific Validation Dimensions for Canary

A traditional canary needs only three metrics: HTTP error rate, latency, and resource usage. An agent canary needs at least five additional dimensions on top of those:

Output Semantic Quality: Every canary request is sent to both the baseline (old) and canary (new) agent instances. Both outputs are fed to an evaluator model, which returns a 0–1 semantic equivalence score. This is not simple string similarity—the evaluator must understand whether the outputs achieve the same intent. “Your order will arrive on March 15th” and “The package will be delivered to you on 2026-03-15” share roughly 40% string similarity but are semantically equivalent. The evaluator must recognize that equivalence.
Tool Call Consistency: For the same user request, did the old and new agents invoke the same tools? With matching parameters? If the old agent called get_order(order_id=38291) and the new agent called search_orders(query="38291"), the end result might be identical but the tool path has changed materially—this should be flagged during canary.
Reasoning Step Count: The number of reasoning steps (LLM invocations + tool calls) is the primary cost driver for an agent task. If the new agent averages 5.7 steps versus the baseline’s 3.2, per-task cost has nearly doubled—even if output quality is unchanged. The canary stage should block progression to wider traffic.
Hallucination Rate: Is the new model more prone to hallucination? Measurement requires dedicated evaluation—checking, against known-answer test cases, whether the agent’s output contains information not present in the context. Canary traffic can estimate the relative change in hallucination rate through sampling + spot-checking.
Token Throughput: The rate at which the model produces tokens directly impacts user-perceived responsiveness. If throughput drops from 45 tokens/s to 18 tokens/s, users will feel noticeable “lag”—even if the final output is correct.

Of these five dimensions, semantic quality and tool call consistency are hard gates (fail → auto-rollback). Reasoning step count, hallucination rate, and token throughput are degradation warnings (exceed threshold → alert on-call, human decides whether to proceed).

Canary Deployment YAML Configuration

Below is a complete canary deployment configuration template covering traffic control, semantic evaluation, and rollback conditions:

# canary_deploy.yaml — Agent Canary Deployment Configuration
# Routes 8% of traffic to the new agent version, validates behavioral
# consistency via semantic evaluation and tool call comparison.
# Any hard condition failure triggers automatic rollback within 15 minutes.

deployment:
  name: "customer-service-agent-v2.3-canary"
  pipeline_id: "agent-deploy-20260724-001"
  timestamp: "2026-07-24T14:30:00Z"

# ─── Canary Traffic Control ───
canary_traffic:
  routing_strategy: "request_sampling"     # request_sampling | user_hash | session_sticky
  canary_ratio: 0.08                       # 8% of traffic to new agent
  baseline_ratio: 0.92                     # 92% stays on old agent
  sampling_seed: 20260724                  # Deterministic seed for reproducibility
  max_canary_duration_minutes: 60          # Auto-terminate canary after 60 minutes
  shadow_traffic: true                     # Mirror an extra 5% to new agent without returning results

# ─── New Agent Version ───
canary_version:
  model: "gpt-4o-2024-05-13"
  model_temperature: 0.3
  system_prompt_version: "v2.3"
  tool_registry_version: "tools-v4.1"
  context_window_max_tokens: 128000
  agent_config_uri: "s3://agent-configs/customer-service/v2.3.0.yaml"

# ─── Baseline (Old) Agent Version ───
baseline_version:
  model: "gpt-4-turbo-2024-04-09"
  model_temperature: 0.3
  system_prompt_version: "v2.2"
  tool_registry_version: "tools-v4.0"
  context_window_max_tokens: 128000

# ─── Semantic Evaluation Configuration ───
evaluation:
  evaluator_model: "claude-sonnet-20250601"  # Independent evaluator, not one of the agents under test
  semantic_equivalence_threshold: 0.95       # Score ≥ 0.95 to pass
  tool_call_match_threshold: 0.99            # Tool call match rate ≥ 99%
  hallucination_delta_threshold: 0.02        # Hallucination rate increase ≤ 2 percentage points
  reasoning_steps_ratio_max: 1.25            # Reasoning step increase ≤ 25%
  token_throughput_ratio_min: 0.70           # Token throughput drop ≤ 30%

  eval_prompt: |
    You are evaluating two agent responses for semantic equivalence.
    Score from 0.0 (completely different outcome) to 1.0 (identical outcome).

    User Request: {user_request}
    Baseline Response (old): {baseline_output}
    Canary Response (new): {canary_output}

    Scoring criteria:
    - 1.0: Same conclusion, same reasoning, same tool calls
    - 0.8-0.9: Same conclusion, slightly different reasoning or wording
    - 0.5-0.7: Similar direction but different details or tool usage
    - 0.2-0.4: Different approach, possibly different outcome
    - 0.0-0.1: Contradictory conclusions or completely different outcome

    Return ONLY a JSON object: {"score": float, "reason": "string"}

# ─── Rollback Conditions ───
rollback:
  auto_rollback: true

  # Hard conditions: any one triggers automatic rollback
  hard_conditions:
    - metric: "semantic_equivalence_score"
      threshold: 0.95
      operator: "lt"                         # less than — semantic score < 0.95
      min_samples: 50                        # Require at least 50 canary samples
      window_minutes: 5                      # Rolling time window

    - metric: "tool_call_match_rate"
      threshold: 0.99
      operator: "lt"
      min_samples: 50

    - metric: "http_error_rate"
      threshold: 0.01                        # HTTP error rate > 1%
      operator: "gt"

    - metric: "hallucination_rate_delta"
      threshold: 0.02                        # Hallucination rate increase > 2pp
      operator: "gt"
      min_samples: 30

  # Soft conditions: alert on-call but do not auto-rollback
  soft_conditions:
    - metric: "avg_reasoning_steps_ratio"
      threshold: 1.25                        # Reasoning steps increase > 25%
      operator: "gt"
      action: "alert_oncall"

    - metric: "token_throughput_ratio"
      threshold: 0.70                        # Token throughput drop > 30%
      operator: "lt"
      action: "alert_oncall"

  # Rollback execution: switch 100% traffic back to baseline
  rollback_action:
    type: "traffic_switch"
    target: "baseline_version"
    drain_canary_connections: true
    notification_channels:
      - "slack:#agent-deploy-alerts"
      - "pagerduty:agent-oncall"

# ─── Observation Window & Data Collection ───
observation:
  metrics_export:
    - type: "prometheus"
      endpoint: "pushgateway.agent.internal:9091"
    - type: "s3_log"
      bucket: "agent-canary-logs"
      prefix: "canary-20260724-001/"

  sampling:
    request_log_sample_rate: 1.0             # 100% of canary requests logged
    response_log_sample_rate: 1.0
    max_stored_samples: 10000

The core philosophy behind this configuration: a canary release is not “let some traffic through and see what happens.” It is a complete experiment—with an explicit hypothesis (the new agent’s semantic behavior matches the old), quantified success criteria (semantic score ≥ 0.95, tool call match rate ≥ 99%), and pre-set termination conditions (any hard metric breaches threshold → auto-rollback).

How the Semantic Evaluator Works

The semantic evaluator is the most critical component in canary deployment. Here is how it operates:

Every request routed to the canary is sent simultaneously to both baseline_version and canary_version agent instances. Each instance uses its own model, system prompt, and tool registry to complete reasoning independently.
Both outputs are fed into the evaluator_model (note: the evaluator must be a third model, independent of both the old and new versions—using either as the evaluator would introduce bias). The evaluator scores semantic equivalence on a 0–1 scale per the criteria defined in eval_prompt.
All scores are aggregated over a 5-minute rolling window. If the average semantic equivalence score across 50+ samples falls below 0.95, automatic rollback is triggered.
Scoring logs are written to Prometheus and S3 for post-incident analysis. Every rollback carries a complete scoring history—which requests caused the score drop, whether the pattern is systemic (all requests declined) or localized (specific request types).

Why 0.95 and not 0.99? In production, old and new agent outputs will never be perfectly identical. Even unannounced minor model updates (provider-side hot-swaps) can introduce 1–3% micro-drift. Setting the threshold at 0.99 would cause false-positive rollbacks on nearly every canary. 0.95 is a battle-tested practical threshold—it tolerates minor phrasing differences while catching substantive reasoning changes. For safety-critical agents (financial transactions, medical diagnosis), raise it to 0.98. For low-risk informational agents, 0.92 is a workable choice.

Canary Failure Modes and Resilience Pairing

Canary deployment itself can fail—the evaluator model becomes unavailable, the canary agent instance OOMs because the new model demands more resources, or canary traffic sampling introduces selection bias. These failure modes require a paired resilience layer. Within the broader agent resilience framework, the canary evaluation pipeline should be protected by a circuit breaker (stop scoring when the evaluator model fails consecutively, rather than letting the canary run unprotected), and canary agent instances should be isolated with a bulkhead (prevent the canary’s OOM from affecting production traffic). For the complete design of these resilience patterns, see Agent Resilience Patterns.

When the canary’s semantic score breaches threshold and triggers auto-rollback, the problem isn’t solved—you still need to understand why it failed. Was it the model itself, a system prompt change, or a tool registry difference? Rollback stops the bleeding; root-cause analysis requires the audit and diagnostic capabilities covered in rollback design. For post-rollback diagnostics and root-cause tracing, see Agent Rollback Design.

§3 Blue-Green Deployment: Zero-Downtime Agent Version Switching

Canary release answers the question “does the new version behave like the old?” but it leaves another critical question unanswered: when the new agent version must replace the old across all traffic, how do you ensure the switch itself doesn’t introduce failure? This is the design goal of blue-green deployment—maintain two complete agent environments (blue = current production, green = new version awaiting promotion), and perform the transition with a single routing change for zero-downtime, zero-session-interruption version switching.

In traditional microservices, blue-green is relatively straightforward: blue and green each run an independent set of pods; switching means updating the load balancer’s backend pool. But in an agent system, each “environment” is far more complex than “a set of pods.” Every agent environment includes:

LLM Provider connection pool: API keys targeting specific model versions, rate-limit configurations, fallback provider lists.
Tool registry: The complete set of tool definitions, function signatures, endpoint URLs, auth credentials, and timeout configurations. The tool registry version (e.g., tools-v4.1) determines which tools the agent can call and how.
Prompt template library: Versioned system prompts, task templates, and few-shot examples.
Memory and state store: User session histories, long-term memory vector databases, conversation summary caches.
Evaluation and monitoring pipeline: Semantic evaluator configuration, metric export endpoints, alerting rules.

If a blue-green switch changes only the route without synchronizing these components, the post-switch environment enters a “half-blue, half-green” inconsistent state—and that is fatal for an agent. For example, the green environment uses the new system prompt system-prompt-v3, but the tool registry is still the old tools-v4.0—the new prompt references a function name that only exists in tools-v4.1. The agent fails on its first tool call and enters a retry loop.

Three-Phase Agent Blue-Green Deployment Process

Agent blue-green deployment is broken into three phases: environment preparation → smoke test → traffic switch. Each phase has checkpoints specific to agent risks.

Phase 1: Green Environment Full Startup

The green environment is not “start a new set of pods.” It is an independent instantiation of the complete agent runtime. The startup sequence must include:

# blue_green_deploy.yaml — Agent Blue-Green Deployment Configuration

deployment:
  name: "customer-service-agent-blue-green"
  strategy: "blue_green"

environments:
  # ─── Blue Environment (Current Production) ───
  blue:
    label: "production-current"
    model: "gpt-4-turbo-2024-04-09"
    system_prompt_version: "v2.2"
    tool_registry_version: "tools-v4.0"
    memory_store: "redis-cluster-blue.internal"
    vector_store: "pinecone-index-blue"
    metrics_namespace: "agent/blue"
    health_endpoint: "/healthz"

  # ─── Green Environment (Staged for Promotion) ───
  green:
    label: "production-next"
    model: "gpt-4o-2024-05-13"
    system_prompt_version: "v2.3"
    tool_registry_version: "tools-v4.1"
    memory_store: "redis-cluster-green.internal"    # Separate Redis instance
    vector_store: "pinecone-index-green"             # Separate vector index
    metrics_namespace: "agent/green"
    health_endpoint: "/healthz"

    # Startup checks that must pass before green is considered ready
    startup_checks:
      - name: "llm_connectivity"
        type: "api_call"
        endpoint: "/v1/chat/completions"
        expected_status: 200
        timeout_seconds: 10

      - name: "tool_registry_load"
        type: "config_validation"
        # Verify all tool endpoints are reachable and schemas match
        tools_to_validate: ["get_order", "search_knowledge_base",
                            "send_email", "create_ticket", "cancel_order"]

      - name: "memory_store_ping"
        type: "redis_ping"
        target: "redis-cluster-green.internal"

      - name: "vector_store_connectivity"
        type: "pinecone_check"
        index: "pinecone-index-green"

      - name: "prompt_template_compile"
        type: "template_check"
        # Verify variable placeholders are consistent across all templates
        templates: ["system_prompt_v2.3", "greeting_v3", "escalation_v3"]

# ─── Configuration Schema Versioning ───
config_versioning:
  schema_version: "2.3.0"
  compatible_schema_versions: ["2.2.0", "2.3.0"]
  # Refuse the switch if green's schema_version is incompatible with blue's
  enforce_schema_compatibility: true

Configuration schema versioning is critical: An agent’s configuration structure evolves—new tools are added, old tools deprecated, prompt template variables renamed. The config schema must carry a version tag. If green uses schema_version: 3.0 while blue uses schema_version: 2.0, and the two are incompatible, a post-switch rollback (back to blue) would leave historically accumulated v3.0-format data that blue’s v2.0 schema cannot parse. Schema compatibility checking is the last line of defense preventing “can’t go back” scenarios in blue-green deployments.

For environment isolation safety—including network policy separation, resource quota isolation, and independent credential management between blue and green environments—see the complete runtime isolation design in Agent Runtime Isolation.

Phase 2: Agent Blue-Green Smoke Test

Once the green environment is fully started, and before any production traffic is switched, a dedicated agent smoke test suite must execute. The fundamental difference from a traditional smoke test: it doesn’t just verify that green “works”—it verifies that green and blue produce consistent results for the same set of inputs.

The smoke test has three layers:

Layer	Test Content	Sample Size	Pass Criteria
L1 — Endpoint Connectivity	LLM API, tool endpoints, memory store, vector store all reachable	1 per endpoint	100% success
L2 — Tool Call Consistency	Same inputs, blue vs. green tool call sequences match	50 predefined test cases	≥ 98% tool call match rate
L3 — End-to-End Semantic Consistency	Blue and green final outputs for identical requests are semantically equivalent	50 predefined test cases	Semantic score ≥ 0.95

# smoke_test.yaml — Blue-Green Deployment Smoke Test Configuration
smoke_test:
  enabled: true
  test_suite: "agent-blue-green-smoke-v2"
  timeout_minutes: 10

  layers:
    l1_connectivity:
      parallel: true
      checks:
        - target: "green.llm_endpoint"
          method: "POST"
          payload: {"model": "gpt-4o-2024-05-13", "messages": [{"role": "user", "content": "ping"}], "max_tokens": 5}
          expect: {"status": 200}
        - target: "green.tools.get_order"
          method: "GET"
          path: "/api/v1/orders/health"
          expect: {"status": 200}

    l2_tool_call_consistency:
      test_cases_source: "s3://agent-test-cases/smoke-v2/l2-tool-calls.jsonl"
      sample_count: 50
      match_criteria:
        tool_name_match: true              # Tool names must match exactly
        argument_structure_match: true     # Argument structure must match (minor value differences tolerated)
      pass_threshold: 0.98                 # At least 98% of test cases must match

    l3_semantic_equivalence:
      test_cases_source: "s3://agent-test-cases/smoke-v2/l3-end-to-end.jsonl"
      sample_count: 50
      evaluator_model: "claude-sonnet-20250601"
      semantic_threshold: 0.95
      pass_threshold: 0.90                 # At least 90% of test cases score ≥ 0.95

  # Behavior on failure
  on_failure: "abort_deployment"           # Abort, do not switch traffic
  on_success: "proceed_to_traffic_switch"

L2 tool call consistency is the most undervalued but critical part of blue-green smoke testing. An agent may phrase the same answer differently while remaining semantically equivalent (L3 tolerates this), but a difference in tool calls means the reasoning path has changed—even if the final result is the same, a different tool path could mean performance change (an extra slow external API call), cost change (one more round of LLM inference tokens), or permission change (invoking a tool that shouldn’t have been invoked). Tool call reliability directly impacts both agent cost and security.

Phase 3: Traffic Switch and State Migration

After phases 1 and 2 pass, execute the traffic switch—point the load balancer from the blue environment to the green environment. In an agent system, this switch involves three parallel actions:

Request routing switch: All new agent requests are directed to green. This step is identical to traditional blue-green.
Session state migration: User sessions already in progress on blue (active sessions) have two handling strategies:
- Drain strategy: Blue continues processing its existing active sessions until they naturally end; all new sessions go to green. Suitable when average session duration is short (< 2 minutes).
- Migrate strategy: Active session state (conversation history, context summaries, intermediate tool call results) is migrated from blue’s memory store to green’s memory store. Suitable for longer-duration sessions (5+ minutes), but requires schema compatibility between blue and green memory stores.
Tool registry hot-swap: If green uses a new tool registry version (tools-v4.1), old-version tool endpoints must remain available for blue’s draining sessions until blue traffic reaches zero. You cannot shut down blue’s tool endpoints immediately upon switching—that would instantly kill every still-draining blue session.

# traffic_switch.yaml — Traffic Switch Configuration
traffic_switch:
  strategy: "instantaneous"               # instantaneous | gradual_drain
  connection_draining_seconds: 300        # Blue active connections survive up to 5 minutes

  session_state_handling:
    mode: "drain"                         # drain | migrate
    max_drain_duration_seconds: 600       # Maximum drain wait: 10 minutes
    # For migrate mode:
    # mode: migrate
    # migration_batch_size: 100           # Migrate 100 sessions per batch
    # migration_source: "redis-cluster-blue.internal"
    # migration_target: "redis-cluster-green.internal"
    # schema_version_check: true          # Check schema compatibility before migration

  tool_endpoint_lifecycle:
    # Tool endpoints must not be torn down until blue traffic reaches zero
    blue_tool_drain_policy: "wait_for_zero_connections"
    blue_tool_max_keepalive_seconds: 900   # Maximum keepalive: 15 minutes
    green_tool_warmup_seconds: 30          # Warm up green tool endpoints for 30 seconds

  rollback_trigger:
    # Within 5 minutes after switch, auto-revert to blue if:
    observation_window_minutes: 5
    auto_rollback_on:
      - metric: "http_error_rate"
        threshold: 0.05                   # Error rate > 5%
      - metric: "tool_call_failure_rate"
        threshold: 0.03                   # Tool call failure rate > 3%
      - metric: "session_abandon_rate"
        threshold: 0.10                   # Session abandon rate > 10%

  notifications:
    on_switch_start: ["slack:#agent-deploy-info"]
    on_switch_complete: ["slack:#agent-deploy-info"]
    on_rollback: ["slack:#agent-deploy-alerts", "pagerduty:agent-oncall"]

§4 Progressive Rollouts: Traffic Ladder Based on Performance and Cost

Canary release uses 5–10% traffic to validate semantic correctness. Blue-green deployment ensures environment completeness and reversibility. But one question remains: some agent degradations only surface under significant traffic volume. Cost regression, latency regression, and long-tail semantic errors—these won’t appear in a canary’s 50 samples or a blue-green smoke test’s 50 test cases. They need hours of continuous operation across thousands of real requests before they emerge.

Progressive rollout is the traffic ladder designed for exactly this class of degradation. It breaks full deployment into four stages—10% → 25% → 50% → 100%—with a preset observation window between each stage (typically 30 minutes to 2 hours). Traffic only advances to the next rung when all metrics for the current stage are satisfied.

Three Core Progressive Rollout Triggers

Unlike traditional service progressive rollouts (which only check latency and error rate), agent progressive rollouts require three orthogonal evaluation axes:

Axis 1: Semantic Quality Sustainment

A canary’s semantic score is based on 50+ samples—with a wide confidence interval. Each progressive rollout stage accumulates larger sample sizes (10% → ~500 samples/hour; 25% → ~1,250 samples/hour), enabling more precise semantic quality measurement. The key insight: semantic quality must remain stable as volume increases—it should not “regress to a lower mean” as sample size grows. If the canary stage (50 samples) scored 0.96 but the 10% traffic stage (500 samples) drops to 0.91, the canary samples likely suffered from selection bias (random sampling happened to draw “easy” requests). Halt the progressive rollout and roll back.

Axis 2: Cost Regression Detection

This is the most distinctive axis in agent progressive rollouts. Cost regression isn’t just “total spend increased”—it must be measured precisely as per-task token consumption. The following metrics form the cost regression detection matrix:

Per-task token growth: Take the full token consumption of a single agent task (LLM inference tokens + tool call tokens + evaluation tokens), grouped by task type. If the new agent’s average tokens for customer inquiry tasks rise from 1,200 to 1,700 (a 41.7% increase), even if output quality improves, this is a cost regression that demands a decision. Recommended threshold: per-task token growth ≤ 15%.
LLM invocations per task: Does the new agent perform more LLM calls per task than the old? Each additional LLM call means more tokens and more latency. If average invocations rise from 3.2 to 5.1, total cost increases by roughly 60% even if per-call token consumption stays flat.
Tool call costs: Some tool calls are themselves metered (e.g., external address verification APIs, credit score query APIs). Does the new agent invoke more of these billable tools?

Axis 3: Latency Regression

A new model may be faster or slower. The issue isn’t absolute latency—it’s the latency distribution, especially p95 and p99. If the new agent’s average latency improves from 2.3s to 1.8s, but p95 latency worsens from 4.5s to 9.7s (more than doubling), the new model has a severe tail-latency problem for certain request types. Recommended threshold: p95 latency ≤ 2× baseline p95.

Progressive Rollout YAML Configuration

# progressive_rollout.yaml — Agent Progressive Rollout Configuration
# Gradually lifts new agent traffic from 10% to 100%, with each stage
# gated by semantic quality, cost deviation, and latency regression.

progressive_rollout:
  enabled: true
  total_duration_hours_max: 8              # Hard cap on total rollout time
  auto_advance: true                       # Auto-advance to next stage if gates pass
  on_stage_failure: "rollback_to_previous"

  stages:
    # ─── Stage 1: 10% Traffic ───
    - stage: 1
      traffic_ratio: 0.10
      observation_window_minutes: 30       # Observe for 30 minutes
      gates:
        semantic_quality:
          metric: "semantic_equivalence_score"
          threshold: 0.95
          operator: "gte"
          min_samples: 200                 # Need ≥ 200 samples (10% × 30min ≈ 250)

        cost_regression:
          metric: "avg_tokens_per_task_ratio"
          # new per-task tokens / old per-task tokens
          threshold: 1.15                  # Growth ≤ 15%
          operator: "lte"
          baseline_source: "pre_deploy_baseline_7d"  # 7-day pre-deployment baseline

        latency_regression:
          metric: "p95_latency_ratio"
          threshold: 2.0                   # p95 latency ≤ 2.0× baseline
          operator: "lte"

    # ─── Stage 2: 25% Traffic ───
    - stage: 2
      traffic_ratio: 0.25
      observation_window_minutes: 45
      gates:
        semantic_quality:
          metric: "semantic_equivalence_score"
          threshold: 0.95
          operator: "gte"
          min_samples: 500

        cost_regression:
          metric: "avg_tokens_per_task_ratio"
          threshold: 1.12                  # Tighten to 12% (larger samples → more precision)
          operator: "lte"

        latency_regression:
          metric: "p95_latency_ratio"
          threshold: 2.0
          operator: "lte"

        # Stage 2 addition: tool call success rate
        tool_call_success:
          metric: "tool_call_success_rate"
          threshold: 0.995                 # Tool call success rate ≥ 99.5%
          operator: "gte"

    # ─── Stage 3: 50% Traffic ───
    - stage: 3
      traffic_ratio: 0.50
      observation_window_minutes: 60
      gates:
        semantic_quality:
          metric: "semantic_equivalence_score"
          threshold: 0.94                  # Allow slight drop at scale (sample diversity increases)
          operator: "gte"
          min_samples: 1000

        cost_regression:
          metric: "avg_tokens_per_task_ratio"
          threshold: 1.10
          operator: "lte"

        latency_regression:
          metric: "p95_latency_ratio"
          threshold: 1.8                   # Tighten to 1.8× (less tail latency expected at scale)
          operator: "lte"

        tool_call_success:
          metric: "tool_call_success_rate"
          threshold: 0.995
          operator: "gte"

        # Stage 3 addition: hallucination rate comparison
        hallucination_rate:
          metric: "hallucination_rate_delta"
          threshold: 0.015                 # Hallucination rate increase ≤ 1.5pp
          operator: "lte"

    # ─── Stage 4: 100% Traffic (Full Rollout) ───
    - stage: 4
      traffic_ratio: 1.0
      observation_window_minutes: 120      # Observe for 2 hours at full scale
      gates:
        cost_regression:
          metric: "avg_tokens_per_task_ratio"
          threshold: 1.10
          operator: "lte"

        latency_regression:
          metric: "p95_latency_ratio"
          threshold: 1.5                   # At full scale, latency should approach baseline
          operator: "lte"

        tool_call_success:
          metric: "tool_call_success_rate"
          threshold: 0.995
          operator: "gte"

        # At full scale, stop continuous semantic scoring (too expensive)
        # and switch to daily offline evaluation instead.
        offline_evaluation:
          enabled: true
          frequency: "daily"
          semantic_threshold: 0.95

  # ─── Fine-Grained Cost Monitoring ───
  cost_monitoring:
    # Token statistics broken down by task type
    task_type_breakdown: true
    task_types:
      - "customer_inquiry"
      - "order_tracking"
      - "refund_request"
      - "product_recommendation"
      - "complaint_escalation"

    # Cost regression alert thresholds (pause advance, do not rollback)
    cost_alerts:
      - metric: "avg_tokens_per_task_ratio"
        threshold: 1.25                    # Growth > 25% → alert immediately
        action: "alert_oncall_and_pause_advance"
      - metric: "daily_cost_estimate_ratio"
        threshold: 1.30                    # Estimated daily cost increase > 30%
        action: "alert_oncall"

    # Token exclusions
    token_exclusions:
      - "evaluation_tokens"                # Exclude evaluation tokens (not agent task cost)
      - "health_check_tokens"              # Exclude health check tokens

  # ─── Observation Dashboard ───
  dashboard:
    prometheus_queries:
      semantic_score: |
        avg(agent_semantic_score{version="canary"}) /
        avg(agent_semantic_score{version="baseline"})

      tokens_per_task: |
        sum(rate(agent_llm_tokens_total{version="canary"}[5m])) /
        sum(rate(agent_task_completed_total{version="canary"}[5m]))

      p95_latency: |
        histogram_quantile(0.95,
          sum(rate(agent_task_duration_seconds_bucket{version="canary"}[5m]))
          by (le))

    grafana_dashboard_uid: "agent-progressive-rollout"
    alertmanager_route: "agent-deploy-oncall"

Engineering Cost Regression Detection

Cost regression isn’t detected by looking at your monthly cloud bill—by then you’re 30 days late. Cost regression must be detected with real-time per-task token comparison at every stage of the progressive rollout. Here is the implementation path:

Token instrumentation: In the agent runtime’s LLM call wrapper layer, record prompt_tokens, completion_tokens, and total_tokens for every call. Write these as Prometheus Counter metrics labeled with agent_id, task_type, and version.
Baseline establishment: Before starting the progressive rollout, extract the past 7 days’ median and p95 per-task token consumption from the old agent’s Prometheus data. These become the comparison baseline for every stage.
Real-time comparison: During each progressive rollout observation window, continuously compute avg_tokens_per_task_ratio = canary_avg_tokens / baseline_avg_tokens. If the rolling average of this ratio at the end of the observation window exceeds the stage’s threshold (1.15 for stage 1, 1.12 for stage 2, etc.), pause traffic advance.
Cost estimation: Multiply per-task tokens by the LLM provider’s public pricing to generate a daily_cost_estimate—not a precise invoice amount, but a sufficiently accurate directional signal. If the new agent’s estimated daily cost exceeds the old’s by 30%, even with perfect semantic quality, the team must decide: “Is the improved output worth a 30% cost increase?”

For the complete cost observability design—including token budget management, multi-provider cost comparison, and cost anomaly detection—see Agent Cost Observability. Progressive rollout cost detection is a deployment-time cost snapshot; it depends on the real-time token metrics provided by the cost observability infrastructure.

The silence of cost regression: Among all agent deployment risks, cost regression is the easiest to overlook—it triggers no PagerDuty alerts, generates no user complaints, produces no error logs. Its only symptom is a larger number on the month-end invoice. Precisely because of this silence, cost regression detection must be hard-coded into progressive rollout stage gates—it is not an optional “recommendation” but a hard condition every stage must satisfy. If your system doesn’t yet have per-task token instrumentation, then every deployment decision you make is being made without cost information—equivalent to driving blindfolded.

Progressive Rollout and Observability

Every stage gate in a progressive rollout depends on real-time observability data—semantic scores, token consumption, p95 latency, tool call success rates. If the data pipeline feeding these metrics experiences lag or interruption during a progressive rollout, the rollout enters a dangerous state: traffic has advanced to the next stage, but the gate’s decision basis is stale or missing.

Two mechanisms solve this:

Data Freshness Guard: In each stage gate’s evaluation logic, check not only the metric value but also the metric’s data freshness. If Prometheus’s latest data point is more than 2 minutes stale (e.g., because the metrics pipeline has failed), the stage gate must enter a “waiting” state rather than “pass” or “fail”—when data is absent, you do not make advance decisions.
Observability degradation strategy: If the entire metrics pipeline becomes unavailable, the progressive rollout should pause traffic advancement and hold the current ratio—no more, no less. This is the safety default baked into the progressive rollout configuration. For the complete observability pipeline design and alerting strategy, see Agent Observability Design.

§5 Agent Deployment Risk Monitoring Matrix

Canary, blue-green, and progressive rollouts each address specific deployment phases. But what ties them together is a unified risk monitoring matrix — a dashboard that tracks agent-specific degradation signals across every deployment stage.

Model Drift Detection

Model drift in agent systems differs from traditional ML model drift in one critical way: the "model" can change without any deployment on your side. When a provider silently updates their model (a "hot update"), your agent's behavior may shift overnight with zero configuration changes on your end.

Detection strategy: Maintain a continuous canary — a permanent 5% traffic shadow that runs the new model version side-by-side with the current production version, comparing their semantic outputs in real time. If the semantic equivalence score between the canary and baseline drops below 0.92 for more than 10 minutes, trigger a deployment freeze and alert on-call. See Agent Resilience Patterns for fault tolerance mechanisms that complement this detection.

Tool API Compatibility Monitoring

Agent deployments aren't just about the LLM model — they also involve tool definitions, API schemas, and parameter contracts. A new agent version might change a tool's input schema, making the deployment incompatible with existing tool servers.

Monitoring approach: Before committing to any deployment stage, run a tool compatibility smoke test that invokes every registered tool with canonical inputs and compares results against the baseline. Track tool_call_success_rate and tool_output_schema_match_rate as deployment gate metrics.

Cost Regression Monitoring

As detailed in §4, cost regression detection operates at every progressive rollout stage through per-task token comparison. The risk monitoring matrix adds a longitudinal view: tracking token consumption trends across multiple deployments to detect gradual cost creep — the "boiling frog" scenario where each deployment adds 3-5% more tokens and nobody notices until costs have doubled over six months.

For the complete cost observability framework, see Agent Cost Observability.

State Consistency Validation

If your agent maintains conversational state, session memory, or long-term user profiles, a deployment that changes the state schema can corrupt existing sessions. Blue-green deployment partially mitigates this by keeping both environments intact, but the switch itself must validate that the new environment can read and correctly interpret state written by the old environment.

Agent Deployment Risk Monitoring Matrix

Risk Category	Detection Metric	Hard Threshold	Soft Threshold (Alert Only)	Deployment Stage
Model Drift	semantic_equivalence_score	< 0.92	< 0.95	Canary, Progressive
Tool API Breakage	tool_call_success_rate	< 0.97	< 0.995	Blue-Green, Progressive
Cost Regression	avg_tokens_per_task_ratio	> 1.25	> 1.15	Progressive (all stages)
State Corruption	session_state_integrity_score	< 0.99	< 1.0	Blue-Green
Hallucination Spike	hallucination_rate_delta	> 0.03	> 0.02	Canary, Progressive Stage 3+
Latency Regression	p95_latency_ratio	> 2.5	> 2.0	Progressive (all stages)
Token Throughput Drop	token_throughput_ratio	< 0.50	< 0.70	Progressive Stage 2+

The matrix distinguishes between hard thresholds (trigger automatic rollback) and soft thresholds (trigger alerts without automatic rollback). Hard thresholds protect against catastrophic failures; soft thresholds provide early warning without disrupting the deployment. Calibrate these values against your agent's historical baseline — the numbers above are starting points, not universal constants.

§6 Configuration Architecture: Responsibility Chain from Research to VERIFIED

Deployment configurations — the YAML files defining canary ratios, blue-green environments, and progressive rollout stages — don't exist in isolation. They are artifacts that flow through the same gate chain as the agent code itself: generated during Research, validated during QA, and executed during Deploy.

# deploy_config.yaml — Schema versioning and gate integration
# This config is generated by the Research stage, validated by QA,
# and executed by the Deploy gate.

deploy_config:
  schema_version: "2.0"       # Bump when adding new gate fields
  generated_by: "research"
  validated_by: "qa"
  executed_by: "deploy_gate"

  # Gate chain integration
  gate_chain:
    research_gate:
      action: "generate_deploy_config"
      output: "deploy_config_v2.yaml"
    
    qa_gate:
      action: "validate_deploy_config"
      checks:
        - "schema_version_is_current"
        - "all_thresholds_within_bounds"
        - "rollback_conditions_are_exhaustive"
        - "observation_windows_are_positive"
    
    deploy_gate:
      action: "execute_deploy_config"
      preconditions:
        - "ready_gate_passed"
        - "user_approval_received"
        - "canary_test_passed"

Deployment Configuration Maturity Model

Maturity Level	Config State	Canary	Blue-Green	Progressive	Auto-Rollback
L1 — Research	Generated by research stage	Basic ratio only	—	—	Manual
L2 — Validated	QA gate passed, schema versioned	Full 5-axis evaluation	Smoke test enabled	—	Semi-auto
L3 — Staged	Passed canary smoke test	Production-tested	Traffic switch validated	Stage 1-2 enabled	Full auto for canary
L4 — VERIFIED	Full progressive rollout complete	Continuous monitoring	24h warm standby	All 4 stages active	Full auto across all stages

Each maturity level represents a gate in the xslyl release chain. The deploy config graduates from L1→L2 after QA gate validation, L2→L3 after the canary smoke test passes, and L3→L4 after the VERIFIED gate confirms full progressive rollout completion. This maturity model is the operationalization of the gate architecture described in Agent Release Gate Design.

For the complete gate architecture and how deployment gates integrate into the release workflow, see Agent Release Gate Design.

§7 Practical Workflow: From Config Generation to Production Deployment

Here is the end-to-end deployment workflow, from the moment a new agent version passes QA to the moment it serves 100% of production traffic:

Brief → Config Generation: The research stage produces a deploy_config.yaml capturing the specific deployment strategy for this agent version. For a minor model update, this might be canary-only; for a major architecture change, it might be blue-green + progressive rollout.
Pre-deployment Checks: QA gate validates the deploy config — schema version, threshold bounds, rollback conditions, observation window durations.
Canary Test (5-10%): If semantic equivalence score ≥ 0.95 after 50+ samples → proceed. Otherwise → rollback.
Blue-Green Switch (if configured): Deploy new environment, run smoke tests, switch traffic, keep old environment warm for 24 hours.
Progressive Rollout (10%→25%→50%→100%): Each stage gated by semantic quality, cost regression, and latency regression. Any gate failure → rollback to previous stage.
Full Deploy + Monitoring: At 100% traffic, switch to daily offline semantic evaluation and continuous cost/latency monitoring.

Decision heuristic: Small team, low-risk agent → canary-only (steps 1-3). Medium team, production agent → canary + progressive rollout (steps 1-5). Large platform, high-availability requirement → full blue-green + canary + progressive (all steps).

Deployment Decision Flowchart

                    ┌──────────────────────┐
                    │  Agent Passes QA     │
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │  Canary Release 5-10% │
                    └──────────┬───────────┘
                               │
                         ┌─────┴──────┐
                         │ Semantic   │
                         │ Score OK?  │
                         └─────┬──────┘
                         ≥ 0.95 │       < 0.95
                               │          │
                               ▼          ▼
                    ┌──────────────┐  ┌──────────────┐
                    │ Blue-Green   │  │ AUTO-ROLLBACK│
                    │ Switch?      │  │ (if enabled) │
                    └──────┬───────┘  └──────────────┘
                           │
                    ┌──────┴──────┐
                    │ Required?   │
                    └──────┬──────┘
                     Yes   │   No
                      │    │    │
                      ▼    │    ▼
           ┌────────────┐ │  ┌───────────────────────┐
           │ Blue-Green │ │  │ Progressive Rollout    │
           │ Deploy +   │ │  │ 10% → 25% → 50% → 100%│
           │ Smoke Test │ │  │ (each stage gated by   │
           └─────┬──────┘ │  │  semantic/cost/latency)│
                 │        │  └───────────┬───────────┘
                 ▼        │              │
           ┌────────────┐ │    ┌─────────┴─────────┐
           │ Traffic    │ │    │ All Gates Pass?   │
           │ Switch     │◄┘    └─────────┬─────────┘
           └────────────┘         Yes   │   No
                                        │    │
                                        ▼    ▼
                              ┌──────────────┐  ┌──────────────┐
                              │ 100% Traffic │  │ ROLLBACK TO  │
                              │ + Continuous │  │ PREVIOUS     │
                              │ Monitoring   │  │ STAGE        │
                              └──────────────┘  └──────────────┘

§8 Comparison with Traditional Deployment: The Agent-Specific Challenge

Why can't we just use the same deployment patterns that have worked for decades of software engineering? Because agent systems introduce three dimensions of non-determinism that traditional deployment assumes away:

Dimension	Traditional Deployment	Agent Deployment
Output determinism	Same input → same output (code is deterministic)	Same input → potentially different output (LLM is non-deterministic)
Regression type	Code regression: function returns wrong result	Semantic regression: output is fluent but wrong in meaning
Rollback surface	Infrastructure rollback: revert binary, restart service	State rollback: must also revert session memory, agent state, tool registrations
Cost visibility	CPU/memory usage — visible in standard dashboards	Token consumption — requires custom instrumentation, not visible in standard infra metrics
Validation signal	HTTP 200 + integration tests pass = safe to deploy	Semantic evaluation score + tool call success rate + cost regression check = safe to deploy
Failure detection latency	Minutes (error rates spike immediately)	Hours to days (semantic degradation accumulates gradually)

Three Mindset Shifts for Agent Deployment

From "does it run?" to "does it think correctly?": Traditional deployment validates infrastructure health. Agent deployment must also validate cognitive health — semantic correctness, reasoning consistency, and output quality. These are not measured by HTTP status codes.
From "is it faster?" to "is it worth the cost?": A faster agent that consumes 3× more tokens per task is not an upgrade — it's a cost liability. Every deployment decision must weigh performance gains against token cost increases on a per-task basis.
From "rollback the binary" to "rollback the mind": Rolling back an agent isn't just reverting code — it's also reverting prompt templates, tool registrations, model versions, and potentially session state. A rollback must be atomic across all these layers, or the agent will operate in an inconsistent state.

The core insight: traditional deployment assumes that "the system works if the infrastructure works." Agent deployment must also validate that "the system thinks correctly" — a semantic layer of verification that traditional tooling doesn't provide.

§9 Conclusion and Best Practices

Agent deployment is not a single strategy — it's a layered safety net where each pattern addresses a different class of risk:

Deployment Strategy Selection Guide

Scenario	Recommended Combination	Rationale
Small team, prototype stage, low-risk agent	Canary (5%) → Manual confirmation → Full deploy	Only requires semantic evaluation infrastructure; cost-controllable; simple process
Medium team, production agent, moderate risk	Canary (10%) → Progressive rollout (10%→50%→100%)	Combines semantic validation with cost/performance monitoring; blue-green may be too heavy for medium teams
Large platform, high availability, high-risk agent	Blue-green → Canary (10% within green) → Progressive rollout (10%→25%→50%→100%)	Complete three-layer safety net: environment isolation → semantic validation → cost/performance monitoring
Provider-side model hot update (unannounced)	Canary (continuous, 5% permanent canary)	Provider updates are uncontrollable — a permanent canary detects silent model behavior drift immediately

Three Golden Rules

Always validate with traffic, never with test sets alone: Test sets are static snapshots. The 5-10% traffic in a canary release is the only signal that represents real-world behavior. Test set validation is necessary before deployment, not sufficient for it.
Cost is a first-class deployment gate: In traditional software deployment, cost isn't on the deployment checklist. In agent deployment, per-task token consumption must stand alongside semantic quality and latency as a decision criterion at every deployment stage. Moving cost regression detection from "check at month-end" to "check at every stage" is the hallmark of engineered agent deployment.
Every environment must be independently reversible: Blue-green's blue environment shouldn't be destroyed 5 minutes after switch-over. Progressive rollout's intermediate stages (10%, 25%, 50%) should maintain long enough observation windows to collect meaningful metrics, and short enough rollback paths. If the 25% stage triggers a cost regression alert, the system should revert to 10% — not to 0%.

As agent systems evolve toward multi-agent orchestration, deployment complexity compounds: deploying Agent A on Tuesday with a new model, Agent B on Wednesday with new tools, and Agent C on Thursday with a new reasoning chain — each deployment potentially breaking the contracts the other agents depend on. That future demands not just the deployment patterns described here, but a deployment orchestration layer that sequences agent deployments, validates cross-agent contracts, and rolls back entire agent groups atomically. But that's a topic for another article.

Frequently Asked Questions

What's the difference between canary deployment and A/B testing?

Canary deployment tests whether a new version is safe to roll out to everyone — it's a safety gate, not a comparison experiment. A/B testing compares two versions to see which performs better on a business metric (conversion rate, engagement). A canary asks "is the new version broken?"; A/B testing asks "which version is better?" Canary runs briefly (minutes to hours) with a small traffic fraction; A/B testing runs for days with statistically meaningful sample sizes. In agent systems, canary deployment validates semantic correctness; A/B testing would compare business outcomes like task completion rate or user satisfaction between two agent versions.

Isn't blue-green deployment too expensive for small teams?

For small teams, running two complete agent environments doubles infrastructure cost. The pragmatic approach: start with canary-only deployment. Only add blue-green when (a) your agent has persistent state that must survive deployment, (b) rollback must be near-instantaneous (seconds, not minutes), or (c) you need to validate the full environment (tools, databases, caches) before any production traffic hits it. Many production agents operate successfully with canary + progressive rollout only — blue-green is an option, not a requirement.

How long should each progressive rollout stage last?

The minimum is your observation window (typically 30-120 minutes per stage). The real answer depends on your request volume: you need enough samples for statistical significance. At 10% traffic with 1,000 requests/hour, you get ~100 samples in 30 minutes — enough for latency but marginal for semantic scoring. A practical rule: extend the observation window until you have ≥200 evaluated samples for semantic quality and ≥500 requests for latency/cost metrics. For low-traffic agents, this might mean several hours per stage.

How do I detect silent model drift from provider-side hot updates?

Run a permanent canary: keep 5% of traffic routed to a shadow evaluation pipeline that continuously compares the current model's outputs against a frozen baseline. If the semantic equivalence score between canary and baseline drops below your threshold for more than 10 consecutive minutes, the provider has likely updated the model. Trigger an alert, freeze all in-progress deployments, and investigate. This is one of the four scenarios in the §9 selection guide table.

What happens if the observability pipeline fails during a progressive rollout?

The Data Freshness Guard described in §4 and §5 handles this: each stage gate checks not only metric values but also metric freshness. If Prometheus data is more than 2 minutes stale, the gate enters a "waiting" state — it neither passes nor fails. The progressive rollout pauses at its current traffic ratio until the observability pipeline recovers. If the pipeline remains down beyond a configurable timeout (e.g., 30 minutes), the rollout should automatically rollback to the previous safe stage rather than continuing blind.

Next Steps

Agent Release Gate Design — understand how deployment strategies integrate into the complete release gate framework
Agent Rollback Design — automatic rollback mechanisms when deployment fails
Agent Resilience Patterns — fault tolerance and self-healing after deployment
Agent Cost Observability — real-time cost regression detection during deployment
Agent Observability Design — real-time metrics pipeline for deployment decisions