Reviewing AI-Generated Plans Against Engineering Constraints

Hello again. In the previous lesson, you assembled an evidence-first context packet: a bounded task, selected artifacts, explicit constraints, and visible unknowns. That packet is what makes an AI-generated plan reviewable rather than merely plausible.

Now comes the decision point before code is written. An implementation plan may name the right files and still be unsafe: it might violate an architectural boundary, test only a happy path, make a production regression hard to see, or assume that “Helm rollback” undoes state that it does not actually undo.

By the end of this lesson, you will be able to review an AI-generated implementation plan against four explicit constraint sets:

1. Architecture: boundaries, contracts, dependencies, data, and security invariants.
2. Testing: risks, test layers, assertions, and release gates.
3. Observability: the signals needed to detect and diagnose success or failure.
4. Rollback: reversibility, exposure controls, decision thresholds, and recovery verification.

---

Treat the plan as a change hypothesis

Claude Code’s plan mode is useful precisely because it separates exploration from modification. The plan is not a commitment to execute. It is a change hypothesis:

Given this evidence, these files, these changes, and these validation steps should achieve the desired outcome while preserving the stated constraints.

That hypothesis should be challenged before implementation, when changing direction is cheap.

A weak review asks, “Does this seem reasonable?” A strong review asks questions that could disprove the plan:

• Which existing system boundary does this change cross?
• What must remain true for every request, not just the demonstrated case?
• What production failure would this introduce or fail to detect?
• Which test demonstrates that the important failure modes are protected?
• If the change is harmful, what precisely is rolled back, by whom, and how is recovery verified?

A useful plan therefore has more than a file list. For each meaningful step, it should state:

Plan element	Reviewable form
Intent	What behavior changes, and why this is the smallest appropriate change
Affected boundary	API, event contract, persistence, Elasticsearch index, cache, Kubernetes configuration, or external dependency
Evidence	Ticket, test, ADR, trace, runbook, or code path supporting the step
Invariant	A property that must remain true, such as tenant isolation or backward compatibility
Validation	Specific test or observation, including expected result
Release and recovery	Exposure method, failure threshold, rollback action, and verification

“Update the search handler and add tests” is an implementation sketch. “Modify the query construction path while preserving server-side tenant filtering; add integration coverage against the search adapter; deploy progressively while monitoring p95 latency and tenant-filter failures” is a plan that can be reviewed.

---

Turn evidence into a constraint ledger

Your previous evidence packet may contain a ticket, code path, integration test, mapping history, traces, deployment data, and a runbook. Before reviewing the model’s plan, convert the relevant information into a short constraint ledger.

A constraint is not a preference such as “keep it clean.” It is a condition the implementation must satisfy. Each should have an authority and a means of verification.

ID	Constraint	Authority	Evidence that would satisfy it
A1	Tenant filtering remains enforced by the service, not a caller-provided claim.	Security requirement or ADR	Code-path review and cross-tenant integration test
A2	Existing clients retain their current response contract.	API contract and consumer tests	Contract tests or compatibility tests
T1	The reported regression has a targeted automated test.	Ticket acceptance criteria	A test that fails before the fix and passes after it
O1	Operators can distinguish cache behavior, application time, and Elasticsearch time.	Operational requirement	Metrics, trace spans, and structured events
R1	A harmful release can be stopped and reverted without data loss.	Runbook or release policy	Tested deployment procedure and post-rollback checks

The ledger prevents a common AI failure: it can optimize for the visible request while overlooking a constraint that was present in the supplied context but not repeated in the final ask.

Separate requirements from open questions

Not every uncertainty is a constraint. Mark it separately.

Classification	Example	Reviewer action
Hard constraint	Search results must remain tenant-isolated.	Reject any plan that weakens it.
Acceptance criterion	p95 latency must return below an agreed threshold.	Require a measurement and release gate.
Design preference	Reuse the established search-client abstraction.	Require justification if the plan bypasses it.
Unknown	Are requests repetitive enough for caching to help?	Require evidence before approving cache work.
Hypothesis	The alias change increased query cost.	Require investigation, not a causal claim.

This distinction matters when working quickly across repositories. A model should be allowed to identify an unknown and propose a read-only investigation step. It should not quietly convert that unknown into an implementation assumption.

---

Review with four lenses, not one generic checklist

A production plan must be assessed as a whole. A plan can pass unit tests yet fail operationally; it can include metrics but still be impossible to roll back safely. The four lenses below expose different classes of failure.

The Google SRE production-readiness perspective is a useful complement to code review because it considers architecture, instrumentation, emergency response, change management, and performance together.

1. Architecture: preserve boundaries and invariants

Architecture review asks whether the proposed design fits the system that actually exists.

For a .NET service backed by Elasticsearch and deployed to Kubernetes, inspect at least these concerns:

• Ownership and boundaries. Which component owns validation, authorization, query construction, caching, and retries? A plan should not move an authorization decision from a trusted server boundary to a client request merely to simplify an implementation.
• Contracts and compatibility. Does an HTTP response, event payload, index document, or configuration value change? If so, identify consumers and the compatibility strategy.
• Data semantics. Does the change affect freshness, ordering, completeness, idempotency, retention, or index mappings? These are product and reliability properties, not implementation details.
• Dependency behavior. Does the plan introduce a new remote call, alter timeouts, add retry behavior, or increase fan-out? A “small” code change may increase load on Elasticsearch or a downstream API.
• Infrastructure fit. If memory, CPU, connection count, configuration, secrets, or Helm values change, the plan must acknowledge the Kubernetes and deployment implications.

A plan naming the correct SearchHandler.cs file is not enough. It should explain why that handler is the correct enforcement point, what inputs it trusts, and what downstream behavior it preserves.

2. Testing: validate risks at the right layer

Testing is not “add unit tests.” A sound plan links each material risk to a test type and an expected assertion.

For a typical API and search change, review testing in this order:

1. Targeted unit tests validate pure decision logic: request normalization, cache-key construction, query composition, authorization checks, or retry classification.
2. Integration tests validate the service’s interaction with Elasticsearch, Redis, a database, or another service. They catch serialization, query semantics, mappings, configuration, and connection behavior that unit tests cannot.
3. Contract tests matter when the API response or event is consumed by independently deployed services.
4. End-to-end or smoke tests verify that the deployed application can use its infrastructure with realistic configuration.
5. Non-functional tests are needed where the change claims performance, resilience, or security benefits.

Do not require every category for every small change. Require tests that cover the risks introduced or modified.

A useful review comment is concrete:

The plan adds a cache-hit unit test, but it does not test that two tenants with the same normalized query cannot share an entry. Add an integration-level test that issues identical requests under two tenant identities and asserts distinct, authorized result sets.

That is much stronger than “please add more tests.”

3. Observability: make the change diagnosable in production

Observability is not a dashboard added after deployment. For a plan, it means answering:

If this behaves incorrectly at 2 a.m., can an operator establish what happened, to whom, and at which boundary?

Require the plan to state:

• Success signals: which business or technical outcomes should improve, such as request success rate, p95 latency, throughput, or cache hit rate.
• Failure signals: which errors or degraded outcomes should become visible, such as authorization denials, stale-data responses, dependency timeouts, fallback use, or validation failures.
• Correlation: how an inbound request can be linked to downstream Elasticsearch calls, cache operations, or external-service calls. In distributed services, trace context and request IDs are usually central here.
• Dimensions: which low-cardinality dimensions support diagnosis, such as route, operation, result category, dependency, deployment version, or environment. Do not use tenant IDs, raw queries, emails, tokens, or arbitrary user input as high-cardinality metric labels.
• Operational response: which dashboard, alert, or runbook applies when the relevant signal crosses its threshold.

For a latency-focused search plan, “add logging” is inadequate. A reviewable proposal could include:

• a request-duration histogram for the search endpoint;
• a dependency-duration span or metric for Elasticsearch;
• a counter separating cache hits, misses, and cache failures;
• structured events for fallback or timeout outcomes;
• deployment version attached to telemetry;
• a dashboard comparison of the canary against the current version.

The plan does not need to promise an alert for every new metric. It must ensure that an important new failure mode is not invisible.

4. Rollback: distinguish redeployment from recovery

“Roll back with Helm” is often incomplete. Rollback must consider four independent forms of state:

State involved	Why a simple application rollback may fail
Application code	An image or deployment revision is usually reversible.
Configuration	Feature flags, environment values, and secrets may remain changed after an image rollback.
Data or schema	A database migration, Elasticsearch mapping change, or index reindex can be irreversible or incompatible with older code.
External side effects	Sent events, emails, writes to third-party systems, and user-visible changes may require compensation rather than rollback.

A plan must name the relevant state and its recovery approach. If it says the change is reversible, ask: reversible to what exact prior state?

For deployment changes, require these five items:

1. Exposure strategy: all-at-once, canary, ring-based, feature flag, dark launch, or another bounded approach.
2. Decision threshold: the measurable condition that halts further exposure. For example, error rate, latency, correctness mismatch, or a business metric relative to baseline.
3. Authority: who or what may halt, roll back, or disable the feature.
4. Mechanism: the exact deploy revision, flag, configuration, or operational action that restores the prior behavior.
5. Verification: the post-rollback checks proving both user behavior and telemetry returned to an acceptable state.

Thresholds should come from the service’s SLOs, release policy, or historical baseline, not from a generic template. “Rollback if metrics worsen” is not a usable threshold.

---

A worked review: proposed cache for a slow search endpoint

Consider the search-latency evidence packet from the previous lesson. It established that filtered catalog searches are slow for large tenants, that tenant isolation is non-negotiable, and that the onset may correlate with an index-alias change. It did not establish that users make enough repeated requests for a cache to solve the problem.

Claude produces this plan:

1. Add an in-memory cache in SearchHandler with a fifteen-minute TTL.
2. Key entries by normalized query text and selected filters.
3. Add a unit test for cache hits and misses.
4. Add a Helm value for the TTL.
5. Deploy the new version to production and monitor latency.

The plan has an understandable intent, but it should be returned for revision. Here is how the four-lens review reveals why.

Lens	Finding	Severity	Required revision
Architecture	The key definition does not explicitly include tenant identity and any authorization scope that affects results.	Blocker	State the complete authorization-safe cache key and identify the trusted enforcement boundary.
Architecture	No evidence shows that request repetition is high enough for caching to improve p95 latency.	Major	First analyze trace data for query repetition, cardinality, and freshness requirements.
Testing	Hit/miss tests do not prove tenant separation, expiry behavior, or behavior when the cache fails.	Major	Add tests for cross-tenant isolation, expiry, bypass/failure behavior, and integration with the actual request path.
Observability	“Monitor latency” cannot distinguish a cache benefit from Elasticsearch behavior or a partial outage.	Major	Add request, cache, and Elasticsearch timing signals with trace correlation and a canary comparison view.
Rollback	The plan has no exposure stage, rollback trigger, or post-rollback verification.	Major	Specify a progressive release, threshold, disable mechanism, and recovery checks.
Rollback	In-memory cache state disappears per pod and produces non-uniform behavior during a rollout.	Question	Establish whether this is acceptable for freshness and consistency expectations.

Notice two important review habits:

• The reviewer does not prescribe Redis, a new index, or a different architecture without evidence. The appropriate conclusion may be that caching is not justified.
• A plan can be rejected without calling it bad. It is incomplete relative to known constraints and unknowns.

A revised first phase could be entirely read-only:

1. Measure repeated request shapes, cache eligibility, and required freshness from a bounded production sample.
2. Confirm whether the latency is application time, Elasticsearch query time, or infrastructure contention.
3. Compare slow traces with index-alias and mapping changes.
4. Decide whether query optimization, index work, or caching is the justified next design.

That is still progress. It avoids implementing a confident-looking but weakly evidenced fix.

---

Make the review efficient and repeatable

For complex work, use a short, explicit review request after Claude has produced a plan. Supply the plan and constraint ledger, then require a structured result.

Review this implementation plan against the supplied evidence and constraint ledger.

For every material plan step:
1. Cite the supporting evidence IDs.
2. Identify architecture invariants and dependencies affected.
3. Map each meaningful risk to a specific test and expected assertion.
4. Specify the telemetry needed to detect success and failure in production.
5. State rollout, stop, rollback, and verification requirements.

Classify findings as:
- Blocker: violates a hard constraint or makes recovery unsafe.
- Major: a material risk lacks validation, evidence, or operational control.
- Question: an unknown must be resolved before implementation.
- Minor: clarity or maintainability improvement.

Do not invent repository facts. If evidence is missing, state the smallest
read-only investigation needed to obtain it. Return a revised plan only after
listing findings and unresolved questions.

This prompt asks the model to critique its own proposal, but its output still needs engineering judgment. A practical review routine is:

1. Read the task and constraint ledger first. This prevents the plan’s confident wording from becoming the frame of reference.
2. Trace each planned change through the relevant boundary. Start from caller input and follow authorization, application logic, dependencies, and state.
3. Find the plan’s largest untested assumption. For performance work, this is often the causal mechanism; for API work, it is often compatibility; for Kubernetes changes, it may be runtime resource behavior.
4. Require release evidence. A production-impacting change needs defined signals and a recovery path, not only CI success.
5. Return findings in severity order. Focus attention on constraint violations and missing safety controls before naming style issues.
6. Approve only an executable, observable, recoverable plan. “Executable” includes the evidence needed to decide when not to proceed.

The CI/CD pipeline below illustrates why this review must occur before a change enters the delivery path: build and test outputs are valuable gates, but production deployment also needs a controlled transition, approval where appropriate, and a means to respond to live results.

---

Key takeaways

An AI-generated implementation plan is best reviewed as a falsifiable change hypothesis, not as a polished to-do list.

• Build a constraint ledger from authoritative evidence before reading the plan closely.
• Review plans through four independent lenses: architecture, testing, observability, and rollback.
• Require tests tied to risks and assertions, not generic claims that “coverage will be added.”
• Require operational signals that distinguish success, failure, and relevant dependency behavior.
• Treat rollback as recovery of code, configuration, data, and side effects; a deployment rollback alone may be insufficient.
• When evidence does not support an implementation choice, approve a bounded investigation step rather than a speculative fix.

Next, you will take an approved plan and partition it into dependency-aware tasks with clear ownership and explicit integration checkpoints.