Task-to-Model Routing Matrix: Reasoning, Context, Latency, and Cost Criteria

Welcome back. In the previous lesson, you established the measurements that matter for recurring AI-assisted work: active effort, lead time, rework, and confirmed defects. That baseline keeps model choice grounded in delivery outcomes rather than in a vague sense that one model “feels smarter” or “feels faster.”

This lesson turns that evidence into a task-to-model routing matrix: a small, explicit decision table for selecting an appropriate model tier for each kind of request. You will classify tasks by the reasoning they require, the amount of relevant context they need, their latency tolerance, and their cost constraints. The aim is not to force every task through an automated router immediately. It is to replace habitual model selection with a policy you can test and improve.

---

Routing is a policy, not a leaderboard

A routing matrix answers a practical question:

Given this request and its delivery constraints, what is the least expensive and fastest model that has evidence of meeting the required quality bar?

That wording matters. “Best model” is not an absolute property. A model that is excellent for tracing an authorization flaw through a .NET service, an Elasticsearch query, and a Kubernetes deployment configuration may be wasteful for converting a ticket into structured fields or summarizing a single PR diff.

A task-to-model matrix is useful in two settings:

• Interactive engineering work, such as choosing a model in an assistant or coding tool for planning, debugging, review, or implementation.
• Application workflows, such as routing high-volume extraction requests to an economical model while reserving a higher-capability model for complex incident triage.

At this stage, treat the matrix as a manual routing policy. It should guide your own AI usage first. Later, if an application has sufficiently variable traffic and a stable evaluation suite, the policy can become an automated routing component.

The central trade-off has four dimensions:

Dimension	What you are deciding	Common mistake
Reasoning	How much multi-step inference, ambiguity resolution, and trade-off analysis the task requires	Equating long output with deep reasoning
Context size	How much relevant evidence must be considered together	Sending the entire repository rather than a focused evidence set
Latency	How quickly a useful response is required in the actual workflow	Optimizing first-token speed while ignoring total completion time
Cost	What the request may cost at its expected volume and context size	Selecting solely on per-token price while ignoring rework and defects

The IBM Technology video offers a concise framing: model selection begins with the problem to solve, then considers capability, speed, and price.

A useful routing policy does not say “use the cheap model for simple things and the strong model for hard things.” Those labels are too vague to audit. Instead, it defines observable request properties and maps them to named model tiers.

---

Define model tiers before assigning tasks

Avoid tying your matrix to a particular vendor name at first. Provider model names, prices, context limits, and performance change frequently. Define three capability tiers in terms of the work they must perform, then map concrete model IDs to those tiers in your current environment.

Tier	Purpose	Typical reasoning fit	Typical use
E — Economy / fast	High-volume, bounded, low-risk work	Applies explicit rules; transforms, classifies, extracts, or summarizes focused input	Ticket tagging, structured extraction, concise summaries, boilerplate generation
B — Balanced	Default engineering assistant work	Follows a multi-part contract; connects a few files or pieces of evidence; produces a reviewable first draft	Focused implementation, test drafting, PR review, bounded debugging
P — Premium / capability	High-ambiguity or high-consequence work	Reconciles conflicting evidence, reasons across subsystems, compares alternatives, and makes assumptions explicit	Complex debugging, architecture plans, incident synthesis, cross-service refactoring analysis

These are not quality labels. An Economy-tier result can be excellent when the task is appropriately constrained. A Premium-tier response can still be wrong if its context is incomplete, contradictory, or misleading.

Before using the matrix, maintain a small model catalog. For each provider model you can actually call or select, record the properties that constrain routing.

Catalog field	Example of what to record
Model identifier	Exact provider model ID and version or dated alias
Assigned tier	E, B, or P based on your initial evaluation
Input-context limit	Usable input budget after allowing for instructions and output
Structured-output and tool support	Required, optional, or unavailable for your workflow
Observed latency	Measured median and slow-tail latency on representative requests
Current input and output pricing	Provider pricing captured with date and region if relevant
Evaluation evidence	Dataset or representative task sample, pass rate, and known failure modes

The last row is the most important. A vendor’s general capability description helps form a hypothesis, but your matrix should be justified by tasks like yours: .NET changes with repository conventions, TypeScript integrations, Elasticsearch investigation, and operational configuration work.

Anthropic’s model-selection guidance presents this same tiered idea: a faster economical model can be an appropriate starting point for common work, while more capable models are warranted for complex reasoning and long-running work.

---

Turn four vague criteria into usable scales

The matrix needs consistent labels. Otherwise “medium complexity” becomes whatever you happen to feel in the moment.

1. Required reasoning

Reasoning is about the structure of the decision, not the number of source files or lines of output.

Band	Definition	Engineering examples
R1 — Bounded transformation	The rules are explicit and the answer can be checked locally	Convert known JSON to a TypeScript interface; extract incident fields into a schema; summarize a single diff
R2 — Local synthesis	The model must connect a few related facts and follow local conventions	Draft tests for a changed handler; implement a well-specified Node endpoint; explain a focused stack trace with relevant code
R3 — Multi-source diagnosis	The task requires weighing evidence, tracing causes, or selecting among constrained alternatives	Diagnose an Elasticsearch relevance issue using query, mapping, and sample results; review a multi-file .NET change for regressions
R4 — System-level judgment	The task involves ambiguity, cross-boundary consequences, or decisions expensive to reverse	Plan a cross-service refactor; analyze an incident spanning application logs, cluster events, and deployment history

A request belongs in a higher band when any of these are true:

• The task has several plausible explanations or designs.
• Important requirements are implicit rather than stated.
• The model must reconcile conflicting evidence.
• A missed constraint could cause a security, reliability, or operational failure.
• The answer will guide a consequential decision rather than provide a draft for easy local verification.

A long Kubernetes YAML file does not automatically make a task R4. If you have a known template and a specific field to update, it may be R1 or R2. Conversely, a short request such as “why did this deployment degrade search latency?” can be R3 or R4 because the reasoning burden is high.

2. Context pressure

Context size means the amount of evidence that must be considered at the same time. It does not mean repository size.

Use context-pressure bands relative to the usable input capacity of the smallest candidate model in your routing pool:

Band	Share of usable input capacity	Interpretation
S — Small	Under 10%	One focused file, one ticket, a concise diff, or a short conversation
M — Medium	About 10% to 40%	A feature slice: ticket, instructions, several source files, relevant tests, and a small log sample
L — Large	About 40% to 70%	Several components, extensive operational evidence, or a long technical document set
XL — Excessive	Above 70%	A warning that selection, compression, or staged investigation is needed before asking for a final answer

Leave capacity for stable instructions and for the response. A request that nearly fills the context window has less room for the model to reason over newly supplied evidence and has more risk of truncation or missed details.

For example, “write tests for this OrderValidator change” should normally be S or M: the changed validator, its direct dependencies, existing test conventions, and the acceptance rule. Supplying all validation code in the monorepo makes the request more expensive without reliably improving it.

Azure’s architecture guidance explains why context requirements belong in routing decisions and why a router must make its constraints visible.

3. Latency tolerance

Latency is a workflow requirement, not a property of the task’s importance. Assign one of three labels:

Band	Workflow requirement	Examples
I — Interactive	The engineer or end user is waiting and needs a quick turn	Inline explanation, short extraction, autocomplete-adjacent assistance
W — Working session	A response can take longer while you inspect code, run tests, or read a plan	Implementation plan, PR review, bounded debugging
A — Asynchronous	The workflow can run in the background with status and cancellation support	Batch documentation drafts, repository analysis, incident evidence synthesis

Do not mark a task Interactive merely because you would prefer it faster. If a model needs careful reasoning for an R4 task, pretending it is latency-critical usually creates a poor compromise: an underpowered answer delivered quickly, followed by expensive human repair.

4. Cost constraint

Use cost bands that correspond to the economic role of the request:

Band	Appropriate when	Typical examples
C1 — Minimal	High request volume or low consequence per request	Bulk tagging, routine extraction, brief summaries
C2 — Controlled	Moderate volume and a reviewable engineering output	Test drafts, bounded code changes, routine reviews
C3 — Quality-first	Requests are less frequent or the cost of being wrong is high	Architecture analysis, complex diagnosis, high-risk review

Direct token cost belongs here, but it is not the only cost. If a low-cost choice creates a misleading implementation plan that causes a day of rework, it was not economical. Your baseline metrics provide the evidence to distinguish a cheap request from a cost-effective workflow.

---

Build your first routing matrix

Create one row per recurring request class, not one row per repository. The same task pattern may appear in both .NET and Node.js work, while Elasticsearch and Kubernetes tasks often deserve their own rows because they have distinct operational risk.

Here is a practical first version for your workload. The model tiers are deliberately generic; replace them with entries from your model catalog.

Request class	Required reasoning	Context pressure	Latency	Cost	Primary tier	Escalate when
Extract fields from a ticket, log line, or known document template	R1	S	I or A	C1	E	Input is ambiguous, fields conflict, or a validation failure occurs
Summarize a focused PR diff with stated review questions	R2	S or M	W	C2	B	The change crosses authorization, data integrity, or operational boundaries
Draft unit or integration tests for a bounded .NET or Node.js change	R2	M	W	C2	B	Existing tests are inconsistent, behavior spans services, or failures are nonlocal
Implement a well-specified endpoint or integration change	R2	M	W	C2	B	The task exposes unstated architecture choices or touches critical behavior
Explain a localized build, test, or runtime failure with supporting evidence	R2	S or M	W	C2	B	Several hypotheses remain plausible after the first evidence pass
Diagnose Elasticsearch relevance, mapping, or query behavior	R3	M or L	W	C2 or C3	B, evaluated carefully	Query, mapping, analyzer behavior, and production evidence disagree
Review a cross-cutting .NET change for correctness and operational risk	R3	M or L	W	C3	P	Evidence is incomplete; stop for clarification rather than guessing
Analyze a K8s-related incident from alerts, logs, manifests, and rollout history	R3 or R4	L	A or W	C3	P	Context is XL, authorization is missing, or the required operational evidence is unavailable
Develop a plan for a multi-service refactor	R4	L	A	C3	P	Constraints are ambiguous; obtain decisions before implementation begins
Bulk classify or route support issues with a stable schema	R1	S	A	C1	E	Confidence falls below your validated threshold or the schema cannot be satisfied

This table contains two important design choices.

First, there is no “always Premium” row. Even critical work benefits from decomposition. A Premium-tier model may synthesize the incident analysis, while lower-tier calls perform independently verifiable subtasks such as extracting timestamps, grouping similar error messages, or normalizing ticket fields.

Second, the escalation column is part of the routing policy. A model should not be escalated because its answer is aesthetically unsatisfying. Escalate based on evidence: an unsatisfied schema, a missing required citation, an unresolved conflict, a failed test, an answer that cannot identify its assumptions, or a task whose actual reasoning/context band exceeds the original classification.

The routing pattern looks like this in an application architecture:

For a personal workflow, the “router” can simply be your matrix and a deliberate choice in your AI tool. For a production feature, it may become a service component. In both cases, the policy should remain inspectable: you need to know which model was selected, why, and whether that choice met the task’s measured quality and latency requirements.

---

Add hard gates before cost optimization

The four matrix dimensions rank eligible models. They do not replace non-negotiable constraints.

Before selecting a tier, apply these gates:

1. Data and authorization gate: Is the model and hosting arrangement approved for this client’s data? Do not route proprietary code, logs, secrets, or customer data to a model merely because it is cheaper or more capable.
2. Capability gate: Does the candidate support the required input type, structured output, tools, or other interface features?
3. Context gate: Can the candidate accept the selected evidence with room for instructions and an answer?
4. Quality gate: Has the candidate met a defined acceptance threshold for this request class?
5. Latency gate: Can it meet the workflow’s measured latency target at the expected load?

Only then choose the lowest-cost eligible tier.

This ordering avoids a common mistake: starting with price and trying to rationalize exceptions afterward. The cheapest model is never eligible for a task it cannot safely receive, cannot fit into context, or cannot perform to the quality threshold.

---

Make the matrix testable with your baseline

Your initial matrix is a hypothesis. Validate it using the task log you began in the previous lesson.

For each recurring task class, add these fields:

Field	Why it matters
Matrix row ID	Links the task to a specific routing decision
Selected model and tier	Makes results comparable after provider changes
Primary or escalated route	Shows whether the initial classification was accurate
Actual context-pressure band	Reveals systematic overloading or under-contextualizing
Observed response latency	Tests whether the assigned latency band is realistic
Validation result	Captures tests, schema checks, review outcome, or operational verification
Rework and defect outcome	Connects routing to real engineering quality

Review the results after a meaningful sample, not after one impressive response. A simple interpretation table helps:

Observed pattern	Likely action
Economy tier consistently passes with low rework	Keep the route; consider moving similar bounded tasks to E
Balanced tier produces correct work but misses latency needs	Reduce context, split the task, or test a faster eligible candidate
Economy tier often escalates or causes rework	Move the row to B, strengthen the input contract, or improve the evidence supplied
Premium tier rarely improves validated outcomes	Test B as the default and reserve P for explicit escalation conditions
All tiers fail on the same task class	Do not simply upgrade models; inspect missing requirements, contradictory evidence, or weak validation

The key distinction is between a model limitation and a workflow limitation. If every candidate fails because the prompt contains neither the current API contract nor the relevant tests, upgrading the model will not solve the underlying problem. The next modules on repository instructions and evidence-first context packets address exactly that failure mode.

---

A lightweight operating rule for this week

Use the matrix on live work without changing everything else at once.

For each AI-assisted task, take less than a minute to record:

• the request class;
• its R, context, latency, and cost bands;
• selected tier;
• whether escalation occurred;
• final validation outcome.

If a request does not fit a row, create a temporary “unclassified” row rather than forcing it into the closest-looking category. After several examples, either define a new task class or decide that it is too rare to deserve a dedicated route.

Do not introduce an automatic classifier or router yet unless you already have a production use case with enough volume and evaluation coverage to justify the operational complexity. Dynamic routing can reduce spend, but it also makes debugging, cost forecasting, and performance analysis harder. A clear manual matrix gives you the evidence needed to automate responsibly later.

---

You now have a routing policy built around four concrete requirements: reasoning burden, relevant context pressure, latency tolerance, and cost constraint. The policy selects the lowest-cost eligible tier, makes escalation explicit, and measures whether the choice actually reduces effort without increasing rework or defects.

Next, you will inspect the repository instruction files that shape AI behavior before it sees a task. The goal will be to find ambiguous, stale, conflicting, and unverifiable guidance that can undermine even a well-chosen model.