Translating Business Expectations into Measurable Quality Attributes

Good to see you again. In the previous lesson, you created a prioritized risk register for the authorization-aware evidence assistant. It identified unacceptable or high-priority conditions: unauthorized disclosure, indirect prompt injection, unsupported citations, sensitive-data leakage, stale sources, weak ownership, and unreliable third-party dependencies.

This lesson turns those risks and the business case into measurable system quality attributes. The aim is not to collect generic statements such as “the platform must be fast and secure.” You will define the conditions under which the assistant is genuinely usable, trustworthy, and operable, then express them as testable scenarios, metrics, targets, and release gates.

---

Business outcomes need system qualities to make them achievable

A functional requirement says what the system does:

An authorized analyst can submit a question and receive an answer with cited enterprise evidence.

A quality attribute says how well the system must do it under stated conditions:

For an authorized analyst querying an approved corpus under normal pilot load, the platform returns a grounded, cited response within the agreed latency target without exposing content outside that analyst’s permissions.

The distinction matters because the first statement can be true even when the product fails in practice. A system can generate answers but be too slow for a live workflow, show citations that do not support its claims, fail to index updated policy documents, or expose restricted material. In each case, the feature exists, but the business value does not.

The prior value hypothesis stated that the platform should reduce evidence-preparation effort for eligible requests. That outcome depends on several qualities working together:

Business expectation	System quality attributes that enable it	Misleading shortcut
Analysts prepare evidence packs faster	Responsiveness, availability, retrieval freshness, usability	“The API returns HTTP 200.”
Analysts can trust the output	Groundedness, citation correctness, traceability, transparency	“Every answer contains at least one link.”
Enterprise information stays protected	Authorization integrity, confidentiality, privacy, auditability	“Users must log in.”
The pilot can scale safely	Reliability, resilience, cost control, observability	“The demo worked once.”
Teams can sustain and improve it	Modifiability, testability, clear ownership	“The team can edit the code.”

The central chain of reasoning is:

1. Start with a business outcome or risk tolerance.
2. Identify the system behavior needed to support it.
3. Define how that behavior will be measured in a real user journey.
4. Set a target, release threshold, and owner.
5. Use results to make a delivery, pilot, or investment decision.

This is especially important for generative AI. A fast answer is not valuable if it is unsupported; a highly accurate answer is not useful if it takes several minutes during a time-sensitive workflow; an available service is unsafe if it mixes documents across authorization boundaries.

For the capstone, the relevant business outcome might be stated as:

Reduce the time eligible analysts spend preparing verifiable evidence for internal decision briefs, without weakening authorization controls or requiring analysts to trust uncited AI output.

From this, candidate quality attributes emerge naturally:

• Authorization integrity: only content a user is entitled to access can be retrieved or supplied to a model.
• Groundedness and citation quality: material claims in an answer are supported by the displayed evidence.
• Performance: the answer arrives quickly enough to fit the analyst’s workflow.
• Availability and reliability: the critical journey works often enough when the user needs it.
• Freshness: approved source changes become searchable within an agreed time.
• Recoverability: the platform can restore service and data after disruption.
• Auditability: the organization can investigate which sources, retrievals, model configuration, and decisions led to an answer.
• Modifiability: the team can change a prompt, model configuration, source connector, or policy with controlled validation and rollback.

These are not all equally important. The risk register provides an initial priority order. Authorization, sensitive-data handling, and supported citations are non-negotiable guardrails. A one-second latency improvement should never be bought by relaxing those controls.

---

Quality attributes become useful when written as scenarios

“Secure,” “reliable,” and “easy to maintain” are aspirations. A quality attribute scenario makes an aspiration observable. It is a structured description of a stimulus, the system’s response, and the measure used to decide whether the response is adequate.

The Software Engineering Institute calls such scenarios architectural test cases: they provide the context needed to test whether architecture and implementation satisfy the requirement.

Use this six-part template:

Scenario element	Meaning	Evidence-assistant example
Source	Who or what creates the stimulus	An authorized analyst, ingestion pipeline, security tester, or model provider
Stimulus	The condition or event	A user asks a question; a source changes; a provider times out; a user attempts access to restricted content
Artifact	The system element affected	Retrieval service, authorization policy engine, API, source index, or model gateway
Environment	The operating condition	Normal pilot load, peak load, dependency degradation, recovery mode, or a new deployment
Response	What the system does	Returns an answer, denies access, records an audit event, retries safely, degrades gracefully, or restores service
Response measure	The observable threshold	Percentage, percentile latency, elapsed time, error rate, evaluation score, or zero-tolerance condition

Consider the difference between these two requirements:

• Weak: “The system must be secure.”
• Testable: “When a user submits a query for which all relevant documents are restricted, the retrieval service returns no restricted passages to the application or model context, records a deny decision, and passes all cases in the agreed role-resource authorization test matrix before pilot.”

The second requirement creates architectural consequences: authorization must be enforced server-side; the retrieval layer needs permission-aware metadata and filtering; testing must include adversarial role and document combinations; logs must prove the access decision without leaking content.

Four quality scenarios for the MVP

The following scenarios convert the prior business case and risk register into an initial quality-attribute backlog.

1. Authorization integrity

Source: an authenticated analyst whose role does not permit access to a restricted document.
Stimulus: the analyst asks a question whose likely answer appears in that restricted document.
Artifact: authorization-aware retrieval and model-context assembly.
Environment: normal operation and adversarial authorization regression testing.
Response: the service excludes the restricted document from retrieval results and model context, returns only authorized evidence or abstains, and records the access decision.
Response measure: zero unauthorized-document exposures in the agreed role-resource regression suite; zero confirmed unauthorized exposures in production.

This is a release-blocking guardrail, not a metric that can be traded casually against speed or cost.

2. Grounded, cited answers

Source: an authorized analyst.
Stimulus: the analyst submits a supported evidence question.
Artifact: retrieval, prompt construction, model generation, and citation renderer.
Environment: normal pilot load using approved sources.
Response: the platform returns an answer whose material factual claims are supported by inspectable citations, or it clearly states that sufficient evidence was not found.
Response measure: at least 95% of sampled material claims are supported by the linked evidence on a representative evaluation set; 100% of factual answers either show a source citation or abstain.

The 95% figure is an initial planning target, not a fact established by the previous lesson. It must be calibrated with business stakeholders according to the consequence of error, the request types included in the pilot, and the evaluation method. For consequential claims, the acceptable threshold may be higher, or human review may be required regardless of the score.

3. Response time for the critical journey

Source: an authorized analyst.
Stimulus: the analyst submits an eligible query.
Artifact: web client, API, retrieval service, model gateway, and citation renderer.
Environment: expected pilot load, including up to 20 concurrent active users.
Response: the user receives a completed cited answer or a clear failure state.
Response measure: the 95th percentile end-to-end completion time is no more than 10 seconds over a rolling seven-day window.

This definition deliberately measures the journey from user request to usable answer, not merely the fastest internal API call. A low model-call latency is insufficient if retrieval, authorization, or rendering makes the user wait.

4. Source freshness

Source: an approved enterprise source owner or ingestion connector.
Stimulus: a new document or approved document revision becomes available.
Artifact: ingestion workflow, metadata store, embedding/indexing pipeline, and retrieval index.
Environment: normal operation.
Response: the revised content becomes searchable with its source identifier, owner, version, and timestamp.
Response measure: 95% of approved source updates are searchable within four hours; failed ingestion jobs create an actionable alert within 15 minutes.

Freshness is a business-quality requirement because stale evidence can undermine otherwise well-cited answers. It is distinct from citation correctness: a citation can correctly support a claim while pointing to an obsolete document.

---

Make metrics unambiguous enough to operate

Every important quality attribute needs more than a target number. It needs a measurement contract. Without one, teams can each report a different version of “availability,” “accuracy,” or “latency.”

For every metric, define:

Metric component	Question it answers
Critical user journey	Whose task is being measured, from which meaningful start to which meaningful end?
Population and denominator	Which requests count? Are user cancellations, invalid requests, or planned maintenance excluded?
Aggregation	Is the measure a mean, median, percentile, rate, maximum, or sampled score?
Time window	Is the target evaluated per deployment, per day, weekly, or monthly?
Instrumentation source	Which logs, traces, test suite, audit records, or human evaluations produce the data?
Target and threshold	What result is adequate? What result blocks release or triggers remediation?
Owner and response	Who reviews misses, and what happens when the target is not met?

A metric without a denominator is particularly dangerous. “We observed 99.9% availability” is incomplete until the team agrees what counted as an eligible request and what a successful outcome meant. An API that returns an error message quickly should not count as a successful evidence-answer journey merely because the load balancer received a response.

Likewise, “every answer includes citations” is a weak proxy. Citation presence is not citation support. Your quality measure must evaluate whether cited evidence actually entails, substantiates, or appropriately qualifies the associated claim.

Three categories of threshold

Do not treat every target as the same kind of commitment.

1. Release gates decide whether the system may enter a pilot or production environment.
   Examples include no critical authorization-filter failures in the regression suite, privacy approval for documented data flows, and a minimum citation-support score.

2. Service-level objectives define the ongoing reliability users can reasonably expect from a critical journey.
   Examples include availability, response-time, ingestion freshness, or successful completion rate.

3. Diagnostic or improvement metrics help the team understand performance but may not block release by themselves.
   Examples include average token use, retrieval latency by repository, fallback rate, or distribution of abstentions.

This separation prevents a team from declaring the platform ready because operational metrics look acceptable while a critical safety control remains unproven.

---

Use SLIs and SLOs for the qualities that must hold in operation

An SLI, or service-level indicator, is the measurement itself. An SLO, or service-level objective, is the target for that measurement over a defined time window.

For example:

• SLI: percentage of valid evidence-answer requests that result in a cited answer or an intentional, user-visible abstention within 10 seconds.
• SLO: at least 99.5% over a rolling 30-day period.

The difference is subtle but operationally vital: the SLI tells you what you observe; the SLO says how much unreliability the organization is willing to tolerate before it must intervene.

An error budget is the tolerated amount of failure implied by an SLO. For an illustrative 99.5% availability SLO measured over a 30-day calendar window:

$$720\text{ hours}\times (1-0.995)=3.6\text{ hours}$$

This does not mean a team should plan to spend 3.6 hours unavailable. It means the business has expressed a reliability tolerance. If the service consumes that budget quickly, the team should prioritize reliability work, investigate failures, and potentially slow feature delivery until reliability recovers.

For the evidence assistant, an availability SLO should measure the critical journey, not infrastructure uptime alone. A model provider being reachable is not sufficient if authorized retrieval fails; conversely, a temporary model failure can still be handled acceptably if the platform gives the analyst a clear status, preserves the request where appropriate, and avoids presenting unsupported output as an answer.

A practical first operational SLO can therefore be:

Over each rolling 30-day period, at least 99.5% of valid authorized evidence queries complete with either a cited answer, a clear evidence-insufficient response, or an actionable failure message within the defined interaction boundary.

The definition should exclude malformed client requests but should not exclude failures caused by your own dependencies simply because they are external. From the user’s perspective, a third-party outage still affects the journey.

---

Recoverability, traceability, and changeability are also business commitments

Availability measures whether a service is usable now. Recoverability measures whether it can return to a trustworthy state after disruption.

Two terms make this concrete:

• Recovery Time Objective (RTO): the maximum acceptable time to restore service after a disruption.
• Recovery Point Objective (RPO): the maximum acceptable data loss measured in time, such as the amount of ingestion or audit data that might need to be recreated.

For the capstone, a provisional recoverability scenario might be:

Following a failure of the primary retrieval index, platform engineering restores the service using approved backups and documented procedures. The recovery takes no more than four hours, and no more than 24 hours of ingestion metadata or audit records require reconstruction.

Whether those values are appropriate depends on the workflow. If the platform is used to support an executive decision during a narrow meeting window, four hours may be unacceptable. If it is an early internal pilot with manual fallbacks, it may be appropriate. The target comes from the cost of downtime and data loss, not from a default cloud-service setting.

Two further qualities deserve explicit requirements because of the previous risk register.

Traceability and auditability

A cited answer should be reproducible enough for a reviewer to understand how it was produced. A suitable scenario is:

When an analyst flags an answer as incorrect, the platform records a trace identifier, authenticated user identity, access-policy decision, retrieved source identifiers and versions, model and prompt configuration version, response timestamp, and feedback status. Authorized support personnel can retrieve this record within five minutes without accessing source content beyond their permissions.

The response measure could require 100% trace coverage for eligible production interactions. But auditability must be designed alongside privacy: do not store raw prompts, sensitive passages, or model outputs indiscriminately just because they are useful for debugging.

Modifiability and controlled change

The system will evolve. Sources, policies, prompts, models, and vendor APIs will change. A quality attribute can make safe adaptation measurable:

When a model or prompt configuration is updated, the delivery team can deploy the approved configuration behind a version identifier, run the agreed evaluation suite, and revert to the previously approved version within 15 minutes without changing application code.

This scenario drives concrete architecture choices later: versioned configuration, controlled rollout, evaluation automation, and reversible deployment practices.

---

Draft quality-attribute profile for the capstone

The following profile consolidates the lesson into a usable first artifact. Its numerical targets are proposed starting points. Confirm them with the executive sponsor, product owner, security lead, enterprise data owner, and pilot analysts before treating them as commitments.

Priority	Quality attribute	Measure and initial target	Verification method	Decision owner
P0	Authorization integrity	Zero unauthorized documents or passages enter retrieval results or model context in the approved role-resource test matrix	Automated authorization regression tests; adversarial test cases; access-audit review	Security/IAM lead
P0	Confidentiality and privacy	Documented data flow is approved; no unapproved sensitive fields are present in logs; zero confirmed unapproved provider disclosures	Privacy review, log scans, provider configuration review, incident monitoring	Data protection owner
P0	Groundedness and citation support	At least 95% of sampled material claims are supported by displayed evidence; 100% of factual answers cite evidence or abstain	Representative evaluation set with defined reviewer rubric	AI product lead
P1	End-to-end responsiveness	P95 completed-answer or intentional-abstention time no greater than 10 seconds at expected pilot load	Client-side timing, distributed traces, load tests	Platform engineering lead
P1	Availability of the critical journey	At least 99.5% of valid authorized evidence queries receive a valid outcome during each rolling 30-day period	Journey-level monitoring and SLO reporting	Service owner
P1	Source freshness	At least 95% of approved source updates become searchable within four hours; failed ingestion alerts occur within 15 minutes	Ingestion events, source timestamps, indexing telemetry	Enterprise data owner
P1	Auditability	100% of eligible interactions have a trace ID, policy-decision record, source IDs and versions, and model configuration version	Trace completeness checks and incident drill	Service owner
P2	Recoverability	Restore retrieval capability within four hours; recreate no more than 24 hours of eligible ingestion metadata	Scheduled recovery exercise	Platform engineering lead
P2	Modifiability	Roll back a prompt or model configuration to the last approved version within 15 minutes after a failed evaluation or incident	Deployment rehearsal and configuration audit	Technical lead

A target becomes credible only when its measurement is feasible. If you cannot yet measure claim support automatically, begin with a smaller human-reviewed evaluation set and make the process explicit. If you cannot measure end-to-end latency from the client, record that as an observability gap rather than reporting only server-side timing.

---

Build the quality-attribute backlog before choosing the architecture

Your capstone artifact for this lesson is a quality-attribute backlog. Use one entry per priority quality and record:

ID and quality attribute:
Business expectation or risk addressed:
Critical user journey:
Scenario source:
Stimulus:
Affected artifact:
Operating environment:
Expected response:
Metric definition and denominator:
Target:
Threshold type: release gate, SLO, or diagnostic metric
Test or telemetry evidence:
Accountable owner:
Trade-offs and open assumptions:
Review date:

Start with the P0 entries from the table: authorization integrity, confidentiality and privacy, and grounded citation support. Add responsiveness, source freshness, and availability next. This order ensures that the architecture is shaped first by the conditions required for trustworthy use, rather than by implementation convenience.

Finally, make trade-offs visible. Common ones for this platform include:

Decision pressure	Likely trade-off	Required response
Lower latency	Less retrieval depth, smaller model, or shorter context	Measure citation support and completeness so speed does not degrade evidence quality silently
Rich debugging logs	Increased privacy exposure	Minimize and protect logs; retain identifiers and controlled metadata where possible
More source coverage	Greater metadata, authorization, and freshness complexity	Onboard approved repositories incrementally with named source owners
Stronger access controls	More denial cases or user friction	Provide useful authorization-aware messages and a governed access-request path
More resilient provider setup	Higher implementation and operating cost	Compare the cost to the business impact of a failed critical journey

A quality attribute is not an abstract “non-functional” add-on. It is a statement about what the organization is willing to pay for, tolerate, and trust.

---

Key takeaways

• Business outcomes become real only when the system qualities that enable them are made explicit.
• Functional requirements describe what the evidence assistant does; quality attributes describe how it must behave under meaningful conditions.
• Write quality requirements as six-part scenarios: source, stimulus, artifact, environment, response, and response measure.
• Define a measurement contract: journey, denominator, aggregation, time window, instrumentation, target, owner, and response to misses.
• Treat authorization integrity, privacy, and citation support as P0 release gates, not negotiable performance optimizations.
• Use SLIs and SLOs for ongoing operational qualities such as availability, latency, and freshness; define error budgets only after agreeing what failure means for the user journey.
• Include recoverability, auditability, and modifiability because the assistant must be governable after the initial demo.

Next, you will use these quality attributes to sketch a system context diagram. It will show the people, enterprise data sources, AI providers, platform boundary, and AWS boundary that must interact to meet these requirements.