Spec vs Reality
Read the requirement. Compare it to the built feature. If they don't match, that's a bug. The simplest test technique there is.
1 The Hook
An Revenue NZ tax form spec said: "Revenue NZ number must be in format 000-000-0000." A tester typed 1234567890 with no dashes. The form rejected it. The tester reported a bug. But the spec said dashes were required. The product matched the spec. The real question: should dashes be mandatory? A good tester compares spec to reality, then questions the spec.
2 The Rule
The spec says X. The product does Y. If X ≠ Y, that's a bug.
Read what the requirement demands. Look at what the software actually does. If they disagree, you found a defect. Write it down clearly.
3 The Analogy
Ordering a pizza.
The menu says "Large pepperoni pizza." The box arrives with a small ham pizza. The menu is the spec. The pizza is the product. Your job is to check they match before the customer opens the box.
Senior engineer insight
The shift that changed how I approach this: I stopped reading specs to find bugs and started reading them to find questions. The dangerous gaps aren't where spec and product disagree — they're where the spec is technically met but the intent was missed entirely. A NZ government API spec I worked with said "the endpoint must return a 200 response"; the developer returned 200 with an empty payload on every error. Spec met. Feature broken.
The most common mistake Grad testers make here is treating "spec says X, product does X" as the finish line — without asking whether X was actually the right requirement in the first place.
From the field
A team I worked with was testing a NZ Benefits NZ case management system. The requirements document said "users must be able to update their contact details." The team tested the happy path — enter new address, click Save, confirmation shown — and marked it pass. What nobody checked: the requirement said "update" but the built feature only allowed one address field, with no versioning. Caseworkers were silently overwriting historical addresses needed for audit trails. The spec said update. The product said overwrite. We only found it because a senior tester asked "what does 'update' actually mean in this context?" That question — not the test execution — was the real skill.
4 Watch Me Do It
Follow these five steps every time you compare a spec to a built feature.
- Read the requirement first
- Write one test case for each criterion
- Execute the test exactly
- Compare actual result to expected result
- Document any discrepancy
| Test case | Input | Expected | Actual | Status |
|---|---|---|---|---|
| With dashes | 123-456-7890 | Accepted | Accepted | Pass |
| Without dashes | 1234567890 | Rejected | Rejected | Pass |
| Too short | 123-45 | Rejected | Accepted | Fail |
Test case 3 failed. The spec says too-short numbers should be rejected. The product accepted it. That's a clear bug.
Scenario 2: Spec ambiguity reveals the testing challenge
A specification says: "Users must be able to cancel their booking." But it doesn't say whether they can cancel 1 hour before or 24 hours before. The developer interprets this as "24 hours", but users expect "1 hour". No bug report, just a gap between spec and user expectation.
| Interpretation | Spec says? | Developer built | User expects | Bug? |
|---|---|---|---|---|
| Time window | "Cancellable" (vague) | 24 hours before | 1 hour before | Unclear |
| Refund method | "Refunded" (vague) | Credit card refund only | Money back to account | Unclear |
| Notification | Silent (no spec) | Email sent | SMS also sent | Feature request |
This is not a bug—it's a spec gap. Your job is to flag these before the build ships. Ask clarifying questions: "How many hours before cancellation is allowed?" Make the requirement testable.
5 When to Use It / When NOT to Use It
✅ Do it when...
- You have a written requirement
- A feature is marked "ready for testing"
- You need to prove a feature works as agreed
❌ Skip it when...
- There is no spec at all
- You are doing pure exploratory testing
- The spec is obviously outdated
Before you apply this technique, ask:
- Do you have access to the spec or requirements document?
- Is the spec current or has it been updated since the feature was built?
- Are edge cases documented, or only happy paths?
- Are you comparing against the spec or just against your own assumptions?
6 Common Mistakes
🚫 Testing without reading the spec
I used to think: I can figure out what it should do by clicking around.
Actually: Guessing leads to false bugs. Read first. Test second. Always.
🚫 Ignoring implied requirements
I used to think: If the spec doesn't mention it, it's not a bug.
Actually: Some requirements are implied. A login form with no submit button is obviously broken, even if the spec forgot to say it.
🚫 Reporting "should work like X" when X is not in the spec
I used to think: This would be better if it worked differently.
Actually: A bug is a gap between spec and product. If you want it to work differently, that's a feature request, not a bug.
🚫 Ignoring spec ambiguities
I used to think: If something is unclear in the spec, I should guess what was meant.
Actually: Ambiguities are the spec's fault, not yours. If the spec says "must be fast" but doesn't define timing, that's not a testable requirement. Flag it as ambiguous, don't guess.
When this technique fails
Spec-versus-reality testing fails when you don't have access to the spec or it's outdated and no longer reflects actual requirements. It also fails when specs are ambiguous ("should be user-friendly") but you test them anyway, finding false bugs. The best specs are testable: specific numbers, clear acceptance criteria, edge cases defined. Without that, spec-based testing is guesswork.
7 Now You Try
Spec: A NZ delivery app has a "Postcode" field. The requirements say:
- Must accept exactly 4 digits
- Must not accept letters or spaces
- Must show error: "Please enter a valid NZ postcode"
You test the built feature and see:
- "8011" → accepted
- "80 11" → accepted (no error)
- "abc" → shows "Invalid input"
Your task: How many discrepancies do you see?
Two discrepancies:
| # | Discrepancy | Spec says | Product does |
|---|---|---|---|
| 1 | Spaces accepted | No spaces allowed | Accepts "80 11" |
| 2 | Error message | "Please enter a valid NZ postcode" | Shows "Invalid input" |
Note: "abc" being rejected is correct. Spaces accepted is a functional bug; message text is a content bug.
Why teams fail here
- Testers compare the product against a spec version that was superseded mid-sprint — the doc was updated in Confluence but nobody told QA, so the tests are validating requirements that no longer exist.
- Requirements use vague NZ government language ("must be appropriate," "should align with policy") that sounds testable but isn't — testers pass features they cannot actually verify, and the gap surfaces in UAT or post-go-live.
- Teams only test the stated requirement, not adjacent implied behaviour — the user story says "login must work," so login is tested; nobody checks that a failed login doesn't leak the username in the error message or the URL.
- Spec gaps get closed verbally in a standup or Slack thread but never written down — the product owner says "yeah just make it 24 hours," the developer builds it, and by the time the tester arrives the original spec and the actual implementation tell different stories with no audit trail.
Key takeaway
The spec is your contract — but your job is not just to enforce it, it's to find where the contract is broken, ambiguous, or simply wrong before a real user does.
8 Self-Check
Click each question to reveal the answer.
Interview Questions
What NZ hiring managers ask about Spec vs Reality at the Grad level.
Q1. You are testing a form and notice the spec says the field maximum is 100 characters, but the form accepts 500. Is this a bug?
Strong answer: It depends on the risk. If the spec reflects a business rule (e.g., a database column is 100 chars and storing 500 would cause data truncation or errors), yes, it is a bug. If the spec was just an arbitrary UI guideline and storing 500 chars causes no downstream problems, it may be a documentation issue rather than a defect. I would ask the developer or BA to confirm the intended behaviour. In my bug report, I note the spec says 100, the implementation accepts 500, and ask which is correct.
Q2. A specification says the button should be blue. In production it is teal. Is this a bug worth reporting?
Strong answer: Yes, with appropriate severity. A colour discrepancy from the design spec is a low-severity cosmetic bug — it should be reported so the team can make a conscious decision. The designer may have updated the design system after the spec was written, or the dev may have used the wrong hex code. Testers do not decide what gets fixed; they provide information so the team can decide. If the teal fails the brand guidelines or WCAG contrast requirements at that size, the severity increases.
Q3. How do you know what the expected behaviour of a feature should be when the specification is ambiguous?
Strong answer: I start by asking the BA or product owner — this is the fastest path to correct information. If they are unavailable, I use existing behaviour of similar features in the same product (consistency), industry standards for that feature type, and common sense about user expectations. I document my assumption and flag it as "tested based on interpretation — please confirm." This is not a weakness; discovering ambiguity in a spec is a valuable testing outcome, because if the spec is ambiguous to me, it was also ambiguous to the developer.
Q1. The spec says a button should be blue. The product shows green. Is this a bug?
Yes. The product does not match the spec. Report the discrepancy.
Q2. The spec says the password must be 8 characters. You test 8 and it works. Are you done?
No. Also test 7 characters (should fail). Check both what works and what doesn't.
Q3. You think a feature should work differently, but the spec agrees with the product. Is this a bug?
No. That is a feature request. A bug only exists when the product disagrees with the spec.