Incident Postmortem — Rackbinder

Incident

TitleIncident IDSeverityStatusServices affectedAuthorReviewers

Key times

Fault beganDetectedMitigated (impact stopped)Resolved

To detect: 36m
To mitigate: 39m
To resolve: 2h 49m
Total: 3h 25m

Timeline

Paste notes or a chat log — lines that start with a time become rows

When	What happened	Who	Marks
	First 'no valid IP configuration' ticket logged by service desk.
	Ticket volume rises; grouped as a possible Wi-Fi controller fault.
	Network tech checks the DHCP server: scope 10.10.0.0/24 at 100% leased. Incident declared.
	Lease time cut from 8 days to 2 hours; 37 stale reservations for decommissioned desktops removed.
	Free leases available again; affected users reconnect after ipconfig /renew or reboot.
	Emergency change approved to re-address VLAN 20 as 10.10.0.0/23.
	SVI, DHCP scope and firewall objects updated; lease time restored to 8 days. All clear.

Use roles or teams in “Who”, not names — the document is blameless. Marks feed the key times above.

What happened

Summary — three sentences: what happened, who it affected, how it endedCustomer impactInternal impact (optional)Root cause — the condition, not the personTrigger — what set it off on the dayContributing factors — one per lineDetection — how was it noticed, and by whom or what?Response and recovery

What we learned

Went well — one per lineWent wrong — one per lineGot lucky — one per line

Lessons learned (optional)

Action items

Action	Type	Owner	Due	Pri	Done
Alert when any DHCP scope passes 80% leased
Publish an addressing standard with a growth reserve per VLAN; review every 6 months
Add 'many users, no IP → check DHCP scope first' to the service-desk runbook
Quarterly clean-up of DHCP reservations against the asset register
Reduce default lease time to 1 day on user VLANs

Prevent: stops this class of fault from happening · Detect: makes it visible sooner next time · Mitigate: shrinks the blast radius or shortens recovery · Process: runbook, ownership, communication, training. Aim for at least one Prevent and one Detect.

Review

Nothing a reviewer would bounce this for. Read it once as the person who was on call, then publish.

Postmortem document

# Head-office users lose network access after DHCP scope exhaustion — Incident Postmortem

| | |
|---|---|
| **Incident** | INC-2026-0042 |
| **Severity** | SEV2 |
| **Status** | Draft |
| **Date** | 2026-09-02 |
| **Duration** | 3h 25m (2026-09-02 08:05 → 2026-09-02 11:30) |
| **Services affected** | Head-office LAN (VLAN 20 Users), Wi-Fi onboarding |
| **Author** | — |
| **Reviewers** | Network lead, Service desk lead |

> This is a blameless postmortem. It records what happened and what the systems allowed, so the same conditions cannot produce the same outcome. It is not a performance review.

## 1. Summary

From about 08:05 on 2 September, new devices at head office could not obtain an IP address on the Users VLAN. The DHCP scope for 10.10.0.0/24 had been sized for 200 workstations in 2023; a laptop refresh and a new floor of contractors pushed live leases past the 243 usable pool addresses. Existing leases kept working, so the fault surfaced as a trickle of tickets rather than an outage. The scope was widened by shortening the lease time and reclaiming stale reservations at 09:20, and the VLAN was re-addressed to a /23 in a change window at 11:30.

## 2. Impact

| Metric | Value | Measured |
|---|---|---|
| Time to detect | 36m | fault began → detected |
| Time to mitigate | 39m | detected → impact stopped |
| Time to resolve | 2h 49m | detected → fully resolved |
| Total duration | 3h 25m | fault began → resolved |

**Customer impact.** Roughly 40 staff could not connect between 08:05 and 09:20 (new arrivals and anyone who rebooted). No customer-facing service was affected.

**Internal impact.** Service desk took 31 tickets in 40 minutes; two were misdiagnosed as Wi-Fi faults, which delayed escalation.

## 3. Timeline

All times local to the responders.

| Time | Event | Who | |
|---|---|---|---|
| 2026-09-02 08:05 | First 'no valid IP configuration' ticket logged by service desk. | Service desk |  |
| 2026-09-02 08:25 | Ticket volume rises; grouped as a possible Wi-Fi controller fault. | Service desk |  |
| 2026-09-02 08:41 | Network tech checks the DHCP server: scope 10.10.0.0/24 at 100% leased. Incident declared. | Network | detected |
| 2026-09-02 08:55 | Lease time cut from 8 days to 2 hours; 37 stale reservations for decommissioned desktops removed. | Network |  |
| 2026-09-02 09:20 | Free leases available again; affected users reconnect after ipconfig /renew or reboot. | Network | mitigated |
| 2026-09-02 10:00 | Emergency change approved to re-address VLAN 20 as 10.10.0.0/23. | Change advisory |  |
| 2026-09-02 11:30 | SVI, DHCP scope and firewall objects updated; lease time restored to 8 days. All clear. | Network | resolved |

## 4. Root cause and trigger

**Root cause.** The Users VLAN was sized for its 2023 headcount and never re-planned. There was no addressing standard that stated a growth reserve, and no monitoring on DHCP scope utilisation, so the pool filled silently over weeks.

**Trigger.** Twelve contractor laptops joined VLAN 20 on the morning of 2 September, taking the last free leases.

**Contributing factors**

- Decommissioned desktops still held reservations, wasting 37 addresses.
- An 8-day lease time meant departed devices held addresses for over a week.
- The service desk had no runbook step for 'many users, no IP' that pointed at DHCP first.
- The addressing plan lived in one engineer's spreadsheet and had not been updated since 2023.

## 5. Detection

Detected by users, not by monitoring. Thirty-six minutes passed between the first ticket and the incident being declared, because the symptom pattern was read as a Wi-Fi fault.

## 6. Response and recovery

Once DHCP was identified the mitigation was quick: shorter leases plus reservation clean-up freed addresses within 25 minutes. The permanent fix (re-addressing to a /23) needed a change window because it touched the SVI, the scope and firewall objects that referenced the old subnet.

## 7. What we learned

**What went well**

- Mitigation took 25 minutes once the cause was found.
- The emergency change was approved and executed inside two hours.
- Existing leases kept working, so most staff were never affected.

**What went wrong**

- Detection depended on ticket volume, not monitoring.
- Two tickets were misdiagnosed, which cost 20 minutes.
- The only record of the addressing plan was a stale spreadsheet.

**Where we got lucky**

- The incident started before 9 am, so the morning rush of reboots was still ahead of us.
- The /23 above the existing /24 happened to be free.

## 8. Action items

| # | Action | Type | Owner | Due | Priority | Status |
|---|---|---|---|---|---|---|
| 1 | Alert when any DHCP scope passes 80% leased | Detect | Network | 2026-09-12 | P1 | Open |
| 2 | Publish an addressing standard with a growth reserve per VLAN; review every 6 months | Prevent | Network lead | 2026-09-19 | P1 | Open |
| 3 | Add 'many users, no IP → check DHCP scope first' to the service-desk runbook | Process | Service desk lead | 2026-09-10 | P2 | Open |
| 4 | Quarterly clean-up of DHCP reservations against the asset register | Prevent | Network | 2026-10-01 | P3 | Open |
| 5 | Reduce default lease time to 1 day on user VLANs | Mitigate | Network | 2026-09-12 | P2 | Open |

## 9. Lessons learned

A subnet sized for today's headcount is a slow-motion outage. Size for growth, write the reserve into a standard that outlives the engineer who chose it, and alert on utilisation before users do.

## 10. Review and sign-off

| Reviewer | Role | Date | Approved |
|---|---|---|---|
| Network lead | | | |
| Service desk lead | | | |
| | | | |

---
_Generated 2026-09-07 with the Rackbinder Incident Postmortem. Nothing was uploaded; the document was assembled in the browser._

Summary, impact with time-to-detect / mitigate / resolve, timeline, root cause and contributing factors, detection, response, what we learned, action items, sign-off. Markdown for your wiki; Word for the meeting.