Engagement record
- Client
- Redacted — B2B SaaS, ~450 employees
- Estate
- Azure · 74 subscriptions · ~2,100 resources
- Monthly cloud spend
- $118,000
- Scope weighting
- Cost (40%) · Delivery (30%) · Security (20%) · Reliability (10%)
- Access granted
- Reader + Cost Management Reader, read-only repo access
- Elapsed
- 5 business days
1. Executive summary
The platform is functional and the team is competent, but the estate carries the cost of four years of growth without a platform owner. Three findings are urgent on a security basis and should be addressed this week regardless of any wider programme. The remainder resolve into two themes: spend that nobody is accountable for, and a release process whose slowness is now the main constraint on the product roadmap.
We can substantiate $36,400 per month of recoverable spend — 31% of the current invoice — of which $19,100 requires no architectural change and can be recovered inside two weeks by one engineer. We have listed the specific resources behind every figure so your team can verify each one in the portal before acting.
On delivery: median lead time from merge to production is 4.2 days, of which 3.6 days is waiting rather than working. Two manual approval gates account for most of it and neither has a named owner who can explain what they check.
2. Findings by severity
47 findings total
Active security exposure or single point of failure with no tested recovery path.
Material cost, delivery or reliability impact. Resolve within the quarter.
Worth fixing, no urgency. Batch into normal platform work.
3. Selected findings
Nine of the 47, chosen to show the range. The full report carries every finding in this format.
Fourteen service principals hold Contributor at subscription scope. Four have credentials that have not rotated in over 24 months, and two are used by pipelines that were decommissioned in 2024.
- Impact
- Any one of these credentials leaking grants write access to production. The two orphaned principals would not be noticed if used, as nothing alerts on their activity.
- Remediation
- Revoke the two orphaned principals immediately. Move the remainder to workload identity federation, scope each to the resource groups it touches, and enforce rotation through policy.
4 engineer-days
A storage account connection string and a third-party API key are committed in plaintext in two pipeline YAML files, present in git history since 2023.
- Impact
- Both are live. Every engineer with repository read access holds production credentials, and history rewriting will not help — they must be treated as compromised.
- Remediation
- Rotate both, move to Key Vault references, and add secret scanning as a required check with push protection enabled.
2 engineer-days
The primary transactional database is configured for geo-redundant backup, but no restore has been performed in the 19 months of available audit log.
- Impact
- Recovery objectives are stated in the customer contract but unproven. An untested restore path is not a backup.
- Remediation
- Run a restore to an isolated subscription, time it against the stated RTO, and schedule the exercise quarterly with the result recorded.
2 engineer-days, then 0.5 per quarter
Two GPU node pools (Standard_NC24ads_A100_v4, 3 nodes each) run continuously. Utilization over 30 days averages 8%, concentrated in weekday business hours.
- Impact
- $14,200 per month for capacity that is idle roughly 92% of the time.
- Remediation
- KEDA-driven scale-to-zero on both pools, triggered by inference queue depth. Retain one node warm during business hours if cold-start latency is customer-visible.
5 engineer-days · recovers ~$11,800/mo
Thirty-one non-production environments run 24/7 across dev, test and three per-team sandboxes. No auto-shutdown schedules are configured anywhere in the estate.
- Impact
- $9,400 per month, of which roughly 70% falls outside any working hours in the two timezones your team occupies.
- Remediation
- Auto-shutdown schedules via Azure Automation, with a self-service override so nobody is blocked. Enforce the schedule tag through policy on new resources.
3 engineer-days · recovers ~$6,300/mo
Reservation coverage on steady-state compute is 11%. Baseline analysis shows 78% of current compute has run unchanged for more than 12 months.
- Impact
- Approximately $7,100 per month of avoidable on-demand premium on workloads that are demonstrably stable.
- Remediation
- Three-year reserved instances on the identified baseline, one-year on the less certain tier. Right-size first — reserving an oversized VM locks in the mistake for three years.
2 engineer-days · recovers ~$7,100/mo
Two manual approval gates sit between merge and production. Median wait is 3.6 days combined. Neither gate has a documented checklist, and in 94% of the last 200 releases the approver approved without comment.
- Impact
- The gates are the dominant term in lead time and provide no evidence of catching defects. They are ritual, not control.
- Remediation
- Replace with automated policy checks — test coverage threshold, change-failure signal, security scan — and retain human approval only for changes touching payment or auth paths.
6 engineer-days
Roughly 40% of production resources have no infrastructure-as-code definition and were created through the portal. Staging and production differ in 23 identified configuration values.
- Impact
- Drift makes staging a weak predictor of production, which is a large part of why the manual gates feel necessary.
- Remediation
- Import existing resources into Terraform state, deny-by-default policy on portal creation in production, then close the 23 differences one at a time.
12 engineer-days
Alert rules generated 1,847 notifications in the last 30 days across 61 rules. Nine rules produced 71% of the volume, and none of those nine has ever been linked to an incident.
- Impact
- On-call fatigue and a measurable delay in acknowledging real incidents, which the paging logs already show.
- Remediation
- Delete or re-threshold the nine noisiest rules, require an owner and a runbook link on every remaining rule, and route non-actionable signals to a dashboard rather than a phone.
3 engineer-days
4. Recoverable spend
Every line is traceable to named resources listed in the appendix of the full report.
| Cause | Monthly | Effort |
|---|---|---|
| Idle GPU node pools | $11,800 | 5 days |
| Missing reservations on steady-state compute | $7,100 | 2 days |
| Non-production running out of hours | $6,300 | 3 days |
| Oversized VMs (19 identified) | $4,900 | 4 days |
| Orphaned disks, IPs and snapshots (212 objects) | $2,400 | 1 day |
| Log Analytics retention and ingestion tuning | $2,100 | 2 days |
| Storage tiering on cold blob data | $1,800 | 2 days |
| Total recoverable | $36,400/mo | $436,800/yr |
Against a $118,000 monthly invoice: 31% recoverable, of which $19,100/mo needs no architectural change.
5. Ninety-day roadmap
Sequenced by return on effort, not by severity alone. Security items that take hours come first regardless of where they rank.
Week 1
Stop the bleeding
- Rotate the two exposed secrets and revoke the orphaned service principals (SEC-01, SEC-04)
- Enable secret scanning with push protection across all repositories
- Sweep the 212 orphaned storage objects
Weeks 2–4
Recover the easy spend
- Auto-shutdown schedules on all 31 non-production environments (COST-03)
- Right-size the 19 identified VMs, then purchase reservations against the stable baseline (COST-07)
- Log Analytics retention and storage tiering changes
Weeks 5–8
Fix the delivery constraint
- Replace both manual gates with automated policy checks (DEL-01)
- Scale-to-zero on the GPU node pools (COST-01)
- Alert rule cleanup and ownership (OPS-03)
Weeks 9–12
Close the structural gaps
- Import portal-created production resources into Terraform (DEL-05)
- Workload identity federation for all remaining service principals
- First quarterly restore exercise, timed against the contractual RTO (REL-02)
What we would not do yet
- A service mesh. Your service count does not justify the operational cost, and it would not address any finding in this report.
- A full internal developer platform. Fix the gates and the drift first; a portal over a broken pipeline just hides it.
- Multi-region active-active. You cannot prove a single-region restore yet. Sequence matters.