The Untested Controls Problem: Why Your Defenses Never Actually Improve

Key Takeaways

• A clean pentest report is a snapshot, not a guarantee. Most environments change faster than the testing calendar allows.
• Detection rules break silently. No alert tells you when one stops firing.
• Alert volume is not the same thing as coverage. A busy SOC dashboard can still be blind to real attacks.
• Purple teaming only works when it happens often enough to catch drift before attackers do.
• Continuous, automated validation is what turns a one-time test into an ongoing defense.

A security team we heard about ran a picture-perfect penetration test last fall. Zero critical findings. The board was thrilled. Three months later, a routine log source migration quietly broke four detection rules, and nobody noticed until an incident review months after that. The test wasn’t wrong. It just stopped being true the moment the environment changed.

That gap between “tested once” and “true right now” is where most breaches actually start.

Why a Clean Pentest Report Doesn’t Mean You’re Safe

Every security team has a folder full of pentest reports that say “no critical issues found.” Few of them ask the harder question: found compared to what, and as of when?

Detection gaps persist because tests are frozen in time

A typical four week penetration test only reaches somewhere between 15 and 30 percent of an organization’s real attack surface. The other 70 to 85 percent sits untested until next year’s engagement rolls around. That’s the honest reason why detection gaps persist after penetration tests. The test was never designed to cover a system that changes daily.

Why pentest reports sit unactioned for months

Even when pentests do find real issues, a lot of those findings never get fixed. Critical vulnerabilities get flagged as high priority and then stall out, usually for one of a few reasons:

• Nobody owns the fix.
• Nobody scoped the engineering time it would take.
• The ticket got buried under the next sprint.

That’s why pentest reports sit unactioned for months in so many organizations. The report becomes a PDF nobody opens again.

Why the same attack techniques keep succeeding

When the same weaknesses stay open, attackers don’t need new tricks. Most repeat breaches trace back to a short list of familiar problems:

• Reused or stolen credentials.
• Misconfigured access permissions.
• Systems that were already flagged and never patched.

Not zero days. Old news that never got closed out.

The three quiet failure points

• Scope gap. A single test covers a slice of the environment, not all of it.
• Time gap. New assets, new cloud workloads, and new firewall rules show up days after the test wraps.
• Ownership gap. A finding without a clear owner and deadline rarely gets fixed at all.

The Slow Rot of Detection Rules No One Is Watching

Pentests are only half the picture. The other half lives inside your SIEM, and it’s aging quietly while everyone assumes it’s fine.

Why security controls drift after tuning

Detection rules have a life span. They get written, tested once, deployed, and then mostly forgotten. Meanwhile the environment underneath them keeps moving:

• Parsers get updated.
• Log sources get swapped for new tools.
• Cloud providers change their event formats.
• The engineer who wrote the original rule leaves the company.

Piece by piece, this is why security controls drift after tuning. Nobody broke anything on purpose. The ground just shifted under a rule that used to work.

Why detection rules break silently over time

Research analyzing tens of thousands of production SIEM rules across common platforms found that close to one in five detection rules stop firing the way they’re supposed to, often without any warning from the platform itself. That’s exactly why detection rules break silently over time. The dashboard still lights up. The rule still shows as “enabled.” It just doesn’t catch anything anymore.

This creates a strange kind of danger. Alert fatigue gets all the attention because it’s loud and visible, and analysts drown in low-value tickets that everyone notices. A rule that silently stopped working is worse, because it creates the illusion that coverage still exists. Nobody feels the absence of an alert that should have fired.

How alert noise hides real attack behavior

Volume makes this worse, not better. A packed alert queue feels like proof the SOC is working hard. It isn’t proof of anything.

• Thousands of low-priority pings can bury the one signal that actually mattered.
• Analysts trained to triage fast start pattern-matching instead of investigating.
• A busy dashboard and real coverage are two different things, and only one of them is visible.

This is how alert noise hides real attack behavior in plain sight.

Why MITRE ATT&CK coverage is often overstated

Plenty of security programs proudly report high percentage coverage against the framework. But coverage on paper usually means a rule exists somewhere, not that it was ever tested against a real technique. That’s a big reason why MITRE ATT&CK coverage is often overstated. A rule that’s never been triggered by an actual simulated attack is a guess wearing a checkmark.

Why Traditional Testing Can’t Keep the Pace

If detection gaps and silent rule failures are the problem, red teaming and purple teaming are supposed to be the fix. In theory, they are. In practice, most teams can’t run them often enough for that to be true.

Why remediation is slow after adversary simulations

Full scale adversary simulations take real planning. Scoping, coordination between offense and defense, careful sequencing so nothing breaks production. That’s why remediation is slow after adversary simulations even when the exercise itself goes well. Fixing what a simulation uncovers competes with every other item on a stretched team’s backlog.

Why lean teams skip adversary emulation

Smaller security teams feel this hardest:

• A team of three or four analysts can barely keep the alert queue moving.
• Running a full adversary emulation on top of daily operations often just isn’t possible.
• Skipping it isn’t a lack of will. It’s a lack of hours in the day.

That’s why attack simulations overwhelm small security teams, and why lean teams skip adversary emulation altogether even though they know they need it.

Why red team engagements are so expensive

Traditional red team engagements built around expert consultants and manual execution run into real money. That’s why red team engagements are so expensive for exactly the organizations that could use them most often. A yearly engagement is what most budgets allow, which brings us right back to the coverage gap from earlier.

What blocks collaboration between offense and defense teams

Even when the budget and the people are there, offense and defense often speak different languages:

• Red teams think in attack paths.
• Blue teams think in alert volume and shift coverage.
• Reports get handed off instead of worked through together.

That’s what blocks collaboration between offense and defense teams in the first place. And EDR tools, for all their sophistication, still miss known techniques when the underlying telemetry has a gap or a rule hasn’t been tuned to catch a specific variation. That’s why EDR alerts miss known attack techniques more often than vendors like to admit.

Confidence in fully automated fixes for this hasn’t kept pace with the hype either. Willingness to rely on fully autonomous penetration testing actually dropped sharply in 2026, and most security leaders now say they prefer a hybrid setup where a person stays in the loop instead of handing testing fully to automation, according to Dark Reading. Speed matters, but so does judgment.

What Actually Closes the Gap: Continuous Validation

None of this means red teaming or purple teaming is broken. It means once a year was never going to be enough.

Why purple team exercises fail to deliver value

The teams that stay ahead treat control validation as an ongoing habit instead of a scheduled event. They run smaller, more frequent simulations mapped to real MITRE ATT&CK techniques, feed the results straight back to detection engineering, and keep repeating the loop. That structure is exactly what fixes why purple team exercises fail to deliver value in their current form. A one-off exercise generates a report. A recurring one generates a habit of getting better.

What causes false confidence in security controls

This also solves the trust problem that quietly undermines a lot of security programs. A rule that passed a code review and looked correct on deployment day can still be the exact reason why teams end up with what causes false confidence in security controls. Confidence based on a control that was tested once, a long time ago, isn’t confidence. It’s a guess with good intentions.

Why threat detection validation is still mostly manual

Manual validation doesn’t scale to fix this, and everyone in the SOC already knows it. That’s the honest answer to why threat detection validation is still mostly manual in most organizations. There simply aren’t enough hours to hand test every rule against every technique every month, which is exactly the kind of repetitive, well defined task automation is built for.

How untested playbooks fail during real incidents

Waiting for a real incident to find the cracks is the most expensive way to learn about them. A response playbook that looked solid on paper falls apart the first time it meets a real, messy incident, which is precisely how untested playbooks fail during real incidents. Tabletop exercises help, but nothing replaces watching a control actually get exercised.

Why control validation results expire so quickly

Validation itself has a shelf life. An environment that passed a control check last quarter can look completely different today after a cloud migration or a vendor swap. That’s why control validation results expire so quickly, and why “we tested that already” is rarely a complete answer six months later.

Why boards struggle to trust detection metrics

All of this rolls up into a boardroom problem. Executives see dashboards full of green checkmarks and thousands of alerts processed, and they reasonably assume the program is working. Security leaders know the checkmarks don’t always mean what they imply, which is why boards struggle to trust detection metrics even when the numbers look good. Fixing that trust gap starts with metrics tied to actual, repeated testing, not just deployment counts.

Bringing Continuous Validation Into Your SOC

This is precisely the gap Secure.com’s SOC Teammate is built to close. Instead of treating detection validation as a once a year project, it runs continuous, automated checks against your existing SIEM and EDR stack, mapped to real MITRE ATT&CK techniques, so silent rule failures get caught in days instead of during a post-incident review.

• Purple team results feed directly back into detection engineering, so a gap found on Tuesday can turn into a working rule by Thursday, not sit in a backlog for a quarter.
• It’s the same shift toward a repeatable, continuous purple team loop that’s replacing the once a year fire drill industry-wide.
• It pairs naturally with moving away from annual red team testing toward something that actually keeps pace with a changing environment.

For lean teams especially, this matters. You don’t need to double headcount to get continuous coverage. You need the validation work automated so your analysts spend their time on the findings that actually matter.

FAQs

The Bottom Line

Untested controls aren’t broken controls. They’re just unknown ones. The fix isn’t more testing once a year. It’s making validation something that happens continuously, quietly, in the background, so your team finds the gap before an attacker does. That’s the difference between a defense that looks good on a report and one that actually holds up.