Compliance Unfiltered is TCT’s tell-it-like-it is podcast, dedicated to making compliance suck less. It’s a fresh, raw, uncut alternative for anyone who needs honest, reliable, compliance expertise with a sprinkling of personality.
Show Notes: The Control Worked Yet The Company Still Got Breached
Quick Take
Passing an audit doesn’t mean you’re secure. In this episode of Compliance Unfiltered, Todd Coshow and Adam Goslin expose the critical gap between compliance and real cybersecurity.
Learn why controls can pass every audit yet still fail to stop modern attacks, and why continuous validation is replacing point-in-time evidence.
Discover how organizations should test resilience through penetration testing, red teaming, tabletop exercises, backup recovery, and control effectiveness—not just documentation. If you think “audit passed” equals “risk reduced,” this episode will change the way you measure security forever.
Read The Transcript
So let’s face it, managing compliance sucks. It’s complicated, it’s hard to keep organized, and it requires a ton of expertise in order to survive the entire process.
Welcome to Compliance Unfiltered, a podcast dedicated to making compliance suck less. Now, here’s your host, Todd Coshow with Adam Goslin.
Todd Coshow:
Welcome in to another edition of Compliance Unfiltered. I’m Todd Coshow, alongside the fresh coat of whitewash to your compliance fence, Mr. Adam Goslin. How the heck are you, sir?
Adam Goslin:
Oh, I’m doing good.
And speaking of which, man, I had some poor neighbor across the road, down the street. They’ve got this nice white picket fence. It never fails, man. I think they’re cursed. There’s a turnoff there, and so when somebody’s not paying attention, their car, especially in the winter, will go off the road and just blast through anywhere from 15 to 30 feet of their nice white picket fence.
It’s becoming an annual occurrence. I’m pretty sure they’re tired of it at this point in the game.
Todd Coshow:
Oh, man. Well, shout out to your neighbor’s fence.
We’re gonna talk today about there being some obstacles put up and things still getting through.
But before we do so, we’d like to thank everybody who has taken the time to listen to this very podcast. We would ask that you take the opportunity to give us a rating or a review on your podcast app of choice. It really helps the podcast.
Additionally, if you’d like to reach out, like to share any of your anecdote stories or topics for conversation, please feel free to do so at [email protected].
Today’s topic, Adam, is one that I feel like too many folks are familiar with, and that is the company still got breached despite all of your controls working.
So what happens when every box is checked, every control passes the audit, and the attacker still gets in?
Adam Goslin:
Controls can operate as designed, but still end up failing to protect the organization.
Compliance is measuring whether or not a control exists and if that control was executed. Security measures whether that control actually reduced the risk.
One of the earlier euphemisms that you would hear a lot is compliance doesn’t equal security.
What those people were getting at is that just because I have checked this box that says this thing’s in place doesn’t mean I’m actually secure.
The control working doesn’t necessarily mean the organization’s protected.
We need to make a shift from control execution to control effectiveness so that we can gain risk mitigation.
There’s also the possibility, and this happens to fewer of those organizations, but there’s a possibility that you’ve got a zero day out there.
But even in the case of a zero day, you’ve got a myriad of other controls that ought to be effective, where those detection mechanisms still have the ability to identify, “Hey, Houston, you got a problem.”
Whether it’s identifying control bypass through monitoring of central logging, unusual user activity with behavioral analytics, file integrity monitoring if you’re seeing files changing within the environment, failed attempts from inside the network as attackers are trying this, trying that, and trying the other thing.
There’s a reason why, even with zero days, it ends up seeing the light of day.
It’s about mitigating the amount of time between, “Houston, we got a problem,” and knowing that you have a problem.
Todd Coshow:
What’s the difference between a control operating as designed and a control actually being effective?
Adam Goslin:
When it’s operating, you’re checking the box. This thing happened.
But when a control is effective, it means that the risk was meaningfully reduced.
You can have every single person in your organization going and attending security awareness training and seeing their quarterly security reminders and the piece of paper that’s taped up on the inside of the bathroom door, and yet employees still fall for phishing attempts.
Maybe you’ve got a situation where an organization went in, ran their vuln scans, but they left vulnerabilities unpatched for a period of time.
There’s gonna be some period of time between recognition that I have a vulnerability and getting it cured, and depending on how long of a span that is.
It doesn’t necessarily have to be just critical vulnerability. A lot of people will obviously focus in on critical vulnerabilities.
But what they seem to miss in their vulnerability management is that oftentimes I can take two or three medium-level vulnerabilities and conjoin them to create something that’s far more impactful.
It’s a matter of treating the environment properly.
You wanna stop measuring activity, and you wanna start heading toward realistic outcomes where there’s a material benefit to the organization.
Todd Coshow:
Why do organizations and even auditors sometimes miss this gap?
Adam Goslin:
It’s easier to do an audit and say that something exists. It’s far easier to do that than it is to say, “Is this being done effectively?”
In a compliance arena, you’re focusing on things like policies or screenshots, reports, evidence.
The vast majority of what’s being done during that cycle is also just at a point in time.
I proved out that this was done once, four and a half months ago.
But the harder question would be, would the existence of that control actually stop an attack today?
The organizations confuse measurable activity with actual security improvement.
One of the bigger problems is that you’ve got organizations that, because they have the piece of paper that says that they’re compliant, they equate that to, “Oh, I must be secure.”
The two are not directly related to one another.
Oftentimes, the passing of a compliance assessment is a good sign that you at least have a shot at having all these things done.
But it doesn’t necessarily mean that sometime down the road, when we’re not in the midst of our annual compliance scramble, that we’re actually still doing and maintaining all of the things that we proved at a point in time happened to be there.
Todd Coshow:
Sure. How much of this problem comes from threat actors evolving faster than controls?
Adam Goslin:
Attackers aren’t machines. They’re gonna paint outside of the lines. They’re gonna think outside of the box.
There’s a continuous stream of adaptation that happens.
Even where maybe there was, for a particular attack vector, a pattern that now has been recognized that I can now build into detection, prevention mechanisms, etc., the attacker basically takes that pattern and then tweaks it slightly so that it now bypasses those particular mechanisms.
Some of the best examples of that that we saw over time were where there was a particular new virus that had been issued.
The pattern for that virus has now been built into the platforms looking for the existence of said virus.
So what do the bad guys do?
They go in there and they tweak it.
They tweak it so that the pattern being seen by the detection tools is now not picking up this revised or modified or morphed pattern that would be seen by the system and thereby bypassing it.
To further complicate that, you look at today’s environment where you’ve got the advent of AI able to very quickly modify and morph the patterns that are being seen by the detection mechanisms, and the speed with which the attackers can make those tweaks and changes.
That’s coming at a fast and furious pace.
Some of the modern threats that we’ve got are credential abuse, exposure to a particular set of credentials and being able to leverage those.
I used the example a minute ago where AI-supported modifications to detection patterns for malware.
But you’ve also got AI-assisted phishing, where you’re able to integrate various elements of the target that you’re going after, integrating those elements into your phishing expeditions, and being able to create hyperrealistic attack vectors that will have a much greater chance that especially the uninitiated will fall subject to.
You’ve got things like third-party issues.
I’ve got connected vendors or connected components within my systems that are depending on somebody else’s security causing a problem in my systems.
You’ve got cloud systems that get misconfigured, whether it’s because of the fact that they weren’t set up securely out of the gate or there’s some type of an overhaul of the configuration of the cloud platform that opens up a new hole.
A good analogy here is I can have a front door.
I envision the front door with the big banker wheel vault thingamabob where I turn it, and all of a sudden there’s bolts going in at the top, bottom of the door, into the side, into the door jambs, etc.
Meanwhile, the attacker goes and climbs through an open window.
It’s not doing you a hell of a lot of good having that door when they’re coming in a different direction.
You can have controls that maintain their functional state, but in a real world becoming strategically obsolete.
That’s a big problem for organizations as they’re trying to figure out what’s the best way to approach this issue.
Todd Coshow:
That certainly sounds that way. How should organizations pressure test controls against today’s threats?
Adam Goslin:
Testing controls, you have to test controls individually.
I almost go back to the dev approach.
I need to do unit testing to make sure that each of my specific elements are working in and of themselves.
But then there’s the notion of integration testing, where I am now taking all of these basically elements or units, putting them all together and saying, “Now that I’ve got all these pieces together, are they all working harmoniously and still working appropriately?”
The same premise should apply to an organization, making sure that you’re putting together all of the various individual pieces, not just stopping at unit testing.
Hopefully that’s an analogy that the listeners go, “Yeah, that makes sense.”
Things like tabletop exercises.
There’s a lot of organizations that I’ve seen struggle with the notion of tabletop exercises.
It’s awkward the first time you do it, okay? It really is.
I’ve watched this unfold at a multitude of organizations where the first time they go in and do the tabletop exercise, it kind of feels like you’re back in your early teens. We’re at the school dance, and everybody’s looking at each other awkwardly across the gym.
That’s how a first run at a tabletop feels.
But what you’ll find is that getting through that first awkward run, you do it a second time, a third time, a fourth time, people start to get the rhythm.
You learn more and more how to structure those tabletop exercises over time.
I’ve honestly seen a lot of real material benefit and light bulbs go on during those subsequent tabletop exercises.
They can be astoundingly helpful.
Things like penetration testing.
Do me a favor, Todd, while I’m talking this through. We did a pod on pen testing versus vulnerability scanning. Can you find out what was the date or episode number on that? And then we’ll throw that out a little bit later on as we’re going through this.
Back to the pen testing.
Whenever you’re doing pen testing, and I believe I talk about this in that pod, but I’ll give a high-level overview here.
There’s various scales of penetration testing.
At the lowest end of the food chain, you’ll see pen testing vendors hawking these penetration scans.
The companies that are hawking this stuff basically have this notion that, “We integrated some of our findings, automated them, effectively scripted them into an automated penetration scan. And so now it’s just as good as if you went ahead and actually did an appropriate pen test.”
It’s simply not the case.
All it is, it’s nothing more than a glorified vulnerability scan with some additional doodads tapped on top.
I can’t express the importance enough of when you’re going in and you’re doing pen testing.
Yes, there’s a place for tool-based testing. There’s a place for reviewing and validating the tool-based testing.
But the element that is critical is real, live human beings, security engineers that can go through and take a look at the environment.
They can think outside of the box.
They are not gonna follow a pattern.
They’re going to do things that are unexpected.
They’re gonna enter things that the system wasn’t expecting.
They’re taking multitudes of different vulnerabilities they’re now aware of and trying to gang them together to create an even bigger vulnerability.
The attacker isn’t just gonna run a set of tools and walk away.
They’re gonna put their head down and try to figure out, “How can I bypass this? How can I get in? How can I gain access to sensitive data or information I shouldn’t be able to get to?”
That’s the way that you’ve got to approach your pen testing.
The final piece I want to say on the pen testing side is you do want a reasonable balance between the pen scan and then on the other end of the spectrum, I like to call them the ninjas-dropping-from-ceiling-tiles companies.
They’ll bring an entire team in, and you put them up in a hotel for weeks on end, and I have this visual of hired ninjas dropping down from ceiling tiles trying to gain physical entry to the location.
Do you necessarily need to go that crazy?
You can if you want to, and there’s some certain benefits to it.
But for most organizations, sitting in the middle and taking a middle-of-the-road approach is usually good.
Red teaming.
Getting red teaming involved.
Again, taking advantage of the capabilities of the real live human being.
Doing incident simulation.
That could be combined with your tabletop exercises. Maybe they’re one and the same, maybe they’re different.
But running through incidents, making sure that all the right people know what they’re supposed to be doing and when based on what happens.
Who do I talk to?
What if it’s 3:00 in the morning on a Sunday?
It’s really comes down to asking better questions as you’re going through this exercise.
What happens if somebody manages to bypass our MFA?
What happens when we do have an incident at 2:00 in the morning on a Sunday?
Can backups actually restore?
It sounds kind of no-brainer-ish, but I can’t tell you how many companies created backups but then never actually tested that they can bring something back from the backup.
It is amazing some of the things that you run into when you try to go down that route.
You wanna be testing for resilience.
You don’t wanna just look for the existence of documentation. That’s really the key here.
Todd Coshow:
No doubt.
And for those waiting with bated breath for those security vulnerability scans versus pen testing, all the way back to episode 20 for that one, Adam.
Adam Goslin:
Woo-hoo.
Todd Coshow:
And then we also did a penetration testing deep dive on episode 169.
Adam Goslin:
Awesome. Thank you.
Todd Coshow:
If passing the audit doesn’t mean you’re protected, what should organizations really be trying to achieve?
Adam Goslin:
You wanna start moving from compliance to actual assurance.
You wanna be asking questions like, are my controls still operating? Are they still effective? Are they still relevant? Are they doing what they’re supposed to do?
You’ve gotta, when you’re looking at compliance, especially when you’re not—
Okay, compliance means different things in different people’s worlds.
If I go in and I try to do compliance, finger air quotes, with something like HIPAA, HIPAA’s very directional in terms of what they do.
There is a lot of latitude when you’re dealing with a direction.
There’s a lot of directional standards out there.
There’s a lot of latitude in terms of there being mistakes or issues with organizations just doing barely enough to be able to say that they’re doing fill in the blank versus doing it appropriately versus doing it well.
The more prescriptive your security and compliance standard is that you’re leveraging and the more seriously that you’re taking it, that’s where the bar starts to get raised.
Again, I harken back to compliance isn’t security.
You waving around your little piece of paper about how I got compliance seven and a half months ago isn’t gonna stop an attacker.
Validation, continuous validation, that’s gonna be a point-in-time verification or validation every day and twice on Sunday.
Todd Coshow:
Parting shots and thoughts for the folks this week, Adam.
Adam Goslin:
We’ve touched on it throughout the conversation today.
For a lot of organizations, I really feel, especially at the executive and board level, they have this notion that if I passed my audit, we’d be cool.
The two are not necessarily different. Not necessarily.
But really what it comes down to, and the light bulb that I want to have go on in the heads of board level, executive level, mid-management level, is that you as an organization, you need to take this stuff seriously.
You need to start thinking outside of the box.
You need to start stress testing your stuff.
You need to move into a mode of operational compliance, continuous compliance, where we are sanity checking things on a more regular basis.
You need to push the boundary lines.
I’ve seen too many organizations just try to check the boxes so they can get their piece of paper.
There are a lot of them out there that are like that.
Challenge yourselves.
Your obligation, being in business and having something of value that people are willing to pay for, really should mean you have an obligation to those folks to take your obligation seriously, protect their data and information, and to do everything that you can to try to protect it.
Not only will you then be living up to the obligations that you have to the people that are trusting you, but it’s also an active way to be able to help to protect the company.
There’s people that work at these organizations that are dependent on their paychecks, vendors that are dependent on the company not having an issue.
Be part of the solution and don’t continue to be part of the problem.
Todd Coshow:
And that right there, that’s the good stuff.
Well, that’s all the time we have for this episode of Compliance Unfiltered. I’m Todd Coshow.
Adam Goslin:
And I’m Adam Goslin.
Todd Coshow:
Hope we helped to get you fired up to make your compliance suck less.
