Compliance Unfiltered is TCT’s tell-it-like-it is podcast, dedicated to making compliance suck less. It’s a fresh, raw, uncut alternative for anyone who needs honest, reliable, compliance expertise with a sprinkling of personality.

Show Notes: Stop Automating Bad Compliance

Listen on Apple Podcasts
Listen on Google Podcasts

Quick Take

AI can automate compliance, but what happens when it automates bad compliance faster? Todd Coshow and Adam Goslin explore the dangers of scaling broken processes, weak evidence, and checkbox-driven compliance through AI.

Discover why automation without proper foundations creates more problems than solutions, where AI genuinely adds value, and why human oversight remains essential.

Learn how to identify automation opportunities, avoid costly mistakes, and build smarter, more effective compliance programs without sacrificing accuracy, accountability, or security.

Read The Transcript

So let’s face it, managing compliance sucks. It’s complicated, it’s so hard to keep organized, and it requires a ton of expertise in order to survive the entire process.

Welcome to Compliance Unfiltered, a podcast dedicated to making compliance suck less. Now, here’s your host, Todd Coshow with Adam Goslin.

Todd Coshow:

Well, welcome in to another edition of Compliance Unfiltered. I’m Todd Coshow, alongside the sonic to your compliance hedgehog, Mr. Adam Goslin. How the heck are you, sir?

Adam Goslin:

I am doing fantabulous today, Todd.

I just wanted to kick in. I wanted to thank the listeners for joining us for the extravaganza today. Certainly we love hearing from you guys, any input, feedback, etc., that you’ve got.

Certainly send that to [email protected].

As well, do us a favor. If you like listening to us, and I’m guessing you know some people that also may like listening to us, so do us a favor and give us a like, spread the word, spread the love, and let other people know that we’re here and doing what we’re doing, and that you appreciate it.

That would be ever so much appreciated on our side.

Todd Coshow:

All righty. Well, today we’re gonna talk about something that’s kind of out there everywhere. Everyone, Adam, is talking about automating compliance. AI is automating it. Software vendors are automating it. Organizations seem to be racing to automate everything.

But let me ask a different question. Is the automation of things always a good thing? What if an organization is just automating bad compliance?

Adam Goslin:

Well, apparently it’s easier to say than say.

Let’s start with what the heck are we talking about when we’re talking about bad compliance?

This is where the organization has a substandard or bad foundation of their evidence collection.

So if I’m starting with a substandard set of evidence, we always collect this, and it’s not gonna meet the mark, or it’s just barely scraping by or whatever, and now I go into this mode, I’m gonna spend all this time rinsing, repeating, and automating this shaky foundation, we’re not gonna achieve the objective.

We’re just basically gonna be pulling in marginal evidence at best, and meanwhile, we spent a whole bunch of time making it so.

If I’m pulling the wrong evidence for supporting user access review evidence, and then I turn around and spend the time to automate it, it’s almost like I’m shooting myself in the foot, right?

I’m not doing me or anybody else any favors.

So while automation has the capacity to be powerful, it’s not a substitute for sensibility, good governance, making sure you got your I’s dotted and your T’s crossed before you’re going down this path.

Layering technology on top of a house of cards isn’t gonna fix anything.

We’re going to be creating garbage faster and more consistently.

So if you have flaws starting out, then automation just means we can do it at scale.

Before you’re even asking, “Hey, well, how much of this can we automate?” the very first thing you should start with is, is what we have correct?

Should this process exist in its current form?

Let’s get onto a solid foundation before we’re moving into trying to automate the crap that we’ve got.

Todd Coshow:

That makes sense. I guess, where do you see organizations making the biggest mistakes when they automate their compliance programs?

Adam Goslin:

Well, when there’s this push, right?

We referred earlier to the advent of AI.

I don’t make any bones about where my headspace is at with a lot of the AI bullshit.

But bottom line is that you’ve got a whole bunch of people out there doing crap ’cause they think it’s cool.

Before I’m automating the manual work, there’s a whole bunch of things that we need to dig into.

Why the hell does what we’re doing with this manual work, why does it even exist? Is it needed? Is it appropriate? Is it proper?

Am I doing things in the right order, the correct way?

If I’ve got a crap workflow for how I was doing what I was doing manually, and then I turn around and automate it, I’m just perpetuating garbage.

For a lot of organizations…

Todd Coshow:

Crap on crap.

Adam Goslin:

Yeah, exactly. At some point in the game, the mound shall fall.

A lot of organizations have this mentality of, “Hey, we’re just gonna keep collecting what we’ve collected before, because that’s the way we’ve always done it.”

Is it right? Is it wrong? Is it what they should be doing, etc.? I don’t know.

Are there ways to optimize it, etc.?

You wanna fix the underpinnings before you go through the automation round.

Building the technology around inefficiency, it’s not a long-term solution that’s actually going to produce what you’re seeking.

It just means you’re ultimately putting a whole bunch of investment in so you can spend less time producing the garbage you produced before.

Todd Coshow:

Well, can you give us an example of where automation actually makes compliance worse instead of better?

Adam Goslin:

Well, think about it, right?

If for a particular requirement historically we produce these six pieces of evidence, well, do you really need all six of these pieces of evidence?

Does the assessor only really look at two of them?

So now we spend a whole bunch of time automating a bunch of evidence that nobody’s even looking at.

There’s one example where we’re expending time and creating crap that nobody’s even looking at.

Where your recurring control validations are just click-through exercises. There’s no real meat on the bone.

So we’ve automated this thing that’s substandard for what’s needed.

In the case of automation, yeah, okay, so I can produce reporting, as an example, at an astounding rate of speed.

And so I start peeling out these alerts and notifications and reports, and I’ve got all this access to the data, and I’ve just got this stream coming out at the users, whether they’re executives, whether they’re mid-range operational personnel, whether it’s frontliners.

People start basically dialing off to the noise that they’re getting in terms of the alertings.

One of the big problems that I had, especially with the ASV providers in the PCI SAC space for merchants, is they’ve got these systems that they basically designed around the notion that if I get a big green checkbox on this item, then that’s cool and I just keep it moving.

We wanna get away from the zombies just trying to click whatever they gotta click on the interface to make the pretty light turn green.

We don’t wanna be hiding operational issues.

We gotta be able to make sure that we’re building off a solid foundation.

Faster garbage still means you’re just getting garbage at a higher rate of speed.

Todd Coshow:

Right.

Well, Adam, you knew it was coming. Now comes the real question. How does AI change this conversation?

Adam Goslin:

Well, if AI is used in the right vein, it certainly can be beneficial.

I would really highly question some of the organizations that have done the headfirst leap off of the cliff into the chasm, thinking AI’s gonna save everything under the sun.

Todd Coshow:

The AI abyss.

Adam Goslin:

Yeah. We’re not quite there yet, people.

I don’t know what to tell you.

But yeah, you can use AI for doing things.

It can draft policies and summarize evidence and be a glorified, albeit not all that intelligent…

It’s funny when they refer to artificial intelligence and then it’s not all that smart.

For answering questionnaires or provisioning guidance, etc.

Those capabilities, those are valuable.

But now that I’ve got automation in the mix and I’ve got this AI coming through to do its thing, if the AI is just staring at stuff that’s garbage to begin with, then what are we really gonna produce at the back end of it, right?

Todd Coshow:

Well, truth be told is that AI’s acceptable level of failure is not quite where anybody would want it to be at this point in time.

That information is not always stuff you can depend on.

Adam Goslin:

Right.

Well, the reality is that if AI’s looking at garbage to start with, then what?

We’re gonna get a glorious overview of the crap that I fed into the system?

And I’m gonna get it at speed. Okay, great.

Well, I can have this stuff just smacking me repeatedly in the side of the noggin.

Todd Coshow:

This is crap, but yes, it’s fast crap.

Adam Goslin:

Yeah, fast crap.

Organizations are risking just automating bad assumptions instead of achieving operational excellence.

AI used properly off of a solid foundation, etc., has the capability to provide some accelerant.

But in the same sense, at least for the moment, it’s not a, “Hey, let’s go plug in AI,” and now it just outright replaces fill in the blank.

It’s not there yet.

We’ve had conversations before about the whole trust but verify, having humans in the mix, etc.

I think for a lot of organizations, there were a lot of tough lessons learned, have been learned, and are still being learned as we speak about the glories of AI.

Todd Coshow:

Indeed.

So how do organizations know whether they’re automating the right things?

Adam Goslin:

Well, there’s several things that they can ask themselves about each element of the evidence that they’re producing.

Is the generation of it having a positive impact in terms of risk reduction?

Is the information being produced for a particular control actually being leveraged to make decisions?

Is what’s being produced required by the control framework that you’re leveraging, or is it just a bad historical habit that’s been handed from one person to another and carried on till the dawn of time?

Are we simply documenting activity, or are we actually measuring something effectively?

And then the notion of if we have eliminated this task tomorrow, will our security actually get worse?

If the answer’s no to those, don’t automate it.

Then question why you’re doing it is the appropriate direction here.

Todd Coshow:

So if organizations shouldn’t automate everything, what should they be focusing on instead?

Adam Goslin:

Well, starting off with automating those things which are repetitive.

Keeping in mind, this is going under the guiding assumption that you’ve done the vetting, you’ve done the validation.

We’ve got validated controls that are ready to enter that next phase of analysis, is this a good idea or not?

The ones that are the most repetitive, consuming the most time from the humans, start there.

Make sure that you’re ready for stepping into this arena.

And don’t lose sight of the humanization of the judgment call, right?

At the end of the day, we need real, live, breathing humans to be able to use their brains to make calls on things.

So make sure you haven’t disconnected the humans from the loop.

For automation, things like evidence collection, notifications, monitoring of controls, routing workflows, data aggregation tasks.

Those are all good things for automation.

For the humans in the machine, things like using them for risk acceptance, exception handling, the original design of the controls, approval of the policies, strategic decisions that need to be made.

The best compliance programs aren’t replacing people with automation.

They’re eliminating low-value work so that the humans can focus on high-value decisions.

That’s the way that you basically can get your program from where it’s at and leveled up.

Todd Coshow:

Parting shots and thoughts for the folks this week, Adam?

Adam Goslin:

If the messaging comes through loud and clear as part of what we went through here today, then just do yourself a favor and take a dose of sanity with the AI ridonculousness.

I’m seeing a lot of companies. They’re moving too fast.

They’re lopping humans out of the mix, thinking that the companies are looking out from a profitability standpoint and just literally shooting themselves in the foot.

I can’t tell you the number of organizations that drank the AI Kool-Aid, turn off and whack a whole ton of people, only to have to turn around begging them to come back because they made a mistake.

Don’t be those people, right?

Let’s walk into this with a good game plan.

Let’s walk into this slow.

Let’s make sure, do all of the homework up front, then work through all your automation.

And like any other control design or system design that you would do in general, take your time.

Validate, vet, sanity check it, stress test it.

Let’s make damn sure that this is really what we want and we’ve done the right thing.

Do all of the sanity checks and validation.

Heck, bring the outputs of the automation and bring them to your security and compliance consultant.

Bring them to your assessor and get their sign-off.

“Yes, we understand how this is working, and this is gonna be great.”

You’ve done everything right, because that’s where people get into a lot of trouble.

There’s this fervent desire to create the automation, save all of this money, AKA build the profit for the company because now we don’t have to pay all these humans to do stuff, etc.

And it’s just running into problems, running into issues.

I think there’s a place, there’s certainly a place for AI, there’s a place for the automation, but just don’t get too far ahead of your skis.

Todd Coshow:

And that right there, that’s the good stuff.

Well, that’s all the time we have for this episode of Compliance Unfiltered. I’m Todd Coshow.

Adam Goslin:

And I’m Adam Goslin. Hope we helped to get you fired up to make your compliance suck less.

KEEP READING...

You may also like