Scary Stories to tell in the Network

With Halloween around the corner, here’s a real-world firewall policy horror story. (For effect, feel free to imagine this in a scary, raspy cautionary voice… or Morgan Freeman if you prefer.)

As a Sales Engineer, I spend a lot of days doing demos of our products, talking to Security Engineers, Compliance Folks, DevOps Managers, and CISOs about firewall and network security. Sometimes the stories of the folks in the trenches are unbelievable and sometimes downright scary. Here’s a recent tale from a customer that will keep every firewall engineer up at night.

Scenario:
The company had recently adopted a “zero trust” philosophy and had invested significant time to move closer to that goal. Over the last year they focused on cleaning up their firewall policies by writing rules specific to the business need – nothing more – just the specific IPs and networks that needed access. They were making progress when one of their engineers made a horrifying discovery, a dreaded “Any – Any – Any – Accept” rule buried in the middle of the policy.

They scrambled to figure out why this rule existed. They eventually uncovered that a junior network engineer, just trying to get something to work, had created this rule. This one rule circumvented all the progress they had made toward their zero trust goal and exposed the network to tremendous risk.

Clearly the rule had to be removed. But this rule had been in the policy – undetected – for at least 6 months and was certainly allowing business critical traffic. In fact, log traffic indicated this was an extremely busy rule. Of course, it was likely allowing more than just business critical traffic, it was likely permitting malicious traffic as well. They needed to remediate this problem quickly.

First, they had meetings with network folks and guessed at what some of the traffic might be, brainstorming critical applications and traffic that those familiar with the network knew might be using that rule. But ultimately, they decided they just needed to bite the bullet, remove the rule and see who screams.

So, they notified the managers across all the business units of this mistake. With embarrassment, they said that there was going to be a “hotline” set up with operators standing by, and ultimately had to have people testing out all the business-critical functions to make sure that their products, their applications, whatever they needed access to do their jobs and allow their customers/partners/vendors/ to do business with them would only be down until they could set things up. Yes, things did go down, and yes, they were set up to try to create new rules to remediate the issues, but what a nightmare.

Maybe you don’t have a nightmare scenario like I described above, but FireMon can still make your job easier by helping clean up your firewall policies. Here are a few ways in which FireMon could prevent the above scenario. And, if you ever discover a scary overly permissive rule in your policy, be sure to check out #6 below for a pain-free solution to the problem.

1) Compliance Alerting & Reporting:
We would have immediately notified the team if a rule went live that was overly permissive like this. You can basically “set the dial” on how permissive you want rules, such as “Rules allowing access to more than 60,000 destinations” or “Rules with Sources larger than a /16 network”.

2) Change Alerting:
This rule would be listed in our normalized view – no matter what the firewall vendor – and plainly seen that it was added. So it couldn’t be “snuck in”. Some engineers and managers get policy change reports emailed to them automatically every time a change is made – or even just a 30-day snapshot of everything that changed.

3) With our Automation tool, FireMon Policy Planner, in place – it would have stopped the risky rule in it’s tracks – before it even got pushed. Using our “pre-change analysis” the rule would have been identified and flagged before it was pushed to production to allow even a packet of traffic through it. That’s why compliance folks love that we don’t just recommend rules for creation based on need, but like a sandboxed environment, we run our compliance algorithms before we can automatically push the rules. Some clients get Policy Planner JUST for this features – and utilize our APIs to access it.

4) Also with Change Alerting, the team would know for sure who made what changes and when. So in this case, since it was a junior engineer, a manager or leader could have quickly filtered/sorted the changes that that user had made over the last week, or at least monthly, so that they would see what type of changes this user had made. This is also something the compliance team could have looked at, or even the user.

5) From a documentation perspective, FireMon can be the single source of truth for things like “who requested this, who’s the application owner, when was this rule last reviewed, etc”. So, by filtering and sorting rules with rule documentation, this would have been identified more quickly too.

6) I saved the best for last. If you do have an overly-permissive rule – such as if your predecessors had a different, more “open/flexible style of rule creation” policy than you do – we have an easy way to clean up these rules. Using Traffic Flow Analysis our tool will look at each and every IP address that flows through the rule, breaking down an any/any/any into specific flows. You can export these flows and create specific rules based upon them.