Keynote by Darren Bilby: “A Decade of Lessons in Incident Response”

by Valerie Lopez of PRLinks for FIRST
Tuesday, June 13th, 2017

Day 2 of the FIRST Conference got started with keynote speaker Darren Bilby, a manager in Google’s Enterprise Infrastructure protection team, who is also a staff security engineer and self-described digital janitor. A 10-year veteran at Google, Bilby was the tech lead for Google's Global Incident Response Team for six years, managed Google’s European detection team in Zürich for two years and has also worked as a software engineer building out Google's security tools. He was also the founder and a core developer of the open source GRR Incident Response project. 

During his lecture, Bilby discussed the key lessons he learned in incident response at Google over the past 10 years, particularly those that were learned the hard way and what other security experts can take from them.

The first lesson Bilby pointed out was that the quality of a postmortem culture dictates the speed of evolution. “There was too much emphasis on who didn’t do their jobs, looking for blame. That’s human nature,” explained Bilby. “Back then it was about the users and what did they do wrong. It stopped us from evolving.” To overcome this, Bilby told the audience that they should stop being surprised when an incident occurred. “[The incident] happened because the process did not stop it. You own all that.” Instead, security personnel should write what Bilby called a blameless postmortem. The situation should be looked at objectively without trying to find something or someone to blame for it. 

Another lesson Bilby shared was that “you control the tools; the tools don’t control you.” One problem, according to the Google manager, was that many security tools failed to consider scale, automation and speed. Security incidents and experience taught Bilby and his team to adopt tools that are extensible, with open formats and open Application Programming Interfaces (APIs).

Bilby also noted that user education will not protect a company entirely. “We hire tech savvy people. That’s a tool but it’s not protection.” He added that security experts often give users security advice to better protect themselves online. Some of these range from “don’t go to bad parts of the Internet” to “don’t click on links that look suspicious.” While well intentioned, Bilby warned that it is not practical. To better protect users from modern threats, Google developed tools such as U2F security keys and the Password Alerts chrome extension.

Lesson number four for Bilby was that you can’t win on the network. While the model of network detection is effective against many threats, “Attackers that are good with computers and well organized will bypass it,” Bilby said. “The network perimeter has changed, the machine being attacked may actually be on a boat in the Caribbean over 4G and carry out an attack.” In response, Google developed BeyondCorp, an enterprise security model that shifts access controls from the network perimeter to individual users and their devices. In this model, clients are admitted to resources based on their current attested state.

One of the biggest experiences that Bilby learned in his tenure with Google is that “you can’t detect or respond your way to security.” No security system is perfect and compromise is inevitable. Security teams will always have to detect and respond, but Bilby thinks it is important that detection and response grow in parallel with protection mechanisms. “The reality is that detection, however, is only effective in a low-noise floor environment,” Bilby said. “Protection is critical in creating choke points for detection and response.”

No matter how experienced security experts are with incident response, Bilby warned that they can still develop incident fatigue. Security experts, he said, can get to the point where they don’t care so much and fall into a “it’s probably fine” mode. That can undermine a security expert’s effectiveness and affect a company’s security adversely.  Some solutions that have worked well at Google are to keep a 40% maximum time on call, keep mixed skill teams and give security teams the expectation of having the chance to have a time out.

When it comes to incident response, Bilby underlined that there is no substitute for experience. “When real incidents happened, we found out that the [incident response] plan we had before did not work,” he said. “Exercises are necessary but they are not sufficient.” Instead of relying on tabletop exercises and drills, Bilby recommended having realistic red teams and acquiring startups.  “One thing we saw with startups is that they practically invested nothing in security response, which creates a good challenge for us.”

Incident response tends to be reactive rather than proactive, but Bilby warned his audience that “you don’t win in reactive mode.” He added that it is important to have someone to look at the root cause of an incident, take a step back and truly understand it. He explained that the way security experts do root cause analysis changes how prepared the company is to deal with this. Bilby suggested security experts to agree on a five-year wish list. “What you need to prioritize gets you out of survival mode,” he said.

Bilby added that security teams also need to pay attention to firmware and supply chain because not all of them are completely secure. “Dealing with these issues are long-term problems. If we deal with this now, we can have more secure hardware in the long term,” Bilby said. 

In conclusion Bilby noted that “our job is not just to contain threats but to understand the problem and take steps to create real solutions that solve that problem.”