the same time, Mark also met the security leaders and their teams at all of the agencies. During the mid-2000s, almost none had a formally appointed CISO, but most had someone they could point to and call their security leader.
One exception was a large agency with significant citizen privacy responsibilities. Chief privacy officers were even more rare than CISOs at the time, so privacy issues were typically part of the CISO's portfolio of responsibilities. When Mark met with the leadership of this particular agency, he encouraged them to fill the CISO/security leader role as soon as possible since they were accepting a significant amount of risk by failing to have a single point of contact to guide the security and privacy efforts of the agency.
Mark recounts what happened next:
“A few months after the conversation with this agency head, I received a call from someone who said they had just taken the CISO role at this agency and would be very interested in meeting with me to understand how they could quickly integrate into the statewide security leadership group. I remember thinking how odd it was that, even though I had no real authority within this agency and they were under no formal obligation to ask my opinion, they had hired a CISO without consulting with me about writing the job description or even being part of the interview process. Red flag number one.
“When I met the new CISO for the first time I was impressed by their attitude and enthusiasm to pitch in and help me, as we were educating the legislature, crafting statewide security policies, and realigning statewide procurement of security products and services. Once again, however, I remember having a strange feeling that this person didn't seem to really have the kind of experience you would expect for someone taking over the security and privacy responsibilities of a fairly large organization. Red flag number two.
“We developed a pretty good rapport and began speaking once or twice a week when one day several months later I called this CISO but they were out of the office. I left a message to call me back. A week later I hadn't received a call back so I called again and left another message. Another week went by and no call back so I walked over to the agency and asked a receptionist about the CISO. My antennas immediately began wagging when the receptionist appeared nervous and I could tell they didn't want to talk to me. This was truly odd and … red flag number three.
“I walked back to my office and set up an appointment to meet with the agency head. As I walked into their office the following day, I could immediately tell something was askew. The agency head told me in an extremely embarrassed tone that the CISO was no longer employed there. Of course I was shocked and employed my best negotiation skill of sitting quietly, saying nothing, and waiting for them to talk. The rest of the story was slowly revealed.
“In their haste to hire a CISO, this agency had posted a job description, interviewed candidates, and hired a CISO – all without ever conducting a background investigation. Several months after hiring the CISO, a law enforcement organization met with the agency head and informed them that their new CISO had just been released from prison after serving a term for embezzlement. The CISO job was their first employment following a multiyear prison term. The agency head was personally mortified telling me this story and I can only imagine the look on my face as I heard the tale. They kept saying how embarrassed they were since I had offered to help them with hiring a CISO and they simply forgot in the urgency of filling the role.
“This is easily one of the most extreme examples I've been involved with where a simple background check could have eliminated a serious headache, but it is also one that taught me a good lesson. Not checking all the boxes during a critical process like hiring can be very painful.”
No doubt, Mark's story highlights that building the right team to lead the overall cybersecurity program is a complex, difficult challenge. Beyond the CISO, most midsize and large organizations employ managers and/or directors to lead cybersecurity incident response and coordinate with the wider emergency management team throughout the enterprise.
We cover much more about this in Chapter 4.
FREQUENT RANSOMWARE ATTACKS PROMPT RESPONSE CAPABILITY ENHANCEMENTS IN NEW YORK STATE
One thing that Dan learned from his high school football coach is that you can't keep doing the same things over and over and expect a different outcome or result. This concept has proven to be true within the cybersecurity community over the past decade, with the frequency and significant impact of ransomware attacks forcing changes to the strategies and tactics of incident response teams all across the globe. One set of ransomware stories, and how organizations adapted, comes from the former CISO for New York State government.
Deborah Snyder is a senior fellow at the Center for Digital Government, and she has a distinguished career in cybersecurity, most notably as the CISO for New York State until her retirement in late 2019. Deb has a wealth of helpful stories regarding cyber incident response with many practical implications. Specifically, she shared the following ransomware stories.
In the early hours of Sunday, April 9, 2017, Erie County Medical Center (ECMC), a 550-bed hospital in Buffalo, New York, was hit by a cyberattack. Staff noted a digital ransom note on a hospital workstation that demanded $44,000 in Bitcoin cryptocurrency for the key to unlock the hospital's files. Hackers had encrypted ECMC's data, impacting over 6,000 hospital computers.
When a ransomware attack hits a healthcare provider and brings down their computer systems, it can cause significant and life-threatening disruptions, interfering with patient care and public safety. The attack caused ECMC to shut down email and their website, and resulted in a six-week electronic health records systems outage. To avoid further damage, ECMC was forced to revert to paper records and process patient admissions, prescriptions, and other tasks manually for weeks. The FBI investigated, and a cybersecurity firm was brought in to support forensics and recovery efforts. The New York Office of Information Technology Services (ITS) and Cyber Incident Response Team (CIRT) staff provided security guidance and support to the state department of health in protecting state systems and assets used by the facility. This incident was a wake-up call for many, as it dramatically highlighted the implications of interconnected systems and third-party access, and helped everyone involved grasp the serious potential for collateral impact. While ECMC didn't pay the ransom demand, the massive cyberattack came with a hefty price tag – nearly $10 million in hardware, software, response assistance, overtime, and other related expenses.
In another event, on August 30, 2017, the New York State Cyber Command Center (CYCOM) received a call indicating that Schuyler County had fallen victim to a sophisticated ransomware attack. The New York State CIRT team was immediately activated and an investigation confirmed that the attack involved SamSam – the same variant of ransomware ECMC had experienced. SamSam is typically distributed by compromising servers and using them to move laterally through the network to compromise additional devices. Given the touchpoints between state and county governments – both technical and programmatic – the government temporarily shut down network connectivity and access to the state's network and applications to prevent potential damage. Disruption to the county's 911 center and resulting risk to public safety was a major concern. This incident illustrated the potential for cyber events to have significant impact on public safety operations – fire, emergency medical, law enforcement, emergency communications, and other public safety partners – which in turn would directly and negatively impact the health and safety of the communities they serve. Fortunately, while some enhanced functions, such as integrated mapping, were impacted, the county was still able to receive and dispatch calls.
These incidents, along with the September 2017 Equifax breach that compromised personal data of 143 million Americans, including 8 million New Yorkers, served as a catalyst for formalizing comprehensive cyber disruption protocols. Coordinating resources, reporting, and response efforts across all involved state agencies –
