On May 7, 2021, news broke of a DarkSide ransomware attack on Colonial Pipeline, a major U.S. fuel pipeline that supplies roughly 45% of the East Coast’s diesel, gasoline and jet fuel. In response to the attack, the company shut down its pipeline for several days.
The incident has been extensively reported on. Some reports suggest that the decision to temporarily halt operations was done as much for financial concerns as it was for safety. Other reporting points to issues within the company’s IT network that existed prior to the incident and which may have contributed to the company’s vulnerability to attack. Some question the lack of a dedicated executive in charge of cybersecurity.
Colonial Pipeline CEO Joseph Blount was invited to testify at a House Homeland Security Committee hearing on June 9 and 10. The combined four-hour long official Senate and House testimony includes several interesting revelations about the attack, as well as some important guidance for other companies that might someday find themselves in a similar situation.
Here’s a look at some of the important security lessons highlighted by this attack that defenders can take away.
The need to prioritize security
While the testimony did not expressly address the lack of dedicated cybersecurity leadership at the company, this is one area that cannot be overlooked, especially in a company as large and important as Colonial Pipeline.
A chief information security officer (CISO) is a critical role, responsible for ensuring that companies have a comprehensive security program, a strategic vision for cybersecurity, and a seat at the business’s decision-making table.
Colonial Pipeline’s testimony to the Senate revealed that around $200 million had been invested in IT in the last five years, but it was not clear how much of that had been allocated to cybersecurity.
Being able to set cybersecurity priorities for the organization, having sufficient budget to implement them and the necessary authority to enforce those priorities are a key part of securing an organization.
While dedicated CISO roles might not be appropriate and necessary for every company, the discipline and focus that a CISO brings to the environment is.
That means companies need to at least invest in, and commit to, having a cybersecurity program and an incident response plan in place. These should encompass everything from implementing the right tools and creating a security culture, to knowing the steps to follow in the event something goes wrong.
Defaults matter
During the testimony, it was confirmed that the initial entry point into the Colonial Pipeline network was a single stolen password.
In this instance, as it is in many of the cases the Sophos Rapid Response team investigates, remote services were to blame. Specifically, the attackers used the stolen password to gain access to a VPN service that did not have multi-factor authentication (MFA) enabled.
It appears that Colonial Pipeline believed this VPN profile was not in use. This is also a situation we have seen before.
The company has explained that the password complied with strict complexity requirements even though it was not enabled with MFA. The attackers may have obtained the password from an earlier breach and taken a chance that the same password would be valid on Colonial Pipeline’s network.
It’s important to remember that old breaches can bite back, and while you may not be responsible for them, they can still have a dramatic impact on your organization.
This is where a robust security culture can help.
Having employees who are mindful of how they use their credentials can mitigate the effects of third-party security failures. While you’re at it, you can help them out by providing them with a password manager that can be used for both their work and personal accounts.
It is also worth setting a policy that MFA is on by default and can only be disabled by a documented exception. While the lack of MFA on this VPN may simply be down to a misconfiguration, it remains a missed security opportunity.
This is an area where CISOs, or equivalent security leadership, can set policies that enable the organization to do the right thing by default.
Prevention is ideal but detection is a must
According to the investigators, the earliest indicator that the attackers were in the network was April 29, 2021. This means the attackers were in the Colonial Pipeline network for at least eight days prior to the ransomware attack on May 7, 2021.
The company’s lack of visibility into what the intruders were doing during those days was one of the reasons why it decided to shut down the pipeline. Here is an extract from the CEO’s statement:
“…in this case, obviously it was the concern that we really had no vision into our IT or OT systems to understand the degree of corruption, and encryption, and it really took us days even with the help of world class expert [sic] by Mandiant to get there, so again, that’s why that decision was made…”
According to the Sophos Rapid Response team, ransomware is often the first sign that alerts victims to the fact that an attack has occurred.
This is by design. Many of today’s ransomware operators prefer to operate in complete stealth until it’s time to release their final payload. They’ve breached your network, established persistence, elevated privilege, exfiltrated your data, and only then do they deploy the ransomware. This can take hours, days, or months to unfold. In fact, according to Sophos’ Active Adversary Playbook 2021, the observed median attacker dwell time is 11 days, with some companies having attackers in their network for six months or more.
The fact that Colonial Pipeline didn’t have the visibility it needed to understand how badly it had been penetrated is, unfortunately, a common problem for many companies.
Cybersecurity programs are essential, but so are tools to enable them. Endpoint Detection and Response (EDR) tools are invaluable, not only for preventing attacks, but also for enabling your organization to hunt for latent threats.
Remember, just because your security software detected and blocked a threat, that doesn’t mean the job is done. There might be a bigger problem lurking undiscovered in your network.
Plan on failure
Being a large, critical infrastructure company, Colonial Pipeline is no stranger to emergency response plans. There’s little doubt it has comprehensive plans for physical failures of all kinds, from pipeline ruptures to physical security intrusions. However, the company’s position seems less robust when asked about response plans for cybersecurity incidents, although it did say it used outside penetration testing services.
This is important. Organizations of all sizes should perform some sort of assessment of their security controls. Some of the assessments can be done internally, but they should also be supported by outside consultation. Among the risks of relying solely on internal audits is a myopia about your capabilities and a tolerance for compromise because “that’s just how it’s always been.”
Following your evaluations, you will need to work up plans to a) improve the areas where you are weakest, b) prepare a plan for when things will go wrong, and c) test your defenses against the improvements and response plan.
I am reminded of an organization that told me about their quarterly tabletop exercises, whereby they would simulate and work through IT issues in order to assess their readiness and learn how their existing plans could be strengthened. During one of these exercises, they simulated a ransomware attack against their infrastructure, and adjusted their response plan based on learnings from the exercise.
Shortly after the exercise, they suffered a massive storage area network (SAN) failure. Ironically, the SAN failure presented itself very much like a ransomware attack, and because they had run through the exercise, they knew exactly how to quickly respond and recover from the incident.
You never know when or how something will go wrong and being provably prepared is the only way to minimize disruption and downtime.
Sharing is caring
Another interesting question that came up during the hearings is whether Colonial Pipeline participates in an Information Sharing and Analysis Center (ISAC). The company said yes, it does.
Information Sharing and Analysis Centers (ISACs) help critical infrastructure owners and operators protect their facilities, personnel and customers from cyber and physical security threats and other hazards. ISACs collect, analyze and disseminate actionable threat information to their members and provide members with tools to mitigate risks and enhance resiliency.
ISACs are composed of companies operating in the same sector that support each other by sharing important and relevant threat information. While ISACs tend to be mainly focused on critical infrastructure, it doesn’t mean you cannot participate in (or start your own) equivalent group.
For instance, Sophos participates in a number of high profile sharing initiatives, including the CompTIA ISAO, the Global Cyber Alliance (GCA) and the Cyber Threat Alliance (CTA). The goal is to increase resilience against attack, by providing better protection through a collective sharing of information.
These sharing groups don’t even have to be sector-specific, you can also participate in local groups, like the many local DEF CON Groups. We can also leverage advice published by government agencies that have developed guidance based on years of protecting highly sensitive information, like CISA, NCSC and ASD.
The bottom line is, like we’ve experienced in the pandemic, our best prospect at defeating cyber criminals is through an informed and collective effort.
Colonial Pipeline is committed to sharing the full, unredacted Mandiant report upon its completion. Hopefully they will honor that commitment, and we can all learn from their experience.
It rarely pays to pay the ransom
We’re often asked whether it’s okay for companies to pay the ransom and how we can stop this scourge. The answer to both these questions, as it is with so many questions in security is: it’s complicated.
Colonial Pipeline has said it paid the ransom to help the business recover as fast as possible. Unfortunately, many companies find themselves in this scenario and the decision to pay or not to pay is often dictated by many factors.
For instance, there are many examples of companies that were forced to pay a ransom because their backups were corrupt or missing. A simple test could have established the availability and efficacy of those backups.
Others choose to pay to bring the network back online as quickly as possible, and some have opted to pay because it was cheaper than the cost of recovery. Others still choose to pay to avoid having exfiltrated data sold or publicly exposed.
As the cost of ransomware continues to escalate year-over-year, companies are faced with increasing costs in both the ransom amount and recovery fees. That is why having an incident recovery plan and testing its efficacy are so important.
According to Sophos’ 2021 State of Ransomware report, companies that paid the ransom recovered, on average, only 65% of their data. Only 8% of companies managed to recover all their data, and 29% recovered less than half. What’s more, you still have to do the remediation work to address the damage and disruption caused by the attack and ensure this doesn’t happen again.
That said, the decision to pay or not is at the victim’s discretion, but prevention and preparedness can make that decision much clearer.
We are also still struggling with a way to stop ransomware attacks from happening in the first place. One obvious solution is to make it more difficult for criminals to penetrate our networks by deploying advanced prevention and detection technology.
In addition, there’s the question of incentives and deterrence. The financial incentives for ransomware criminals currently outweigh the threat of capture and prosecution. This lack of deterrence and punishment has allowed these criminals to grow bolder in their operations.
To be successful we will need a strong public and private partnership, along with diplomacy and legislation, and an ethical framework that ensures people’s lives aren’t jeopardized. The G7 declaration published on June 13 is a start, but we will have to wait and see how it gets implemented and what the penalties are for non-compliance.
Ask for help
If your organization does not have the internal competencies to effectively manage security and security incidents, there are many partners you can turn to who can help. Part of solving this problem is recognizing which competencies you possess versus those that might best be outsourced.
There has been much talk about a cybersecurity skills shortage that is impeding the ability of businesses to maintain a high level of cybersecurity readiness. The reality is that what’s required to build a solid security foundation doesn’t require a highly trained cyber-ninja. For example, ensuring that all your systems, services and applications are patched is easily done by most competent IT professionals and contributes greatly to your defensive posture.
As the need for more complex skills arise, don’t be afraid to seek help. There are services such as the Sophos Managed Threat Response service that are designed to support organizations who can’t do it for themselves or simply need a little extra help from an elite team of threat hunters.
You don’t have to go at it alone, nor should you.
A path to stronger security
It shouldn’t take an attack for your organization to establish a stronger security posture. Take the time now to assess your position on the security maturity spectrum and act immediately to improve where you can.
In summary, your path to better security starts with:
- Prioritizing security so that everyone in the organization understands their role in maintaining a secure organization
- Providing the security team with the authority and a reasonable budget to achieve their goals
- Employing “secure by default” modes for all your deployments and operations
- Ensuring that you have visibility into every facet of your organization so that you can spot problems before they become full-blown emergencies
- Planning for when you will need to recover from a serious malware attack. Not only will it make you more resilient, but it will also shorten the time and lower the cost of recovery
- Participating in the security community by sharing your successes and failures. Not only will you benefit but you will also be helping others along the way
- If you are a victim, focusing on recovery and remediation, rather than enriching cybercriminals (if it can be avoided)
- Not hesitating to ask for help before you need it. You’ll be glad you did
In the end, the FBI was able to recover some of the bitcoins they paid to DarkSide, which is great news, but it still doesn’t unbreach the network or undo the damage done.
