Site icon Sophos News

Pacific Rim: Learning to eat soup with a knife

A red/brown octopus on a seabed

Embedded architecture devices such as network appliances haven’t historically been top-of-the-backlog when it comes to security features, and during Pacific Rim they became the subject of an escalating arms race – one that blue teamers, and not just those at Sophos, must get a handle on.

The good news is that many of our existing principles transfer extremely well: More recent network appliance technology is based on well-understood OS’s such as Linux variants. The bad news is that some of those principles may need tweaking. While technology has progressed, there is still a high proportion of devices in the field running arcane, security-unaware embedded architectures – sitting on racks collecting dust.

Of course Sophos, as an information-security company, has a dual view of security and response; we respond not only to incidents that affect us as a company, but to incidents that affect our products and services – the “us” that is sent into the wider world. Our incident response processes, therefore, extend beyond our own corporate environment to the very infrastructure we deploy for our customers. It’s a particular kind of double vision, which – we hope – gives us a leg up on thinking about how to evolve incident-response principles to meet current needs.

Actually making the dual-view system work, though, requires close cooperation between the groups that develop our products and the group tasked with responding to security issues concerning them, our Product Security Incident Response Team (PSIRT). Since not all enterprises have (or have need of) a PSIRT, before we dig into our findings, it’s good to explain how our PSIRT operates.

Life in the Sophos PSIRT

Our PSIRT monitors several channels for information about new findings in Sophos products and services. For example, as we mentioned in a recent article which provided transparency into Sophos Intercept X (a follow-up explored our content update architecture), we’ve participated in an external bug bounty program since December 14, 2017 – as it turned out, just short of a year before the first ripples of what became Pacific Rim — and welcome the scrutiny and collaborative opportunities that this brings. Our responsible disclosure policy also offers ‘safe harbor’ for security researchers who disclose findings in good faith. In addition to external reports, we also conduct our own internal testing and open-source monitoring.

When PSIRT gets an incoming security event, the team triages it – confirming, measuring, communicating, and tracking to ensure our response is proportionate, safe, and adequate. If necessary, we escalate issues to our Global Security Operations Centre (GSOC), which is follow-the-sun with over a dozen outposts coordinating on cases 24/7.

Our PSIRT drives remediation, working with our product SMEs to offer technical security guidance, and moving towards resolution alongside response standards – enabling our customers to effectively manage relevant risks in a timely manner. We aim to clearly communicate outcomes in actionable security advisories and comprehensive CVEs – including CVSS scores, and Common Weakness Enumeration (CWE) and Common Attack Pattern Enumeration and Classification (CAPEC) information.

In addition to being just generally best PSIRT practice, this all factors into our commitment to CISA’s Secure by Design initiative. In fact, Sophos was one of the first organizations to commit to the initiative’s pledge, and you can see details of our specific pledges here. (An essay from our CEO, Joe Levy dives deeply into our commitment to Secure by Design and how, with everything we learned from Pacific Rim, we mean to carry that commitment forward.)

Of course, a good PSIRT doesn’t just wait for reports to come to it. In the background, as well as performing its own testing and research, the team also works to mature our product security standards, frameworks, and guidelines; perform root cause analyses; and continuously improve our processes based on feedback from both internal and external stakeholders.

All these tasks inform what we’ll discuss in the rest of this article, as we break down what we learned from iterating and improving our processes over the life of Pacific Rim. We’ll talk about principles – many of which we have implemented or are in the process of implementing ourselves – as a starting point for a longer conversation among practitioners about what effective and scalable response looks like when it comes to network appliances.

What we learned

Telemetry

It all starts with being able to capture state and changes on the device itself. Network appliances can often be overlooked as devices in their own right, as their usual role is as “invisible” carriers of network traffic. However, this distinction is an important step to provide observability on the device – essential for response.

Key challenges:

Streaming

Whether the detection occurs on the device or in a back-end data lake (more on that below), there will inevitably be a point at which the acquired telemetry should be sent off the device. While many of these principles are well-documented for the security monitoring field, there are some unique challenges for network appliances.

Key challenges:

Triggers, tripwires, and detections

The next stage is discerning signal from noise. As cybersecurity specialists, we are often taught to look for the absence of the normal and the presence of the abnormal – but the definition of both varies widely in network appliances.

Key challenges:

Response actions

We’re talking about core network infrastructure, which does not respond well to aggressive tactics. While on a user endpoint we may think nothing of terminating a suspected rogue process or isolating a device from a network, doing either on a network appliance could have catastrophic availability impacts to a user network. In our experience, at this stage some firm guardrails, setting expectations and stopping response activity from making the incident worse, were tremendously helpful.

Key challenges:

Conclusion

Necessity is the mother of invention, and it is fair to say that Pacific Rim has shown us that there is more to do in the field of incident response for network appliances. The application of these basic principles has allowed us to protect our customers to a level that we never thought possible, but it has also identified some important limitations that practitioners need to address – some in their own organizations, some in-house at each vendor, some industry-wide. Topics such as network availability, data privacy, and limits of liability, when it comes to response actions, require not only technical but commercial and legal frameworks. Difficult as these topics may be to discuss, let alone implement, it is a conversation we must entertain in multiple venues if we are to keep up with the evolution of these threats.

Sophos X-Ops is happy to collaborate with others and share additional detailed IOCs on a case-by-case basis. Contact us via pacific_rim[@]sophos.com.

For the full story, please see our landing page: Sophos Pacific Rim: Sophos defensive and counter-offensive operation with nation-state adversaries in China.

Exit mobile version