On December 8, FireEye reported that it had been compromised in a sophisticated supply chain attack: more specifically through the SolarWinds Orion IT monitoring and management software. The attackers leveraged business software updates in order to distribute a malware named SUNBURST, and then used this foothold in the organization to contact their Command & Control (C&C) infrastructure, move laterally using different legitimate credentials, and steal data from the network.
Not long after the attack, it was uncovered that this was a global campaign that affected both private and public high-value entities in North America, Europe, Asia, and the Middle East, including the US Treasury and Commerce departments, along with other US government agencies. The overall attack has been nicknamed “Solorigate” by Microsoft.
SolarWinds Orion software is quite popular, as it is estimated that 425 of US Fortune 500 companies use SolarWinds software to monitor their networks. Although not all versions of the SolarWinds Orion product are affected (only the versions released between March and May 2020), 18,000 of its more than 300,000 customers may have installed a software patch enabling the attack. It is still too early to know how many systems were actually hacked.
Fortunately, there is already a hotfix released in order to patch the affected systems. Network admins in every company need to quickly identify these servers, isolate them from network, and patch them. Perhaps you work on a team that has purposefully built solutions by combining different tools to manage and secure your infrastructure. Defense in depth, anyone? Have you ever installed a “free trial” on a device or spun up a virtual machine (VM) for quick testing and then forgotten about it? Or did a previous network administrator ever deploy a server without really documenting or letting leaders know it exists in the network? You cannot patch what you don't know exists in your network.
For further research, you can consult the following article:
https://blog.talosintelligence.com/2020/12/solarwinds-supplychain-coverage.html
Discovering SolarWinds Orion servers and other NMS within your network
Cisco® Secure Network Analytics (formerly Stealthwatch) and its software-as-a-service (SaaS) version, Secure Cloud Analytics (formerly Stealthwatch Cloud) can help you uncover rogue or forgotten servers in your network that, if left unmonitored, could leave an open doorway for attackers. SolarWinds Orion servers, like many Network Management Stations (NMS), monitor the health and performance of the network in real time, using a combination of tools and protocols. The most common approach is to do SNMP polling from the NMS to all infrastructure devices in your network. It is typical to see SNMP notification from these infrastructure devices sent to the NMS servers. It’s this behavior that we are going to target when looking for them on the network.
In Secure Network Analytics, perform a Top Host search with the following parameters:
Search Type: Top Host
Time Range: Last 30 Days
Subject Host Groups: Inside Hosts
Advanced Options Order By: Flows
Connection Port/Protocol: 161/udp; 162/udp
You will now see a report with all the servers that have been using SNMP during the last 30 days. Focus on the “Host bytes ratio” column. Hosts with 0% indicate clients are trying to reach them, but the devices are not answering (they could have been decommissioned already). For the other servers, investigate all of them, as they are potential SolarWinds Orion servers in your network.
“You cannot patch what you don't know exists in your network.”
Figure 1. Secure Network Analytics Top Hosts Search that uncovers top SNMP servers in your network.
In Secure Cloud Analytics, you can look for the following indicators on your network, related to SolarWinds Orion servers:
- The “New SNMP Sweep” alert will have fired if a server has been attempting to reach a large number of hosts using SNMP. Start a search in Monitor -> Alerts using the name of the alert, and search over the last month.
- The “IP scanner” observation triggers when a device is seen on the network scanning a large number of entities. It can be included in the “New SNMP Sweep” as evidence, among other alerts (like “Outbound SMB spike” or “NetBios connection spike”). Look for this type of observation in your network.
Once you have identified the mentioned alert or observation, you can investigate all the servers related to them, as they are potentially SolarWinds Orion servers.
Figure 2. Secure Cloud Analytics "New SNMP Sweep" alert can warn of the presence of SolarWinds Orion servers. It's enabled by default.
Detecting malicious behaviors
If you were able to find any compromised servers in the network using the above methods, it’s imperative that you patch them with the designated hotfix. After that, the next logical step is to assess if any malicious or suspicious activity has already been taking place in your network. There are a variety of common patterns that have been spotted in SUNBURST variants. If you were monitoring your network with Secure Network Analytics or Secure Cloud Analytics before the attack started, there should have been some signs of suspicious activity that would have surfaced in the form of alerts.
Both products are capable of detecting a range of suspicious activities that are commonly seen in an advanced cyber attack with the purpose of stealing data, such as C&C connections, lateral movement, and data exfiltration. As a result, you will be able to detect other global campaigns, even before they make it to the news, and the IoCs are shared.
These are the alerts you should pay special attention to, in relation to the SolarWinds Orion compromise. You can search for them in your deployment in the last few months:
Alerts in Secure Network Analytics:
- “Exfiltration” alerts, such as “Suspect Data Loss” will trigger if there is an abnormal amount of data being transferred from an inside host to the outside. The ultimate goal in many instances of the SUNBURST attack has been the appropriation of highly valuable data.
- The “High SMB Peers” alert will be a sign of a compromised host, as once the attacker has gained access to the network with legitimate but stolen credentials, it will try to move to other hosts.
Alerts and Observations in Secure Cloud Analytics:
In your portal, review your alert priorities page to ensure you have the desired alerts enabled and appropriately prioritized. The alerts and observations below can help you identify a variety of tactics used by advanced attackers during this campaign.
- “Talos Intelligence Watchlist Hits” will trigger when a device exchanged a significant amount of traffic with multiple addresses on the integrated Cisco Talos® IP Watchlist. As Talos has already identified the IoCs used by the attackers to connect with their threat infrastructure, any traffic to those domains will be flagged.
- “Repeated Watchlist Communications” is an alert that will, in a similar manner, alert on any communication with IoCs as defined in the Cisco Talos IP Watchlist, if a device has established periodic connections with any watchlisted IP (either in a user-defined or integrated watchlist). This alert uses the Watchlist Interaction and Heartbeat observations.
- “Watchlist Interaction Observation” is an observation that includes any interaction with integrated and user-defined watchlists. Since this attack is often low and slow, these alerts may not be triggered by the attacker activity. Review recent observations of this type, narrowing the observations by searching for Talos (and any user-defined watchlists that include IOCs for this campaign).
- The “Suspicious Domain Lookup Failures” alert will be seen if a device tried to resolve multiple algorithmically generated domains (for example, rgkte-hdvj.cc) to an IP address. As Talos explains in its analysis, "The backdoor identifies its command and control (C2) server using a domain-generated algorithm (DGA) to construct and resolve a subdomain of avsvmcloud[.]com.”
- The “Abnormal User” alert will detect users' abnormal activity like a user session that was created on an endpoint that does not normally see sessions with this user. This alert uses the Session Opened observation and requires an integration with either AWS, Sumo Logic, or Active Directory.
- “Anomalous Windows/Mac Workstation” will detect abnormal behavior activity either on a Mac or Windows.
- The “AWS API Watchlist IP Hit” alert will fire if the malware tries to connect to any IoC domain already identified as malicious inside AWS. This alert uses the AWS API Watchlist access observation.
- “Azure Activity Log IP Watchlist Hit” is a similar alert for Azure. In this case, the event will be detected on the Azure Activity Log, and was initiated by an IP address that matched a user-defined or an integrated watchlist. This alert uses the Azure Unusual Activity observation.
- “New External Connection” is an alert that detects that a device normally doesn't talk to external devices but has done so recently. This alert uses the New External Connection observation, and it could detect lateral movement when the attackers leverage the hosts in the network.
- “Unusual External Server” is an alert that highlights when a device has repeatedly communicated with a new external server. This alert uses the New External Server and Persistent External Server observations and can detect communication with the C&C server.
- “Domain Generation Algorithm Successful Lookup” is an alert that will trigger when a device succeeded in resolving an algorithmically generated domain (for example, rgkte-hdvj.cc) to an IP address. This alert uses the Domain Generation Algorithm Success observation and may indicate a malware infection or botnet activity. We know that attackers were using these types of domains in the attack, and hence this is a way to detect them.
- The “Outbound Traffic Spike” alert signals when a device sends a much larger amount of traffic to external destinations than usual. This alert uses the Historical Outlier observation and the New Large Connection (External) observation and may indicate data exfiltration.
- “Potential Database Exfiltration” will identify a statistically unusual amount of data transferred from a database server to a client. This alert uses the New High Throughput Connection observation and may indicate data exfiltration.
- “Potential Data Exfiltration” will trigger an alert when a device downloaded data from an internal device that it doesn't communicate with regularly. Shortly after that, the device uploaded a similar amount of data to an external device. This alert uses the Potential Data Forwarding observation and may indicate that sensitive data is compromised.
- “Static Device Connection Deviation” alert focus on devices that are normally static on the network, talking to the same devices, with a similar traffic pattern each time. This detection will be triggered if these devices deviate from this behavior, including communicating with a new external host. This alert uses the Historical Outlier and New External Connection observations and may indicate misuse or a compromise.
- “Static Device Deviation” alert focus on devices that are normally static on the network, talking on the same ports, or to the same devices, with a similar traffic pattern each day. If the device starts behaving differently, the alert will trigger. This alert uses the Historical Outlier observation and may indicate misuse or a compromise.
Additional proactive actions for further protection
Now that you have searched for and identified potentially compromised servers and had a look at detections that alert on malicious behavior in the network that might be associated with the attack, you can go ahead and define a set of actions that will further protect your organization, and also allow for automated response.
Additional actions in Secure Network Analytics:
- You can create two host groups in order to track the SolarWinds Orion servers in your environment and the respective legitimate servers they communicate with. The first one would be a subgroup of “internal hosts” and the second one would be a top-level host group (not inside or outside). It has already been explained in the “Discovering SolarWinds Orion servers and other NMS within your network” section how to find SolarWinds Orion Servers. In order to find legitimate servers that they communicate with, we can use the “top peers” report.
After you have both host groups defined and all the relevant servers classified in one of those two groups, you can create a Custom Security Event (CSE) in order to alert on any traffic from your SolarWinds Orion servers that is not going to your already defined legitimate peers. Additionally, you can define an Identity Services Engine (ISE) Adaptive Network Control (ANC) policy in order to quarantine servers that are communicating with peers who are not allowed.
Figure 3. Adaptive Network Control (ANC) policies allow you to automatically quarantine SolarWinds servers that are communicating with nonlegitimate servers.
- Additionally, it is always recommended to actively segment your network. As a network administrator, you know better than anyone who should be talking with who. In order to do this, you can use Custom Security Events (CSE) in Secure Network Analytics, and the Internal Communications Watchlist in Secure Cloud Analytics in order to detect lateral movement.
In summary
In this article, we have identified how to look for compromised SolarWinds Orion servers that might be installed in the network but not documented. We have also reviewed which signs of compromise in the form of alerts you can look for in both Secure Network Analytics and Secure Cloud Analytics as well as ways of responding to this threat and proactively protecting your network further, in the event of other cyber attacks.