Fox-IT has uncovered a large-scale exploitation campaign of Citrix NetScalers in a joint effort with the Dutch Institute of Vulnerability Disclosure (DIVD). Over 1900 instances were found to have a backdoor in the form of a web shell. These backdoored NetScalers can be taken over at will by an attacker, even when they have been patched and rebooted.
A web shell is a malicious script used by an attacker with the intent to escalate and maintain persistent access on an already compromised web application. The scripts are placed on internet-facing servers and devices so they can be reached remotely.
In July, the Cybersecurity and Infrastructure Security Agency (CISA) added a critical unauthenticated remote code execution (RCE) vulnerability in Citrix NetScaler ADC and Citrix NetScaler Gateway to its Known Exploited Vulnerabilities Catalog, based on evidence of active exploitation.
The Common Vulnerabilities and Exposures (CVE) database lists publicly disclosed computer security flaws. The CVE that the cybercriminals used to plant the backdoor is listed as:
CVE-2023-3519 (CVSS score 9.8 out of 10): a Citrix NetScaler ADC and NetScaler Gateway code injection vulnerability. The vulnerability can lead to unauthenticated RCE. It affects appliances configured as a Gateway (VPN virtual server, ICA Proxy, CVPN, RDP Proxy) or as an authentication, authorization and accounting (AAA) virtual server.
Fox-IT (in collaboration with the Dutch Institute of Vulnerability Disclosure) scanned for the web shells to identify compromised systems. As of August 14th, 1828 NetScalers remain backdoored, 1248 of those have been patched but still remain vulnerable. So, it seems that many administrators saw the need to patch for the vulnerability, but didn’t realize that patching was not enough to deal with an already established backdoor.
Several factors indicate that the biggest part of this exploitation campaign took place between late July 20th and early July 21st. Some systems have been compromised with multiple web shells. In total, the scans revealed 2491 web shells on a total of 1952 compromised NetScalers.
The campaign was likely targeted at European organizations. Of the top five affected countries, only one is located outside of Europe, in Japan. Germany alone accounts for over 500 backdoored instances.
On August 10, 2023, the DIVD started reaching out to organizations affected by the web shell. It used its already existing network and responsible disclosure methods to notify network owners and national CERTs. There is no reason to wait for such a notification however.
Prevention, detection and response
If your Citrix server hasn’t been updated to a secure version, we strongly advise you to patch it as soon as possible, especially if you’re utilizing any of the following features:
- SSL VPN
- ICA Proxy
- CVPN
- RDP Proxy
- AAA virtual server
If you are not using one of these servers, we still recommend that you patch to a non-vulnerable version to prevent your appliance from becoming vulnerable when you start using one of these functions in the future.
Regardless of whether and when the patch was applied, it is recommended that you perform an Indicator of Compromise check on your NetScalers.
There are several resources available that document the in-the-wild exploitation of Citrix appliances where forensic artifacts can be found:
- https://www.shadowserver.org/news/technical-summary-of-observed-citrix-cve-2023-3519-incidents/
- https://www.mandiant.com/resources/blog/citrix-zero-day-espionage
- https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-201a
- https://support.citrix.com/article/CTX561482/citrix-adc-and-citrix-gateway-security-bulletin-for-cve20233519-cve20233466-cve20233467
- Fox-IT has provided a Python script that utilizes Dissect to perform triage on forensic images of NetScalers.
- Mandiant has provided a bash-script to check for Indicators of Compromise on live systems. Be aware that if this script is run twice, it will yield false positive results as certain searches get written into the NetScaler logs whenever the script is run.
If you find that your Citrix NetScaler has been compromised, make sure to set up a clean system from scratch, or at the very least backup/restore from a safe snapshot. But first, or from a forensic copy of both the disk and the memory of the appliance, investigate whether the backdoor has been used by the attackers. Usage of the web shell should be visible in the NetScaler access logs. If there are indications that the web shell has been used to perform unauthorized activities, it’s essential to perform a larger investigation, to see whether the adversary has successfully taken steps to move laterally from the NetScaler.
We don’t just report on vulnerabilities—we identify them, and prioritize action.
Cybersecurity risks should never spread beyond a headline. Keep vulnerabilities in tow by using Malwarebytes Vulnerability and Patch Management.