Blog post January 15, 2020 by Matthijs Koot, Principal Security Specialist & Ralph Moonen, Technical Director at Secura
Blog alleen beschikbaar in het Engels
Severe Citrix Vulnerability Update: Implications and Protection
The dust has not even settled yet from the fallout of the PulseVPN vulnerability in 2019, and the next big one has already started. This time it is in various Citrix products, used by a large number of companies to secure their remote access facilities. An exploit was released before Citrix issued a patch.
The vulnerability is easily exploited and therefore there is urgency in addressing this issue. The Dutch NCSC has evaluated this issue as a 9.8 severity (on a scale to 10) and warnings have been given out by many CSIRT’s and CERT’s. The Dutch newspaper Financieel Dagblad and Trouw reported on it and the Dutch National Cybersecurity Center, the NCSC, warned about the issue.
Who is vulnerable?
Search engines such as Shodan.io and censys.io, together with tools released by the U.S. DHS can help you determine if you are vulnerable. If you are vulnerable, you should immediately apply the mitigation measures as described here: https://support.citrix.com/article/CTX267679.
You would think that Citrix, who were made aware of this vulnerability in 2019, would have rushed to issue a patch. However, as can be seen at https://support.citrix.com/article/CTX267027, patches are scheduled for end of January. And the workaround provided by Citrix is a very bad example of how to inform your customers. Leaving a patch for so long, and not responding to current events for nearly a week, is also extremely questionable behaviour from a major vendor.
If you use Citrix products that are vulnerable, you should be aware that there seem to be more ways to exploit this vulnerability than just the one that the exploit used. Secura scanned the Dutch IP space and found many hundreds of vulnerable servers, including listed companies, government bodies, banks, hospitals and critical infrastructure.
It says a lot, that such companies are dragging their heels with applying the workaround (which was available since December 2019). Apparently, no one cared until it was too late.
What will the fallout be this time?
Without speculating too much, it seems reasonable to assume that over the weekend of 11-12 January, most, or at least a significant portion, of the companies that were vulnerable at that time were compromised. We know this, because monitoring systems showed tens of thousands of requests over the weekend from parties scanning and exploiting the vulnerabilities. Mass-scale automated exploitation is currently being performed by multiple parties. The effect seen until now have been cryptominers installed on server, backdoors installed, credentials harvested and attempts at lateral movement into the victims networks. Please note that Citrix servers play a central role in securing perimeters of infrastructures, and often contain credentials (such as password hashes or encrypted tokens) for accessing other business applications. It is not just the Citrix server that is at risk here, it is potentially the whole internal network and application landscape.
How do I protect myself?
If you applied the fix after the weekend, you still run a risk because attackers have been observed placing backdoors on the servers, that allow the attackers back in at a predetermined time. It is unclear what other attack vectors exist that can be abused due to this vulnerability. But if you have an SSL/TLS-certificate installed on your server, it could very well turn out you also need to generate new keys and purchase a new certificate.
Since the Citrix server can also contain user credentials, it could also mean that all user passwords will need resetting. And in the case of a full compromise it might mean a full reinstallation from scratch.
There are a few ways to check for signs of compromise though. First, the exploit leaves XML files containing the word ‘BLOCK’ in the folder /netscaler/portal/templates. If such files are there, the vulnerability has been exploited to run a command --- or multiple commands --- on your Citrix system. If you're lucky, it was a benign test. If not, you may have been compromised."
However absence of these files can mean that the attacker has cleaned up after him- or herself. In that case, the webserver logfiles should be checked for access to the XML files and the vulnerable perl scripts, for instance using the following commands:
grep -iE 'POST.*.pl HTTP/1.1" 200 ' /var/log/httpaccess.log -A 1 grep -iE 'GET.*.pl HTTP/1.1" 200 ' /var/log/httpaccess.log -A 1 grep -iE 'GET.*.xml HTTP/1.1" 200' /var/log/httpaccess.log -B 1
Other signs of compromise can be found in the bash.log file, in the crontab file and in the process list.
Use:
crontab –l –u nobody
to see cron jobs. There should be no scheduled jobs. Scheduled jobs for user ‘nobody’ are indicative of compromise. Also check all processes of user nobody with:
ps -aux | grep nobody
Additionally, any files modified in the last week are suspicious, and can be found with the command:
find /netscaler -mtime -7 -type f -print0 | xargs -0 /bin/ls –ltr
Unfortunately, the urgency of fixing the issue has not yet permeated through to all companies that are still vulnerable. In the Netherlands, there are at this time, around 500 servers waiting to be exploited by anyone who can run a python script.