In this article, we explain what happened, why we made that decision, and how we protected our customers throughout the incident.
On Wednesday 21 October, a critical security vulnerability known as PHP2Shell was publicly disclosed, prompting Solidi to take its website offline for the first time in over a decade.
Why did Solidi decide to go offline?
On investigation, it quickly became clear to us that the PHP2Shell vulnerabilities were an unusually serious security incident.
Given the potential scale of the problem, we decided to act pre-emptively and take our website offline, in order to protect customer funds.
This was not a decision we took lightly. However, after more than a decade operating in the cryptocurrency industry, we understand that the security of customer assets must always take priority over the availability of our website.
Why was this incident so bad?
It was a zero-day exploit – this means that attackers could potentially break into vulnerable systems before a security fix was available, without needing a password or legitimate access to the system.
The attack vector was hidden within normal web traffic, making it exceptionally difficult to distinguish malicious requests from legitimate customer activity.
The vulnerability existed within PHP itself, potentially allowing malicious requests to execute before the application had an opportunity to inspect them. This made exploitation even harder to detect through conventional application logging and monitoring.
Although subsequent malicious activity might leave evidence on a compromised server, standard intrusion-detection and server-monitoring systems could not necessarily be relied upon to identify the initial attack. The absence of suspicious activity in ordinary web logs was therefore not sufficient reassurance that a system was safe.
The situation was made worse by the fact that information about the vulnerabilities became available to attackers before the official security updates were released. There was consequently a period during which vulnerable systems could potentially be attacked without an effective patch being available.
What is PHP2Shell?
PHP2Shell is the name given to two critical security vulnerabilities discovered in PHP, a programming language used by approximately 70% of the world’s websites.
Successful exploitation of these vulnerabilities could allow attackers to take control of affected servers and steal sensitive data.
The vulnerabilities were each assigned a critical severity score of 9.2 out of 10, reflecting the possibility of unauthenticated remote code execution. Given the widespread use of PHP and the potential consequences of exploitation, PHP2Shell is being regarded as one of the most serious security vulnerabilities in recent years.
More technical information available here:
https://www.akamai.com/blog/security-research/understanding-php2shell-everything-you-need-to-know
How did it affect Solidi?
Solidi uses PHP within several of our older systems, including accounting, customer support, banking integrations, and engineering tooling. Additionally, many of our third-party providers use PHP in their systems, including blogging software, banks, cloud storage platforms, and marketing partners.
This meant that our response needed to consider not only our own infrastructure, but also the potential exposure of systems and services upon which we depend.
What did Solidi do during the 38-hour shutdown?
- Following initial research, we took the decision to disable withdrawals.
- Our security team conducted an in-depth analysis of the problem, the attack vectors, the detection mechanisms, and the long-term implications.
- We determined that this situation was potentially one of the most serious security incidents that we had ever analysed, and concluded that the only correct course of action was to take our website offline, in order to protect customers’ assets.
- We then audited all relevant systems within our platform. All affected systems were patched and upgraded, and in some cases we chose to replace the relevant servers entirely.
- We also accelerated the migration of legacy systems away from older PHP versions, reducing our exposure to future vulnerabilities.
- Once we were satisfied with our security upgrades, we brought the platform back online.
Throughout the shutdown, Solidi remained open for business. Our team continued answering telephone calls and emails, and we executed customer buy and sell orders manually. Although our website was offline, our business remained operational.
Did attackers gain access to Solidi’s systems?
To the best of our knowledge, no.
Customer funds have not been withdrawn. This is the most critical aspect of security for a crypto exchange.
90% of customer funds are held offline in cold storage. Even if an attacker were able to gain access to an internet-facing system, almost all funds would remain safe.
This separation of customer assets from internet-connected infrastructure is a fundamental part of Solidi’s security architecture, and one we have maintained throughout more than a decade of operating in the cryptocurrency industry.
Do customers need to take any action?
No.
Our website has been restored, and customers can continue using Solidi as normal. All customer funds remain safe.
Would Solidi take the same action again?
Security is a constant threat and we remain diligent.
Our cold-storage-first approach is one of our most important defences against the evolving threat landscape on the internet.
However, technology alone is not enough. Security also depends on making the right decisions when faced with uncertainty, even when those decisions cause inconvenience.
Solidi was founded in 2015 and has now operated through more than a decade of extraordinary change in the cryptocurrency industry. You do not build that sort of track record by taking unnecessary risks with customer security.
We are proud of our response to this incident and would not hesitate to take the same action again if necessary. The security of our customers’ funds has always been, and will remain, our highest priority.
We would like to thank our customers for their patience and understanding during the shutdown. We recognise the inconvenience it caused, but we firmly believe that taking the website offline was the responsible decision.
Regards,
StJohn Piano
CTO @ Solidi Exchange
