SSD Cloud Hosting provider, Linode, has forced all customer passwords to be reset after discovering two Linode user credentials on an external machine. It made the announcement on it status page, and in a blog post. All customers will be prompted to set a new password when they next log in.
The user credentials on the external machine were discovered after a security investigation into the unauthorized access of three separate accounts. This implies that there was a security breach at some point that allowed the details to be read from a Linode database. The information found consisted of a user table containing usernames, emails, hashed passwords as well as encrypted two-factor seeds.
Linode could not determine whether the same group or persons was behind the two malicious acts, and they have not been contacted by anyone claiming to be responsible.
It is unclear whether this is related to the DDOS attack they have been suffering since Christmas day, but they did state on the 31st December 2015 that:
They have been proactive in hardening themselves against the majority of the forms of the attack, but the they confirm that the vectors of the attack have changed to adapt to their actions. Linode confirmed that they will be sharing a technical post-Mortem about the incident, but at present, none has been made. It is unclear whether this attack is still ongoing, although the status post is still marked as active.
Having your servers compromised is never ideal, but we must give credit to Linode to proactively contacting their customers, as well as taking steps quickly to ensure that login credentials were reset. They have shared a lot of information about the breach and been very upfront with their customers, which is always a good thing.
Update 11th January 2016
We reached out to Linode with some further questions about the breach, but are still awaiting a response.
In the meantime, however, further information has come to light that indicates that the scale of this breach is much more serious than we initially thought. It appears the breach has been in place since July or earlier (we will clarify this further in a moment, with reference to sources).
Furthermore, with Linode acting as the Data Center partner for some reputable hosting providers, such as WP Engine, the scale of the breach could be significant.
Connection to PagerDuty Breach – July 2015
In a post made on Hacker News on the 5th January 2015, a PagerDuty employee (and ex-Linode Employee) spoke out about a data breach they suffered in July via the Linode Manager. In this case, the hackers used access from the Linode Manager to reset the root passwords on several systems. Fortunately, for PagerDuty, they were alerted to the breach within 60 minutes after the first compromise and deactivated the relevant user account. After investigating the breach, this is what they found:
In our situation the attacker knew one of our user’s passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It’s worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode’s side.
We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise.
Ultimately, PagerDuty, migrated away from Linode due to the breach.
WP Engine – 9th December 2015
We are writing today to let you know that we learned of an exposure involving some of our customers’ credentials. Out of an abundance of caution, we are proactively taking security measures across our entire customer base.
While we have no evidence that the information was used inappropriately, as a precaution, we are invalidating the following five passwords associated with your WP Engine account. This means you will need to reset each of them. Instructions for how to reset these passwords are at the bottom of this email.
In their final update on the matter on the 31st December 2015, they confirmed that the point of entry was through one of their cloud infrastructure partners. Knowing that Linode is one of their partners, and it coinciding broadly within the same timescale as the problems announced by Linode it is possible that the two are related, although we have no independent confirmation of this. It is of course, possible these are completely unrelated.
Update 12th January 2015
You can read Linode’s response to us here.