Web hosting provider, 123-reg, has deleted many customer websites situated on their VPS platform. The error was made during maintenance on the servers, and while attempts are being made to recover the data, some websites have been lost.
123-reg is a major UK web hosting provider, and part of HEG, the largest privately-owned hosting company in Europe. The Group also includes brands such as Webfusion, Heart Internet, Host Europe and Hosteurope Spain. 123-reg currently host over 3 million domain names, and over 1.7 websites. It is unclear how many websites are hosted on their VPS platform.
According to their status page (archived link), the error was first reported on the 16th April 2016, with the latest updates indicating data loss and the attempted recovery of data using various data recovery tools.
The update on the 18th April was quite revealing:
We are currently working on restoring your VPS packages using data recovery tools; in other words, we are attempting to recover your packages bit by bit, while checking data integrity at the same time.
This in itself is quite alarming, especially as 123-reg claim to offer VPS backups as part of their service. Whatever caused the data loss, it appears to be more widespread that just a few servers, and possibly include their backup servers as well. According to one user that emailed us, “the fault was limited to 67 servers out of 115,000 across Europe”, but that could easily affect several hundred, if not a few thousand websites.
On the 19th April, the wording of the updates worsens. It is evident there are issues relating to some or all of the backups and that they are using third-party tools to try and recover deleted data.
Our system engineers are testing multiple third party data recovery software and we are definitely seeing some results. As this is referring solely to data recovery, the next thing on our list is actually reimaging the hosts with the data we have retrieved (aka getting the VPS packages back online).
The reimaging process will take some time as well (we’re provisioning new hosts, on new servers), however, things will get back on track for part of the customers pretty soon.
On a plus side, 123-reg confirmed that the issues did not extend to any other services, including Premium Hosting. However, as you will see later in this article, 123-reg has already started informing customers that their data has been lost.
What Caused The Data Loss?
According to 123-reg, the problem occurred when they were performing a “cleanup” operation on its VPS servers when an error in the code used within its software “effectively deleted” some customer websites.
123-reg has not clarified why this cleanup operation affected both the VPS servers themselves and backup servers, or whether they had other issues affecting their backups.
Some insight as to the exact cause of the issue was made a user at inn-master.
At 7 am on Saturday morning, someone at our hosting provider ran a script that had a catastrophic error in it. The result was that the script deleted the servers and websites of all their customers! This is a major embarrassment for a large web hosting group and judging by what we have seen on Twitter it has left some of their customers who had no backup or emergency plan with no website and potentially no business any more! – We can assure you, we are not one of those!
Of course, inn-master are a company that had a disaster recovery plan in place and are already up and running on their backup server.
To get an idea of how such an error could occur, there was an interesting story on server fault a few days before this incident, where an error in a script was able to delete all files on multiple servers, including backups. While the story has now been proven to be fake, the concept is sound, and should serve as another warning to ensure you have your own off-site backups.
Whole Businesses Lost
The implications of lost data can be significant for small business, especially if those businesses were relying on the host for backups (we always recommend having your own off-site backups). While 123-reg are trying to recover the deleted files, it is becoming clear that in many cases they are not successful.
Daniel Naracci, one of many complaints on Twitter, posted “@123reg HAVE LOST MY ENTIRE BUSINESS” with a link to an email showing 123-reg confirming that his website has been lost:
— Daniel Naracci (@DanielNaracci) April 18, 2016
The text of the email you see in the above Tweet is as follows:
I am very sorry to say that we have not been able to recover your VPS image, and it looks like we will not be able to. he specialist team of data scientists will continue to work on recovery, but the outcome does not look positive at this point. will update you when I have a final answer and the team have exhausted all options available. understand that this VPS is likely a key part of your business and offer my strongest apologies.
One of the first customers to complain about the “wrecking” of their business is @itsapuzzlething, who stated “this should hit the news. Just to make others aware. This will wreck my business & plenty of others I am sure”:
— Its a Puzzle Thing (@itsapuzzlething) April 18, 2016
It seems that the email is being sent out to other customers on Monday, but we do not know exactly how extensive the problem of lost data is:
— Weighton Coin (@WeightonCoin) April 18, 2016
Finally, we have received a copy of some correspondence by a user of 123-reg (who has asked not to be named), which sheds some light on the real problems that are being caused for some customers.
Two emails from the customer:
From: unnamed customer
Date: Mon, Apr 18, 2016 at 9:59 PM
To: SMB VPS TEAM 123 <[email protected]>
You are kidding me? You have lost my business> and all i get is an apology.????? I want compensating I want the money i have invested paid back to me. This is not fair you have lost everything my entire business up in smoke.
Anger is nothing compared to what I am feeling now I am financially destroyed and out of business I want compensating for the cost of putting my website back to where it was. The custom module has cost me thousands of pounds to design and implement on my website.
You have lost everything my entire livelihood
From: unnamed customer
Date: Tue, Apr 19, 2016 at 12:51 AM
To: SMB VPS TEAM 123 <[email protected]>
I justed wanted to add to my previous message. I am aware that this message will be read by someone at 123reg who is paid to just perhaps reply to messages, instructed by the people above them. This message may never see anyone as high as the directors of the company but I feel even if it gets through to just one employee of 123reg then it will be worth it.
I have spent 3 years of my life trying to get a better life for me and my family. After many set backs and 3 years of 16 hours shifts to save up money, to save enough money to get my own business so I could start trying to get a better life for my family. There where so many days that I thought I just could not continue, 3 years of working and not seeing my kids grow up because I wanted a better life for them. I had given everything I had to the business and sacrificed my own health and time because I knew in the end it would pay off and I could once the hard work had been done enjoy a better life with my family.
I am fairly new VPS customer to 123 reg but you see the work that was installed on your VPS was put together by several developers 3 of them are based in India and they worked purely on the website last month costing 6k in total to get it ready for launch including a custom plugin that now can never be recovered. In short I have a backup when I first started that is a shell of the website that I had. It has no module, no cart, no text, no custom pages, no products, no categories, no landing pages in short it is just a template of magento.
In one swoop and massive failure on your part that should have never happened, you have lost my entire business, years of savings and sacrifice. I am not just financially ruined but completely heart broken I wish I had spent that time seeing my kids grow up instead.
123reg will continue and you will continue making large profit and this will just be a blip in time, and I do not want you to suffer because you employ staff who depend on you for a living, however for me it is game over. Come Tuesday I start looking for a job.
You have ruined my dream of a better life and have taken so much from me, for you it just some data but for me it is 3 years of my life. I hope you learn from this that money isn’t everything and treating people with respect by properly looking after the people that pay you and trust you to make sure their livelihoods are in good hands is worth so much more than making a few extra pounds by cutting corners. You can if you like recite your T&C’s that clearly are set out to make sure you do not need to pay compensation but I feel you have an obligation to help your customers get back some of what they have invested and lost.
The reply from 123-reg:
Before we post the reply, we are really surprised they are trying to relate what has happened to how hard they themselves have worked building up 123-reg. We are not sure this is a reply that will provide any ounce of comfort to a user that has lost all their data:
From: SMB VPS TEAM 123 <[email protected]>
Date: Tue, Apr 19, 2016 at 2:24 AM
To: unnamed customer
Hi (unnamed), you are wrong. I have been working on responding personally myself since the weekend, and late in to the nights. I do care, I am reading what you are saying.
I truly do understand and feel your situation. I have spent years working 16 hour days to get where I am today, and this may indeed break me as well. What we do as a business has a big impact on our customers, as well as us as individuals and our colleagues, as well as your employees, and your family.
There are many things that have gone wrong to put us where we are now over time, from both the platform, to the teams managing it and everything else between.
I also feel that the last six years where I have worked extremely hard to build and grow 123 Reg, improving the products and services we offer, building our support teams, updating products and many other areas is somewhat wasted, today you are, and will never be, a fan of 123 Reg. All the good things that have changed make no difference when we have failed you like this.
I can only apologise, and promise you that I will continue to work forward to ensure that we limit things like this happening again in the future. It is too late to save you, but I do care about our customers, and I want to ensure that I can assist with each customer being successful.
I am sorry. Richard.
What Compensation is Available?
Unfortunately, most web hosting providers have fairly broad terms and conditions that limit their liability for any errors or disruption to their service. However, there is hope for minor reimbursement, on the basis that 123-reg did not take reasonable care and skill that ultimately caused the data loss. The amount of reimbursement, though, shall not exceed 1.25 times the amount the customer paid to 123-reg in the 12-months before the incident. Failure by 123-reg to adhere to this reimbursement could make them open to further liabilities. We are not lawyers, so you should seek professional legal advice rather than relying on our interpretation.
The relevant parts of the terms and conditions are as follows:
“Services” — those development, implementation, consultancy, hosting and other services (if any) provided to the Client pursuant to the Agreement, as described on a relevant Order Confirmation, together with any Support Services and Domain Services
4.3 — 123-reg warrants that it will provide the Services with reasonable care and skill. 123-reg will not be liable for a breach of such warranty unless the Client notifies 123-reg in writing of such failure within 14 days of the Client becoming aware of the failure.
4.4 — If the Client makes a valid claim against 123-reg based on a failure by 123-reg to comply with the warranty set out in clause 4.3 123-reg may, at its option, take such steps as it deems necessary to remedy such failure or refund such part of the Fees as relates to such Services, provided that the liability of 123-reg under such warranty will in no event exceed one and a quarter times the amount of the Fees paid to 123-reg by the Client (excluding VAT and expenses) in the 12 month period before the date on which the Client makes the claim. If 123-reg complies with this clause, it will have no further liability for a breach of the said warranty.
As you can imagine, with more expensive VPS servers being affected by the data loss, the liability to 123-reg could be significant.
What Lessons Should Be Learned By This?
Ultimately, users should not rely on their web host for disaster backups. There are many different off-site backup services available, with some working on an application level (VaultPress \ Codeguard), and others that can backup whole VPS or cloud images to places like Amazon S3.
With the cost of lost business, replacement website and lost content it seems sensible to protect yourself for around $5-$10 per month by making regular off-site backups. If your business is important to you, we see no reason not to implement such contingency planning. The important thing is to ensure the backups are stored off-site and that they are under your control (i.e. not your web host). If a similar situation should occur, you will have the option to move to a new hosting provider, and get back up and running without delay.