Major European cloud host "OVH" loses a datacenter to fire (March 10, 2021)

[winnielinnie]

Millions of websites offline after fire at French cloud services firm

"A fire at a French cloud services firm has disrupted millions of websites, knocking out government agencies’ portals, banks, shops, news websites and taking out a chunk of the .FR web space, according to internet monitors."

[...]

"Europe’s largest cloud services provider told clients [...] to activate their disaster recovery plans following the blaze."

I was wondering why some websites and downloads were acting wonky this morning, until I noticed a few homepages posting a picture of a building on fire.

Take fore example the home page of ClipConverter.cc, today on March 11, 2021.

OVHcloud datacenter in Strasbourg, France



I shall cover my TrueNAS server in a layer of asbestos. No fire can get to my stuff! (/sarcasm)

Another reason why DIYers prefer to hold physical responsibility for their own data. My emergency setup is not bullet-proof, but I do have my data physically scattered at locations only accessible to myself. (The poor person's approach of actual drives, not using remote NAS servers.)

We are the small, fast, and agile deck boats of the sea; while enterprises captain the cruise ships.

 

[SecCon]

Yeah, seen this as well. Unbelievable there was no offsite backup for some sites.

 

[c77dk]

Yeah, seen this as well. Unbelievable there was no offsite backup for some sites.

A lot of times you'll see the C*-layer people only regarding backup as an expense and won't pay for it. When something like this happens, they're all "why are there no backup??!", and rarely admits they didn't want to pay for it.

 

[SecCon]

I know backup is expensive, or can be. I am betting on a number of lawyers getting involved very soon.

 

[Arwen]

Some companies that supply critical infrastructure or services, (like my last 2 jobs), perform disaster recovery exercises, quite regularly. Even during the pandemic, (which weirdly enough, went easier and with fewer problems than prior DREs). One company even has a local DR site, in case of local issues, (like building fire). But, they also have a geographically isolated DR site, in case of something more major.

 

[danb35]

I saw this story on another forum, and it hit kind of close to home--my personal web/mail server is hosted with a VPS, also in Europe, though with a different provider. I think it shows that my backup regimen isn't too paranoid--I back up to one cloud provider (different host, in a different datacenter, on a different continent), and also to my TrueNAS box, and have a local hot-synced backup server that I can bring online in a matter of minutes if I need to. I had occasionally wondered if that was excessive. Not any more.

 

[winnielinnie]

and have a local hot-synced backup server that I can bring online in a matter of minutes if I need to

Can you explain this one? Is it that you make occasional backups to it (from your TrueNAS box?), but it stays offline most of the time? I might have read that wrong, and you're saying that it serves as a continuously updated replica of your TrueNAS box that can "take its place" if/when necessary?

My approach is way less sophisticated. I use external drives (of previously replicated snapshots) that I keep in an air-conditioned storage unit at a separate building.

 

[danb35]

Can you explain this one?

It's a feature of the server software I use (Nethserver)--I have an installation in the VPS, and a bare-metal installation at home. The former syncs to the latter every 15 minutes.

Now, if I wanted a hot-synced TrueNAS box, replication should handle that--but that isn't what I meant.

 

[jgreco]

People forget that "the cloud" isn't some magical thing, but really just someone else's compute resources. You have no control and little (if any) visibility into the viability of their business plans, the competence of their staff, the security of their facility, the stability of their location, etc.

When you buy that "cheap cloud VM", be very aware that if you don't also buy the backup service that goes along with it, which might cost more than the VM itself, your stuff is just sitting there on the cheapest PoS (that's not point of sale, kids) that they could make work sorta reliably at scale. In the cloud business, the answer to a hardware failure isn't to dispatch someone to repair the broken ${thing}, it's just to forget about it, maybe power the ${thing} down, and get on with life. It is highly unlikely anyone is going to try to rescue your stuff off of that cheap $5/month VM.

One of my data centers is in the flight path of Dulles International (Equinix DCx). Another is in the flight path of O'Hare International (DFT CH1). A third is in the Bay Area and will probably bite it if the big one hits. A fourth one had wildfire within a mile of it last year. It isn't worth trying to find the fully fortified solution to this, because even if you find that magic data center that's built like Fort Knox, who knows if the business plan of the operators is stable and sane.

If you want uptime, assume that your ${thingy} is going to go away when someone inadvertently starts flooding the server room with diesel when a generator tank refill goes horribly awry. Make your stuff so that it is served from multiple data centers, or, at least, have a warm standby site ready to take over, kept continuously up-to-date.

 

[Arwen]

One place I worked had hardened data centers. Not too hardened, but the local ones were in Tornado Alley and would generally survive intact. (Not saying that the external infrastructure would though...)

When I worked at Storage Tek, DC4 was in a tough building. Formerly manufacturing space, so the floors could handle very heavy weight. And lots of vibration without failing. It also used halon :-(. I kept complaining, (and not too seriously), that they did not have the cute oxygen masks used in Terminator II. That DC4 was small enough to easily hold your breath before getting to a door. (Unlike another place I worked, that had 70,000sq ft of drop floor... little chance of holding your breath to get to a door, if you were in the middle.)

Of course, it does not help a hardened data center, with good fire protection, if an outside contractor, (for Google, the now assumed to be evil empire , cuts your main fiber cables. During Christmas time. And in snow. The side benefit from that was it identified a network path flaw. Because of that, they implemented a backup path, (which physically existed, just was not part of a disaster recovery plan).

When Storage Tek was bought by Sun Microsystems, (I was still at STK when this occurred), they modified a 4 office story building with lab / data center basement, into mostly lab & some data center space. I kept worrying that I would show up one day and see the upper 4 floors pancaked into the basement. But, 15 years later, the building is still standing. So I guess they did both the engineering design and the contruction retro fit correctly.


Note: Google used to have a motto, "Don't be evil". They dropped it a few years ago. So, my assumption is that they are evil now.

 

[Patrick M. Hausen]

Largest data centre seems to be slightly exaggerated. The facility rated at 2 Megawatts energy consumption.

Kristian Köhntopp did a rough but plausible calculation of their hardware and client structure based on publicly available figures. Looks like mostly dedicated server workloads and less VM/container:

https://twitter.com/isotopp/status/1371460159881494533?s=21

Some more background info on data centres by Kris:

Rechenzentren und ihren Stromverbrauch regulieren

blog.koehntopp.info

(a handful of links to good resources in English in that article)

 

[SecCon]

I have no first hand experience with datacenters, but when I recently worked at this big car brands local HQ I was part of their disaster plan and had some infrastructure responsibility. From day one we had two server locations at two different datacenters kilometers apart. Sure, we kinda started from scratch so we could make those decisions and we were independent in that way but in recent years pressure is on to consolidate to more efficient and cost effective mega big centers that may have internal redundancy but only on the same location, as far as I know.
We saved a big buck, but the overall cost was never ours in the first place, and we lost flexibility and control. Now all the infrastructure guys do are filling forms and submitting tickets to EDC, and not very well handled either.

As for my own little turf, well I have a 14TB NAS in my SOHO and a 20TB FS Server in my garage. I think the house can burn to the ground and the garage server should survive, there are a few significant meters and walls in between, but sure, I should get a real off site place.

 

[winnielinnie]

Largest data centre seems to be slightly exaggerated. The facility rated at 2 Megawatts energy consumption.

There's comma in the thread title. I tried to format it like news article headline. I was referring to OVH (as the largest cloud services provider in Europe.)

 

[jgreco]

If you don't mind well-intentioned criticism, I would agree that the title is a rocky read.

Largest cloud services firm in Europe, datacenter destroyed by fire (March 10, 2021)

Maybe

Major European cloud host "OVH" loses a datacenter to fire (March 10, 2021)

 

[jgreco]

And actually if we were looking for truth-in-advertising, I don't think OVH is the largest. Certainly no one seems to think so.

https://www.fiercetelecom.com/opera...microsoft-reign-supreme-european-cloud-market

etc

I remember when they took over the vCloud Air business that they were hungry and eyeing ways to get market share, and were considered a bit of an unusual player for the purchase.

 

[winnielinnie]

Done and done. However, I took Reuters at face value when they wrote in the article "Europe’s largest cloud services provider".

 

[jgreco]

Done and done. However, I took Reuters at face value when they wrote in the article "Europe’s largest cloud services provider".

Well, and Reuters probably took the PR flak from OVH at face value.

I'm sure by some measure they are Europe's largest cloud services provider.

Perhaps they have the largest number of e-mail spam spewing nodes in Europe. The NANOG discussion seems to support that.

Perhaps they have the largest number of randomly spark-emitting cheap Asian motherboards. That might explain the fire.

Who knows.

 

[SecCon]

Never heard of them before , well, I do recognize OVH, but not as a datacenter acronym...

tried to find a list, but all info seems to be behind paywalls, not even WikiPedia seems to have a list

 

[sretalla]

I don't think OVH is the largest

You might be right, but to split hairs, what measure are we using to determine "largest".

You might count by number of hosted websites (OVH may have it there... they were the most popular before AWS and MS came on the scene... just a guess as to how somebody might have been claiming it).

You might also count by count of CPUs in the building or amount of RAM or Storage... Microsoft and Amazon will probably win everything on that side.

Amount of power drawn from the grid (or self generated)... no idea on that one as it will relate to how old some of the equipment is and where it's located... OVH seems to be in metropolitan Paris (and some other cities), so probably counts on the grid for nuclear, but has no space for solar or geothermal... not sure about Microsoft or Amazon. I guess OVH has the most "old" gear though.

 

[jgreco]

You might be right, but to split hairs, what measure are we using to determine "largest".

Yeah, as I said a few posts later.

My proposed measures of size were a lot better than yours heheh