URGENT HELP! SERVER CRASHED!

[jgreco]

Posted fer the lulz, the picture doesn't quite do the damage justice. That's an SC847 that got ... dropped, I suspect.

 

[Jailer]

Ouch! How did the rest of the hardware fare?

 

[Elliot Dierksen]

Think you could call that a hard crash...

 

[joeschmuck]

Ouch, was it new and delivered this way or was it dropped by someone at work? And I really liked the eye catching title of the thread.

 

[jgreco]

Ouch! How did the rest of the hardware fare?

No clue. Just another day of life at the data center. By the time I saw it, it was sitting there all forlorn knowing that there isn't a "body shop" for servers. Supermicro's less than 10 miles down the 880, so I imagine someone might sheepishly have called in to find what a bare 847 chassis was going to cost them...

 

[jgreco]

Ouch, was it new and delivered this way or was it dropped by someone at work?

Unknown. Reading the tarot cards in the form of a large pile of filler-less 3.5" trays next to the unit, my guess is the unit was fully loaded with drives. which may explain why there isn't more significant "crush-in" - the drives worked to prevent any major deformation. The left front bracket appeared to have sheared off, also what you'd expect if a very heavy server was dropped on the bracket corner, and that millimeter wide vertical gap between the outer steel and the inner assembly is ungood. The inner tray sheet metal is also a bit off-kilter in a way that the picture doesn't do justice, except for the bottom left one which is obviously way wonky. My guess is that most of the "naked eye" damage was actually done trying to recover those drives on the bottom left.

So I tend to think that it was dropped on the concrete floor. Shipping damage could absolutely bend the thing, but the shearing of the bracket seems inconsistent with a shipping box. I don't know what to make of the face damage on the right hand side.

And I really liked the eye catching title of the thread.

I get silly when I've spent too many days at the data center.

 

[cobrakiller58]

Is it bad that my first thought when I saw this picture was "will it be parted out?" lol

 

[Dan Tudora]

Unknown. Reading the tarot cards in the form of a large pile of filler-less 3.5" trays next to the unit, my guess is the unit was fully loaded with drives. which may explain why there isn't more significant "crush-in" - the drives worked to prevent any major deformation. The left front bracket appeared to have sheared off, also what you'd expect if a very heavy server was dropped on the bracket corner, and that millimeter wide vertical gap between the outer steel and the inner assembly is ungood. The inner tray sheet metal is also a bit off-kilter in a way that the picture doesn't do justice, except for the bottom left one which is obviously way wonky. My guess is that most of the "naked eye" damage was actually done trying to recover those drives on the bottom left.

So I tend to think that it was dropped on the concrete floor. Shipping damage could absolutely bend the thing, but the shearing of the bracket seems inconsistent with a shipping box. I don't know what to make of the face damage on the right hand side.



I get silly when I've spent too many days at the data center.

send that to recycle (bin)
maybe was delivery with a parachute or with the new drone amazone delivery
OR send back to furnissor

 

[rvassar]

Supermicro's less than 10 miles down the 880

Newark has datacenters now? A mile deep pile of mud sandwiched between two very serious faults... Ugh!

 

[jgreco]

Newark? I guess that's just to the west of Fremont. Fremont is home to Hurricane Electric, a major global network with native IPv4 and IPv6, and they also happen to operate two colocation sites in Fremont, with some fairly aggressive pricing (but you have to know how to go about it). It is home to the FCIX internet exchange and also a significant presence for SFMIX and I believe also AMS-IX Bay Area. It's not Equinix or Dupont Fabros (now merged), but it's been very stable and the staff is friendly.

 

[rvassar]

Newark? I guess that's just to the west of Fremont. Fremont is home to Hurricane Electric, a major global network with native IPv4 and IPv6, and they also happen to operate two colocation sites in Fremont, with some fairly aggressive pricing (but you have to know how to go about it). It is home to the FCIX internet exchange and also a significant presence for SFMIX and I believe also AMS-IX Bay Area. It's not Equinix or Dupont Fabros (now merged), but it's been very stable and the staff is friendly.

Newark is a "suburb" of Fremont, to the north west. The whole place was citrus orchards when I was a kid. Yes, I know about HE in Fremont. Not a decision I would have made, but I've never aspired to those lofty management circles...

I grew up a few miles further north, and have a BSci Geology from a local Uni. Just wait... Planet Earth will have the last laugh. Have a proper disaster plan...

 

[jgreco]

Newark is a "suburb" of Fremont, to the north west. The whole place was citrus orchards when I was a kid. Yes, I know about HE in Fremont. Not a decision I would have made, but I've never aspired to those lofty management circles...

Well, I don't know what your objections are. I'm well aware of the IPv6 peering situation, but having been doing this stuff since before the birth of the commercial Internet, I've seen gobs of much crazier stuff.

I grew up a few miles further north, and have a BSci Geology from a local Uni. Just wait... Planet Earth will have the last laugh. Have a proper disaster plan...

For what, exactly? For many years, our east coast point of presence was at Equinix DC2 right under the glide path of one of the major Dulles runways. Our home city POP was in a building more than 100 years old, with structural clay tile floors. As the building transformed into a carrier hotel, there was some significant concern about upper level tenants who might not be taking appropriate engineering steps. We had an engineering firm come in and reinforce the original I-beams by welding on a reinforcing bar of modern steel, and then laid tube stock across, which the engineers assured us would allow us to safely put racks and UPS gear weighing tons. There's a nagging fear of what might be going on with Idiotnet Colo on the 12th floor where they didn't appear to have taken such care, and really once the clay tile starts to give way, whatever static load it had been holding becomes a gravity-driven battering ram on the way down to the subbasement.

At the end of the day, it's all just equipment, and you should never put all your eggs in one basket. My disaster plan is to sit back and enjoy a Coke. I try hard to avoid running monolithic servers, and replicate them to various locations instead. This has the neat side effect that if some of it goes away, it isn't really a crisis.

 

[Spearfoot]

View attachment 43274

Posted fer the lulz, the picture doesn't quite do the damage justice. That's an SC847 that got ... dropped, I suspect.

This reminds me of the time FedEx destroyed our Sun E4500 server, back in the early oughts. The thing was built like a tank; probably weighed 200lbs; and was shipped on a pallet, same as your server.

FedEx must have dropped it from a pretty good height, as it bent the frame! Luckily we'd insured it for $10k, which covered replacement cost (but not the man-hours spent in setting it up).

 

[Dan Tudora]

Sun E4500 server

this must to be "sooooo long time a go"
delivery in this century was mooore quickly

 

[rvassar]

Well, I don't know what your objections are.
(snip)
At the end of the day, it's all just equipment, and you should never put all your eggs in one basket. My disaster plan is to sit back and enjoy a Coke. I try hard to avoid running monolithic servers, and replicate them to various locations instead. This has the neat side effect that if some of it goes away, it isn't really a crisis.

I got to experience Loma Prieta. It was fully 50 miles away, and it still made my Mom's parked mini van an "assault low-rider" as I tried to run away from the cracking brick facade I was standing under. I didn't know a mini van could bounce like that! :)

I just don't consider the inner bay area plain "stable" real estate. Not stable enough for critical infrastructure when there's so many other places in the country that are far more resilient, and have cheaper electricity. Even the other side of the east bay hills is a better choice, at least it's compacted, and less likely to liquify. HE locating in Fremont was always a curiosity to me. I figured there was a reason at the time, that I couldn't give consideration to. But 20+ years later I find myself wondering to what degree it's been revisited.

 

[jgreco]

I dunno, Cisco and Intel and Supermicro and iXsystems and many other places are headquartered there. As far as I can tell, the development of the Internet and its major hubs had a lot less to do with "stability" than other factors such as practicality. You get your occasional 350 Cermak that is built like a fortress because it happened to serve the printing industry and needed rail access, which worked out well when it became a major Internet hub, but many of the most critical hubs got built at a time when practical considerations, rather than "critical infrastructure stability," were a concern. Think of all the dumpy downtown transformed buildings, 56 Marietta, 216 Jackson, 60 Hudson, etc. Some have been refurbished and repurposed since I may have formed my opinion, but anyways.

I'm sitting in here in Santa Rosa in Sonic's machine room right now, and within just a few miles there are burned down houses. Where exactly is "safe"?

Anyways, I haven't asked, but Mike's goal doesn't seem to be colo, it appears to be creating a good global IP network. As someone said to me a few years ago, "colo hasn't been a growth industry in some time." I suspect Mike views the colo stuff as a bonus.

 

[rvassar]

I'm sitting in here in Santa Rosa in Sonic's machine room right now, and within just a few miles there are burned down houses. Where exactly is "safe"?

Anyways, I haven't asked, but Mike's goal doesn't seem to be colo, it appears to be creating a good global IP network. As someone said to me a few years ago, "colo hasn't been a growth industry in some time." I suspect Mike views the colo stuff as a bonus.

True... True... I have family in Cloverdale. Twas a rough fall...

If you're wine tasting this weekend, and honestly I don't know what's open at this point, I recommend Brutacao up in Hopland, and Fenestra in Livermore.

 

[Constantin]

I had a similar situation years ago with a client for whom I was developing a new appliance. We had some enclosures laser cut, bent, and welded up as a proof of concept and shipped across TN. They arrived a short while later... damaged and covered with the drivers boot prints and cud. Thankfully, I had my polaroid camera handy and a few pictures later, my client got the parts for free and the driver likely got a $900 worth of talking to. Since the project was "full speed ahead", I cleaned, unbent, and rewelded the enclosures to make the deadline. The cud removal was pretty gross though.

It's all relative though. Imagine having your 12 ton delivery go sideways out of your building and fall 28 stories to the sidewalk below. I'm sure explaining that made some insurance adjusters day. Thankfully, no on was hurt.