Article: Bloomberg allegations of Supermicro hack

[iRefugee]

I came across this article about a tiny chip that was implanted in supermicro server motherboards. It's an interesting read.

https://www.bloomberg.com/news/feat...filtrate-america-s-top-companies?srnd=premium

 

[Spearfoot]

Yikes! I've got four Supermicro systems at home, and umpteen Supermicro-based systems at work.

From the Bloomberg article:

1) A Chinese military unit designed and manufactured microchips as small as a sharpened pencil tip. Some of the chips were built to look like signal conditioning couplers, and they incorporated memory, networking capability, and sufficient processing power for an attack.

2) The microchips were inserted at Chinese factories that supplied Supermicro, one of the world’s biggest sellers of server motherboards.

3) The compromised motherboards were built into servers assembled by Supermicro.

4) The sabotaged servers made their way inside data centers operated by dozens of companies.

5) When a server was installed and switched on, the microchip altered the operating system’s core so it could accept modifications. The chip could also contact computers controlled by the attackers in search of further instructions and code.

 

[Elliot Dierksen]

It may take a while, but I fear Supermicro is toast....

 

[HoneyBadger]

Now look up the Intel ME (Management Engine) and realize that there's been a vendor-provided backdoor at the chip level for years.

The only reason this is news is because it's not a domestic actor, and the Red Scare is a good headline-grabber again.

(Hi, NSA.)

 

[Elliot Dierksen]

The only reason this is news is because it's not a domestic actor, and the Red Scare is a good headline-grabber again.

(Hi, NSA.)

Good point. https://hackaday.com/2017/12/11/what-you-need-to-know-about-the-intel-management-engine/

 

[Ericloewe]

I'm not saying this is impossible, but it sounds too crazy and convoluted to be practical.

The bottom line is that PCB manufacturing isn't good enough for what is being claimed to be effective, at least when compared to simpler alternatives. The package shown is positively tiny, around 1.5 mm in length, and has six pins. That's not impossibly tiny, but it is "holy crap, our yield is terrible" tiny. It's probably not out of place on very high-end, very dense motherboards, but that brings us to the next point: They need the board layout files to alter them to cram in the tiny IC, its power supply and the data lines. Power should be inconspicuous enough, but four data lines are going to be a pain to route to the BMC.

Now, let's assume they get this far and their hack has okayish yields and doesn't break the system when it's defective. What can you do with four data lines? Not PCIe, not DDR3/4. USB maybe, but a controlled impedance differential pair might too difficult to cram in. That leaves I2C/SMBus and SPI. The BMC could be vulnerable to an attack on the SMBus and some sort of man-in-the-middle between the BMC and its firmware EEPROM device seems doable. I'd go for the second option, since it's more robust - you can attack some part of the firmware image that is unlikely to change, maybe something seemingly trivial like the graphics stuff. That's a plus because you're not getting firmware updates for your little bug (it might even be a mask ROM, to save the trouble of programming such a tiny device out of circuit).

Thing is, there's a much simpler attack that can be done: Bug the ASpeed BMC's microcode - introduce a subtle bug that messes with the firmware loading in a way that does nothing with a good firmware chip but loads malicious firmware from a compromised EEPROM device. That saves you a ton of trouble. You only need to compromise the BMC, which is easily done by replacing the mask for the metal layer that defines the static portion of the microcode (it's the kind of thing that's easy to change) and the supply of EEPROM devices for the BMC - you don't even need to compromise the manufacturer, just replace the real ICs with your own custom ones and drop the good ones off the back of a truck in some alley in Shenzhen.

 

[Arwen]

I was going to report the same Supermicro issue.

Though it sounds like a firewall for the IPMI Ethernet port would block it. (Except some of the boards are designed to allow both dedicated Ethernet ports and shared Ethernet ports. So you may have to give up a production port to be more certain you blocked the IPMI.)

 

[Ericloewe]

By the way: The theory that the bug might be inserted in a fiberglass layer of the board is farfetched. It would be a substantial departure from standard PCB manufacturing processes and I wonder if anyone out there does this sort of thing - I suspect not.

 

[Ender117]

Well, on the good side, maybe will see lots of supermicro gears hitting the used market? I will definitely get myself some

 

[zamana]

Oh, no!

Chinese government is watching all my porn movies. And for free!!!

That's a really disaster for all of us, domestic power users...

 

[danb35]

Curious that this would be in Bloomberg, rather than a tech-oriented publication. And while the idea of the Chinese government being able to contaminate the supply chain in such a way is plausible, the capabilities described in such a small device are questionable (IMO).

 

[danb35]

Some further information: https://www.lightbluetouchpaper.org/2018/10/05/making-sense-of-the-supermicro-motherboard-attack/

Edit: I wonder what this is going to do to the prices of used Supermicro gear. Might be time for an upgrade...

 

[jgreco]

I've retitled this thread because it is not prudent to perpetuate likely falsehoods.

While it is certainly *possible* that the Chinese (etc) might have done this, as of yet there is no evidence, and there is equal likelihood that there are other vendors whose gear could have been tampered with. In particular, I believe that there have been some concerns about Cisco, and IBM's sale of its PC and server businesses to Lenovo generated this sort of spectre-paranoia as well.

Conspiracy theories are great fun. Look at 9/11. People weave these intricate conspiracy theories that somehow, "demolition crews" wired the WTC, and the planes were military planes, and that all the victim families are lying, or that the victims are sequestered on some remote island, etc., etc., and if you start to examine the implications of any of these, it would require a cast of thousands of co-conspirators to have pulled off 9/11. I, however, am a firm believer in Ben Franklin's wisdom: "Three can keep a secret, if two of them are dead." This advises us that a large conspiracy, while not impossible, is highly unlikely.

A better conspiracy theory would be: "Dick Cheney (picked because he had sufficient power and also industry connections likely to benefit) ordered a CIA officer to approach Bin Laden with a suitcase of money and convinced him to pull off 9/11. On the way home, that officer was killed by another CIA officer who was told that the first was a traitor. 9/11 unfolded as reported by the news." This is a much BETTER conspiracy theory from a practical perspective. It makes Occam's Razor happy. However, for whatever reason, it does not please conspiracy theorists, who seem to prefer Rube Goldberg schemes.

I don't believe either one to be true, by the way.

So the thing here is, as noted above, there's no good reason for China to be modifying the hardware. ANYBODY who doesn't understand this should know that the CS world has been acutely aware of the software threat model for a long time, certainly back to the Ken Thompson hack or earlier. That's 35 years ago, by the way, so most of today's kids think they're discovering all this stuff for the first time, but they're not.

If Bloomberg had come forward with a claim that the BMC firmware was being subverted with a "call home," I'd say "figures" and move on. Dell and HP firmware are susceptible to corruption as well. This is entirely non-shocking and to be expected, even.

It is definitely possible to do the hardware hack, but it leaves evidence, and when discovered, it would ultimately result in methods to be able to identify affected boards. It's much more plausible that one of the third-party vendors that supplies the IPMI and BMC firmware would be subverted to install some obfuscated code, which would have the bonus of not only infecting newly shipped systems, but also legacy systems that received an in-the-field firmware update.

This is something to take seriously. Not Bloomberg's crap, which should have been accompanied by proof of such a claim, which wouldn't be hard to come by if it was true. But the threat model is definitely there. Your IPMI should not be live on the Internet. It shouldn't even be able to REACH the Internet. Even giving it DNS query access provides a vector through which data could be exfiltrated from your network.

http://www.sol.net/dldns/dldns.c

Compile and run that (It's safe. Just a download-via-DNS example that prints a message to your screen.)

 

[Arwen]

@jgreco, I've come to similar conclusions, (on the Bloomberg story).

In fact, it sounds so much like a stock market scam, that's my current conspiracy theory :).
(There appears to have been a similar stock market scam earlier this year against AMD, which supposedly had LOTS more security issues...)

On the subject of security and foreign tampering, I worked at a place that was replacing all IBM x86 servers due to the Lenovo purchase. It was going to cost a lot of money, but they were not in a position to risk it.

 

[danb35]

I, however, am a firm believer in Ben Franklin's wisdom: "Three can keep a secret, if two of them are dead." This advises us that a large conspiracy, while not impossible, is highly unlikely.

I subscribe to (ficticious) Adm. James Greer's maxim that the odds of an operation being blown are directly proportional to the square (or was it the cube?) of the number of people involved--which reaches the same conclusion, perhaps even more forcefully. At a bare minimum, several thousand people would need to be in on it, with none of them having cracked in 17 years. Though not impossible, "highly unlikely" significantly understates the case.

 

[jgreco]

@jgreco, I've come to similar conclusions, (on the Bloomberg story).

The general threat is clearly non-fictional, and as an industry, there's been some failure to take this sort of thing seriously. I've been looked at kind-of strangely when I talk about "restricted-management" networks and other segmentation/isolation strategies. Then there are those that totally get it, and even level-up to very restrictive strategies, including airgapped networks with draconian filtering. The thing is, I would expect that some of these networks would have identified any rogue traffic attempts. It is, after all, how we've become aware of some of these things in the past.

In fact, it sounds so much like a stock market scam, that's my current conspiracy theory :).
(There appears to have been a similar stock market scam earlier this year against AMD, which supposedly had LOTS more security issues...)

I'll grant that that's plausible, with Supermicro's current stock listing issues.

On the subject of security and foreign tampering, I worked at a place that was replacing all IBM x86 servers due to the Lenovo purchase. It was going to cost a lot of money, but they were not in a position to risk it.

And that's the thing. There's certainly an issue of whether or not you trust your vendors. The problem is that virtually all of them contract out for subsystems at some level - Dell doesn't make their PERC ("LSI") RAID, for example - so there are many unlikely-yet-possible threat vectors in our servers, even if we trust the direct vendor. To the best of my knowledge, all of the biggies still subcontract out for server builds to companies such as Foxconn, and it is probably impossible to guarantee that your gear is actually secure. There's too many places where the supply chain can be subverted. There's too many things that could be done. At some point we need to assume that this is not only possible, but also likely, and it may be due to deliberate malfeasance by the vendor or their subcontractors, but also the even more likely threat of third-party hacks that attack discovered-to-be-vulnerable subsystems at a later date.

 

[jgreco]

I subscribe to (ficticious) Adm. James Greer's maxim that the odds of an operation being blown are directly proportional to the square (or was it the cube?) of the number of people involved--which reaches the same conclusion, perhaps even more forcefully. At a bare minimum, several thousand people would need to be in on it, with none of them having cracked in 17 years. Though not impossible, "highly unlikely" significantly understates the case.

I would gently suggest that you're wrong about "more forcefully." I think the more forceful line is Franklin's. It basically suggests that a secret held by more than one is always a risk, and suggests a reprehensible course of corrective action on top of it.

The Greer line does have validity for operational security, and the concept definitely comes into play if you accept the possibility of a "vast conspiracy." However, every time I hear the words "vast conspiracy," my bullshit detector buzzes until it melts. At scale, these things just don't pan out in practice. People can be compromised for any of a number of reasons, including financial, disillusionment, etc. I'm sure it's possible to find a handful of people who are willing to conspire towards a given goal, but even the task of finding those people becomes risky as the number desired increases, and keeping the secret of such a conspiracy indefinitely is probably a nonstarter even at small group sizes. Which brings me back to Franklin.

In the news this morning, a guy planning to blow himself up in the DC mall. He told one other guy he trusted. Now he is in the hands of the FBI. Actually funny, I had paused that, complete with a caption "FBI RAIDS HOME OF BOMB PLOT SUSPECT," on the TV while I took time to compose this reply.

Trust is a funny thing.

Anyways, as I get on in years, I find that I spend a lot more time analyzing risks and mitigating. Secured networks, single-purpose VM's, etc. There are so many threat vectors to our computing systems. Some paranoia is not unwarranted.

 

[danb35]

Some paranoia is not unwarranted.

Just because you're paranoid, it doesn't mean they aren't out to get you.

 

[Arwen]

When I worked at Sun Microsystems, they had some relatively new hardware, the Mid-Range 3800-6800 servers. At one point we were told that the system controllers, (SCs), used for management, (including domain consoles), should be on an isolated sub-net. Turns out their was a bug in the SC software, (NOT directly a security issue), that if on the same sub-net as chatty MS-Windows servers, (using broadcast sutff), that could crash the SCs. After it was fixed, it became the recommendation to continue that practice for security reasons. That's back in 2004 if I remember correctly.

Back then, using a SSH jump server to access that isolated sub-net was quite reasonable. Today, if you have to do so, you need to make sure your SSH jump server supports tunneling. Or has local browsers. Otherwise you may not be able to get access to the remote consoles of x86 servers.

 

[danb35]

Some further reading:
https://www.servethehome.com/investigating-implausible-bloomberg-supermicro-stories/