Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Article: Bloomberg allegations of Supermicro hack

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,795
I work in network security and know something about TCP/IP, and in my mind it is totally feasible for there to be a NIC with an embedded (and hidden) chip that could copy packets and launch outbound connections to foreign addresses unbeknown to the user, or even the system. I don't know what exact components are used physically on the mobo, but if it's possible to copy traffic on a cable, I see no reason why packets could not be copied without interruption to the normal packet flow and without the mobo being any wiser to what is going on...
You say you work in network security?

If you did work in network security, I'm sure that you'd be aware that IPMI interfaces are very often segmented from the Internet on an airgapped network, and that many places have intelligent IDS systems that alarm on *any* unexpected traffic. Unless you have some magic Voldemort grade silicon that can transmit packets around the world without the minor formality of actually transiting real copper or fiber cables, it is absolutely certain that SOMEONE would have noticed by now, and we'd have proof that it exists. It only takes one site to spot the "outbound connections to foreign addresses unbeknown to the user". This has not happened, which suggests that there is no such silicon subversion, or, it isn't being used in any meaningful way.

Besides, any attacker embedding things in silicon is a fool, given that the firmware would be so much easier to subvert; that would give you code that could hide on a protected subsystem in the server, and yet be able to vanish with barely a trace. This really just turns into an argument to avoid placing your critical infrastructure and storage in a position on the network where it can reach the general Internet, something that those of us who actually do work in network security have advised ... pretty much forever. This is, of course, commonly ignored by CTO's and others who do not subscribe to such levels of security paranoia.
 

naskit

Neophyte
Joined
Apr 19, 2021
Messages
7
You say you work in network security?

If you did work in network security, I'm sure that you'd be aware that IPMI interfaces are very often segmented from the Internet on an airgapped network, and that many places have intelligent IDS systems that alarm on *any* unexpected traffic. Unless you have some magic Voldemort grade silicon that can transmit packets around the world without the minor formality of actually transiting real copper or fiber cables, it is absolutely certain that SOMEONE would have noticed by now, and we'd have proof that it exists. It only takes one site to spot the "outbound connections to foreign addresses unbeknown to the user". This has not happened, which suggests that there is no such silicon subversion, or, it isn't being used in any meaningful way.

Besides, any attacker embedding things in silicon is a fool, given that the firmware would be so much easier to subvert; that would give you code that could hide on a protected subsystem in the server, and yet be able to vanish with barely a trace. This really just turns into an argument to avoid placing your critical infrastructure and storage in a position on the network where it can reach the general Internet, something that those of us who actually do work in network security have advised ... pretty much forever. This is, of course, commonly ignored by CTO's and others who do not subscribe to such levels of security paranoia.
I see where there is some confusion here. I should have been more explicit. Sorry. Forget corporate networks and data centres....that is a different story, and I agree with you 110% that all Management interfaces should be segregated to restricted/secure networks (that not even normal users can reach). But I was only asking this in the context of a home network, and I wasn't asking about IPMI/Management interfaces, rather, about the in-band NICs...Corporate environments are significantly different from home networks. At the networking level, one of the most important differences is that most home networks have just a single gateway to the Internet which not only allows all outbound connections by default, but if any devices on the home network have UPnP turned on, most home gateways (typically supplied by the ISP) are also configured to open up inbound ports based on automatic UPnP requests from LAN devices (e.g. TV, Setop Box, Gaming PCs, and, you guessed it, NAS devices). This means that for most home users, they would be unaware that their home network is open to the internet via a number of ports. Why do I say this? Because the very reason I started looking into FreeNAS was because one day not long after I plugged in my new QNAP I discovered it had used UPnP to open a whole lot of ports on my home gateway that were unclear as to the purpose. Needless to say I promptly disabled UPnP on my gateway and turned my NAS off for awhile while I pondered the best solution. So, in my mind, it is ENTIRELY possible for a nefarious NIC manufacturer to embed hidden FPGAs into the NIC at a component level which are programmed to use UPnP, or even just to copy packets and duplicate them to some unknown IP on the Internet. You home gateway will simply let it out and will not do anything to block it. I hope that clears it up where I was coming from.
 

naskit

Neophyte
Joined
Apr 19, 2021
Messages
7
Here is a followup article on this subject. It does mention the SuperMicro incident was denied (but of course, what would you expect Apple/Amazon to do, confess?)

The point is, it is possible. In my view that warrants caution over where I source my parts. I just wanted to know if anyone else felt the same and had some alternative suggestions for Motherboards and PCI cards :)
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,795
I see where there is some confusion here. I should have been more explicit. Sorry. Forget corporate networks and data centres....that is a different story, and I agree with you 110% that all Management interfaces should be segregated to restricted/secure networks (that not even normal users can reach). But I was only asking this in the context of a home network, and I wasn't asking about IPMI/Management interfaces, rather, about the in-band NICs...Corporate environments are significantly different from home networks. At the networking level, one of the most important differences is that most home networks have just a single gateway to the Internet which not only allows all outbound connections by default, but if any devices on the home network have UPnP turned on, most home gateways (typically supplied by the ISP) are also configured to open up inbound ports based on automatic UPnP requests from LAN devices (e.g. TV, Setop Box, Gaming PCs, and, you guessed it, NAS devices). This means that for most home users, they would be unaware that their home network is open to the internet via a number of ports. Why do I say this? Because the very reason I started looking into FreeNAS was because one day not long after I plugged in my new QNAP I discovered it had used UPnP to open a whole lot of ports on my home gateway that were unclear as to the purpose. Needless to say I promptly disabled UPnP on my gateway and turned my NAS off for awhile while I pondered the best solution. So, in my mind, it is ENTIRELY possible for a nefarious NIC manufacturer to embed hidden FPGAs into the NIC at a component level which are programmed to use UPnP, or even just to copy packets and duplicate them to some unknown IP on the Internet. You home gateway will simply let it out and will not do anything to block it. I hope that clears it up where I was coming from.
So, what exactly is different about my assessment, in this case?

The fact of the matter is that there are lots of different security appliances and strategies that would have detected this sort of nefarious activity if it was happening. This is actually how many (most?) security problems are first detected these days, with unusual or suspicious traffic being noted. This isn't being noted.

Perhaps if you had some very smart silicon that could detect that it was only on a home network, without any sort of IDS, and had a direct connection to the Internet, but then this raises new questions, such as, what value is there to an attacker in your home user data?

See, the thing is, all sorts of things COULD happen. The 9/11 conspiracy theorists have all sorts of interesting theories about how the WTC was laced with explosives by a demolition company, how smaller airplanes were used, how the people who were "killed" were actually secreted off to some mysterious Themyscira-like island, etc., but none of this stands up to inspection or reason. Demolition companies talented enough to pull this off are large organizations of people who have made a career of public safety, none of the killed people have managed to pull an Oliver Queen, etc. All of these intricate fantasies about things that COULD happen, well, okay, maybe you could engineer such a thing somehow, but why bother? Look, if 9/11 was something that the Bush/Cheney administration instigated, it is much more reasonable to assume something like this: Cheney, as VP, was able to prevail upon a CIA operative to bribe Bin Laden with a suitcase of money under false pretenses, then had a second operative kill the first one afterwards, and all the remaining events unfolded as we know. This is a very workable theory, as it involves only Cheney, Bin Laden (who didn't know the money came from the CIA), a dead CIA operative who knew what went down, and one living CIA assassin who knew nothing of consequence. The truth would only be known to Cheney, and, while I don't believe this theory, if it did turn out that the US was behind 9/11, I would bet that this is much closer to the truth than the wild conspiracy theories, almost all of which require a conspiracy of at least dozens or hundreds of participants. Secrets like that don't keep.

At the end of the day, you have to differentiate between what might be possible and what is actually practical. It's really good to be aware of what might be possible, because it helps inform you of what to be looking for. I am absolutely NOT discouraging you from being aware that these things can be subverted. They can. But you should be equally concerned that someone could hack the download server and subvert the download ISO's for FreeNAS, gaining a much more powerful platform that could do analysis of traffic and then backchannel delivery from the operating system.
 

danb35

Wizened Sage
Joined
Aug 16, 2011
Messages
12,070
The point is, it is possible.
Nobody's saying otherwise. It's similarly possible that Dell, Lenovo, and/or HPE have been compromised in like manner. But you joined this forum to ask if anyone was are of the security vulnerabilities "found" on Supermicro gear, and in the context in which you're speaking, there aren't any--no evidence has as yet (three years after the initial allegations) been shown to support this fantastic claim.
So, in my mind, it is ENTIRELY possible for a nefarious NIC manufacturer to embed hidden FPGAs into the NIC at a component level which are programmed to use UPnP,
Note that you're proposing a completely different vulnerability than was claimed against Supermicro. I'll leave to others to discuss how plausible this attack is, but it isn't what Bloomberg said happened to SM.

There is some truth in the concern. One of the few kernels of truth in the article is that the IT supply chain includes a very large and efficient country that isn't entirely friendly to the western world, and it's pretty much impossible to avoid China at some level. And we all agreed, IIRC, three years ago (have you read this thread? It isn't very long) that, in principle, there's a concern with having your IT gear come from less-than-friendly countries--which would affect pretty much everyone. But Bloomberg printed, you believed, and you for some reason joined this forum to ask about, a claim of a very specific vulnerability--and as yet, three years later, there's simply no evidence to support it.

And now you're back-pedaling to try to suggest there could be something--why? And why would you think that would uniquely affect SuperMicro, and not raise similar concerns about any/every other manufacturer?

So to answer your direct question, these allegations have had zero effect on how I view SM gear--but to the extent Bloomberg had any credibility in my view, that's pretty much gone.
 

ornias

Neophyte Sage
Joined
Mar 6, 2020
Messages
1,370
Might I suggest the user is just trolling?
He joined to forum to respond on a threaddiscussing (and dismissing) the suggested breach, asking if we where aware of the breach.

He didnt even care enough to read the thread he was responding on.
 

danb35

Wizened Sage
Joined
Aug 16, 2011
Messages
12,070
He joined to forum to respond on a threaddiscussing (and dismissing) the suggested breach
According to the moderator's note, his post was moved from a different thread; it wasn't initially a response here. But even so, he kind of looks like a troll.
 

naskit

Neophyte
Joined
Apr 19, 2021
Messages
7
I think it's important to distinguish between two very different, though somewhat-related, assertions:
  • Our IT supply chain is vulnerable
  • Our IT supply chain has been compromised, in a specific way, at a specific time
It sounds like you're conflating the two. I don't think anyone can reasonably argue against the first, particularly when a great deal of it involves a country that isn't entirely friendly to the United States (or to the western world in general). But that isn't the assertion you made--you said, in a post you joined this forum to make, that SuperMicro hardware has been found to have specific vulnerabilities due to a specific compromise. And you've shown no evidence of that. You didn't even point to the Bloomberg articles, the only known source of such an assertion, which also contain no evidence to support it. Nor do you show any indication of understanding the claims that are made (no, you didn't mention IPMI or the BMC, but Bloomberg did), nor of any of the discussion that happened (even in this very thread, much less anywhere else) regarding these bombshell allegations.
Thanks danb35. I agree: "is vulnerable" and "has been compromised" are different things. Yes, I did specifically refer to the SuperMicro compromise claim. No, I do not have direct evidence myself. About all these, you are correct.

I did, however, post a link to another article which goes into a bit of detail where Monta Elkins was able to solder a chip to the motherboard of a Cisco ASA 5505 firewall giving him serial console access. Proof it can be done.

Bloomberg has since posted a more in depth article here:

In this, they also accuse Lenovo:
"
Another Pentagon supplier that received attention was China’s Lenovo Group Ltd. In 2008, U.S. investigators found that military units in Iraq were using Lenovo laptops in which the hardware had been altered. The discovery surfaced later in little-noticed testimony during a U.S. criminal case—a rare public description of a Chinese hardware hack.
“A large amount of Lenovo laptops were sold to the U.S. military that had a chip encrypted on the motherboard that would record all the data that was being inputted into that laptop and send it back to China,” Lee Chieffalo, who managed a Marine network operations center near Fallujah, Iraq, testified during that 2010 case. “That was a huge security breach. We don’t have any idea how much data they got, but we had to take all those systems off the network.”
Three former U.S. officials confirmed Chieffalo’s description of an added chip on Lenovo motherboards.
"
There does seem to be a lot of regurgitation, however the Court Testimony is a little hardwer to brush away, and the link is here: https://assets.bwbx.io/documents/users/iqjWHBFdfxIU/r9dKMMM0Gi5I/v0
Goto page 69, line 20. Read down to page 72 line 9.

The question really is: do you believe Bloomberg.com, and this Court Testimony, or do you believe those who do disagree with Bloomberg about this story?
I ask myself: who disagrees and why? What is their financial stake in this story being false?

And looking at this from a Risk Management point of view, what would you be expected to do if you were in charge of IT Security for a large company?
Is it really enough just to say "it's just a 'bombshell' conspiracy" simply because you haven't yet seen the evidence?

I appreciate your replies.
 

naskit

Neophyte
Joined
Apr 19, 2021
Messages
7
Might I suggest the user is just trolling?
He joined to forum to respond on a threaddiscussing (and dismissing) the suggested breach, asking if we where aware of the breach.

He didnt even care enough to read the thread he was responding on.
Thanks for your suggestion. No, I am not trolling. If I missed some earlier posts, I apologise. I came here because I am genuinely trying to spec up a FreeNAS system and I am trying really hard to pick HW from sources I trust, and although I have no direct evidence of the SuperMicro breech, stories like this disuade me from using any components from that part of the world. I wanted to know if anyone else had the same concerns, done similar research, and found alternative HW sources, and if so, would they be happy to share their recommendations?

If you have, I'm interested in what you chose ;)

This will be the 6th or 7th system I've spec'd up and I have found it very difficult trying to find a good combination of ECC RAM, a Mobo and a CPU that all support each other and are not from a politically compromised country with a government that appears to be quite hostile to the west...(I concede that this is my opinion of that system of government).
 

naskit

Neophyte
Joined
Apr 19, 2021
Messages
7
https://www.cultofmac.com/585868/apple-bars-bloomberg-from-ipad-event-as-payback-for-spy-chip-story/

https://www.theverge.com/2018/10/22/18011138/china-spy-chip-amazon-apple-super-micro-ceo-retraction

https://www.zdnet.com/article/super...rg-chip-hack-story-in-recent-customer-letter/

https://www.zdnet.com/article/secur...chip-hack-investigation-casts-doubt-on-story/

https://www.cyberscoop.com/dan-coats-bloomberg-supply-chain-the-big-hack/

I feel it's important for people to recognize that spreading this sort of conspiracy theory is problematic. It is certainly something that is *possible* to do in hardware, but it leaves evidence that the supply chain was subverted. It's much better to do in software. Just to give you an idea... go download an IPMI .bin file. Scan it using "binwalk" and extract the two CramFS portions to files. Then, mount them on a Linux system.

mount -o loop -t cramfs cramfile1.bin filesystem1

etc. You too can poke around inside Supermicro IPMI firmware. If you were going to subvert things, this would be a great way to do it. All that frakking code and so much space to hide stuff in.

So keep these things off the Internet. Don't let them access the Internet, either.
"So keep these things off the Internet. Don't let them access the Internet, either."
I think that is really good advise.

I also think this is one of the best responses to the Bloomberg SuperMicro Hack claim:
 

danb35

Wizened Sage
Joined
Aug 16, 2011
Messages
12,070
Bloomberg has since posted a more in depth article here:
Yes, we know. It was posted on this very thread over three months ago. If you aren't a troll, you need to stop acting like one; one important part of that would be at least a passing familiarity with what's gone in the only thread on this whole forum that you've posted in.

You continue to conflate the entirely uncontroversial assertion that our supply chain is vulnerable (which nobody is disputing) with the highly controversial claim that Supermicro has in fact been compromised in a specific way, and to do so even after acknowledging that they're completely different claims. The only support for the latter assertion is two articles in Bloomberg (not exactly known as a tech-savvy publication) which make outlandish claims, while providing no evidence whatsoever to support them. But your conflation is obvious:
The question really is: do you believe Bloomberg.com, and this Court Testimony
Why must these go together? They're completely different sources, making completely different claims. I do not believe Bloomberg, as they've given me no reason to believe them. I have no opinion with respect to the court testimony, nor do I see its relevance to this thread--if you think a purported compromise of Lenovo laptops 13 years ago has some relevance to this forum, maybe you could start a new thread about it.

But thankfully, this forum has a very effective ignore list.
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,795
Moderator note: thread closed. If anybody has constructive updates, send me a private message to discuss, but right now this is about as productive as a discussion of the merits of ECC memory or vi-vs-emacs.
 
Status
Not open for further replies.
Top