Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Article: Bloomberg allegations of Supermicro hack

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,795
https://www.cultofmac.com/585868/apple-bars-bloomberg-from-ipad-event-as-payback-for-spy-chip-story/

https://www.theverge.com/2018/10/22/18011138/china-spy-chip-amazon-apple-super-micro-ceo-retraction

https://www.zdnet.com/article/super...rg-chip-hack-story-in-recent-customer-letter/

https://www.zdnet.com/article/secur...chip-hack-investigation-casts-doubt-on-story/

https://www.cyberscoop.com/dan-coats-bloomberg-supply-chain-the-big-hack/

I feel it's important for people to recognize that spreading this sort of conspiracy theory is problematic. It is certainly something that is *possible* to do in hardware, but it leaves evidence that the supply chain was subverted. It's much better to do in software. Just to give you an idea... go download an IPMI .bin file. Scan it using "binwalk" and extract the two CramFS portions to files. Then, mount them on a Linux system.

mount -o loop -t cramfs cramfile1.bin filesystem1

etc. You too can poke around inside Supermicro IPMI firmware. If you were going to subvert things, this would be a great way to do it. All that frakking code and so much space to hide stuff in.

So keep these things off the Internet. Don't let them access the Internet, either.
 

danb35

Wizened Sage
Joined
Aug 16, 2011
Messages
12,070
but it leaves evidence that the supply chain was subverted.
...and, of course, if Bloomberg really wanted to prove its story, provide a board. This, I think, is the biggest hole in the story. Lots of (claimed) anonymous sources, a few named sources who have since said they were taken out of context, but zero physical evidence. And, as you say, there would be physical evidence--there would be thousands of boards with these chips on (or in) them. We could see what the chips are connected to and how, we could dissect the chip to figure out who made it, etc. But despite the claimed scope of the hack, nobody's shown an affected board.

As conspiracy theories go, it's ingenious. Of course everyone involved would want to hush it up. Apple/Amazon/OVH/etc. don't want their customers or shareholders to know they've been pwned. Supermicro certainly doesn't want their customers (or prospective customers) to know that their supply chain had been compromised for years. And the government wouldn't want the public to know how badly they'd been compromised either. So no matter how earnest or specific the denials, they fit right in with the theory. And all the compromised hardware was destroyed, of course, in the interest of security (or to perpetuate the cover-up). As is so often the case, the absence of any solid evidence of the conspiracy, is simply further evidence of how pervasive the conspiracy is.

So keep these things off the Internet. Don't let them access the Internet, either.
Is it physically possible for the BMC to access any network port other than the dedicated IPMI or LAN1 ports, or is that only a firmware restriction? Because if the controller is only physically connected to those two ports, that really does seem like a (relatively) way to protect against any vulnerability in the BMC.
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,795
...and, of course, if Bloomberg really wanted to prove its story, provide a board. This, I think, is the biggest hole in the story. Lots of (claimed) anonymous sources, a few named sources who have since said they were taken out of context, but zero physical evidence. And, as you say, there would be physical evidence--there would be thousands of boards with these chips on (or in) them. We could see what the chips are connected to and how, we could dissect the chip to figure out who made it, etc. But despite the claimed scope of the hack, nobody's shown an affected board.
And it's likely, given the type of people some of these companies employ, that at least a few boards have been stripped down looking at what the components actually are. Not a guarantee that you can successfully identify a compromised board, of course.

As conspiracy theories go, it's ingenious. Of course everyone involved would want to hush it up. Apple/Amazon/OVH/etc. don't want their customers or shareholders to know they've been pwned. Supermicro certainly doesn't want their customers (or prospective customers) to know that their supply chain had been compromised for years. And the government wouldn't want the public to know how badly they'd been compromised either. So no matter how earnest or specific the denials, they fit right in with the theory. And all the compromised hardware was destroyed, of course, in the interest of security (or to perpetuate the cover-up). As is so often the case, the absence of any solid evidence of the conspiracy, is simply further evidence of how pervasive the conspiracy is.
The only problem with that being that "all the compromised hardware was destroyed" runs counter to how these things are actually deployed. There's no realistic way for someone up the supply line to specifically target a particular Supermicro customer, or for them to get that particular Supermicro customer to deploy those servers in a useful way, or to force those machines to have BMC Internet reachability in a way that extra traffic won't be noticed.

Is it physically possible for the BMC to access any network port other than the dedicated IPMI or LAN1 ports, or is that only a firmware restriction? Because if the controller is only physically connected to those two ports, that really does seem like a (relatively) way to protect against any vulnerability in the BMC.
It's more complicated than that. There's basically a bunch of threat vectors.

So if I wanted to do something insidious, like send surreptitious reporting back to a mothership, you could note that the BMC has write access to the BIOS, and that this means you can trojan in a WPBT (Google: "WPBT BIOS") so that if Windows is loaded on the platform, it'll automatically run your crap. And of course Windows might be doing that on not-LAN1 (like LAN2 or 10G ports or whatever) and your Windows box is more likely to be on a network that has at least SOME Internet visibility.

The BMC has significant platform access. Think about it: this includes the ability to create USB devices, interact with the VGA display, interact with the BIOS/POST systems, and in some cases even things like accessing hardware RAID. Those are things IT ACTUALLY DOES ON YOUR SYSTEM. AS DESIGNED.

I could make up substantially better and harder-to-disprove crap than what Bloomberg is putting out there. :smile:
 

file_haver

Member
Joined
Sep 19, 2018
Messages
55
I was actually surprised Supermicro didnt take legal action for defamation to their name. If they did, it must have been resolved silently.

I use a Supermicro board in my FreeNAS box, and I use one of their Z170 microATX boards in my gaming PC. I have long supported them and had a lot of doubts about the allegation when I saw the Bloomberg article.

There was just too much in that article that didn't actually add up if you understand the behind-the-scenes on how computers work and how they're manufactured. The proof is how many people who focus on the tech world like us saw various red flags in the article.
 

danb35

Wizened Sage
Joined
Aug 16, 2011
Messages
12,070
I was actually surprised Supermicro didnt take legal action for defamation to their name.
...and this is what some folks believe lends some credibility to the allegations.
 

danb35

Wizened Sage
Joined
Aug 16, 2011
Messages
12,070
Yeah, they're really in a no-win situation there.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
17,060
Somewhat surprisingly, Bloomberg is digging up this story again.

I think the original stories did not have any names attached, this time they have a handful of people on the record saying some of this took place.

Now, it's been more than two years and nobody has produced a single shred of evidence that the super-secret chip exists. The new piece describes an earlier attack based on compromised system firmware, which is 100% possible and far more believable than the super-secret chip story (which they're doubling down on).

That said, their credibility is limited, to say the least, after the 2018 story. I'm curious about the named people who were said to have been briefed by US authorities on the matter, it'll be interesting to see if they're credible.

Now, technical side of the firmware compromise:
A quick bit of google-fu suggests that X8-generation boards used 4 MB flash ICs for the system firmware (probably legacy BIOS only, at that point, but I'll let users of those boards confirm), X9 and later use 16 MB ICs.
My non-expert intuition is that either case is about as easy/difficult to analyze in search of malicious code: Much of the ROM is boilerplate stuff (Intel microcode, firmware for random devices on the board, standard AMI code) which can easily be compared against other vendors (assuming AMI wasn't compromised). Supermicro's own code should be a fairly small portion of the total.
Now, let's assume the attackers aren't complete morons, you'd need some effort to reverse-engineer their code. It's going through the NIC, so either something's running in System Management Mode or the NIC firmware itself was compromised. It may be impractical for someone without specialized tools to discover and analyze this exploit, but it shouldn't be too difficult for a specialized lab to figure it out.
 

Arwen

Neophyte Sage
Joined
May 17, 2014
Messages
1,354
If it's firmware, either in the IPMI side, or BIOS/UFEI side, they would have to be careful to make the version numbers match current.

In the data center, SysAdmins tend to look at firmware versions as part of the server build. When I was building Sun Microsystems servers, (SPARC), we loaded current firmware that was downloaded directly from the manufacturer. And checked after reboot to verify all was loaded. Since we had to keep the firmware up to date, (part of patching), this allowed us to avoid one step in a month or 2. Plus, if this "new" firmware blew up, we would want it to blow up on a new server. Not an existing production server being patched.

So, if the exploit allowed a firmware update, it would not last long in that environment.

This does sound more reasonable than some mysterious chip.

Oh, if the supposed network exploit actually existed, normal firewalls should have block some out bound traffic. They would have had to carefully select the ports & IPs carefully. The place I work now, (not miltary), basically blocks all in-bound and out-bound server traffic unless their is a specific need.

Of course, now that I have a bit more understanding of ARM chips, adding a SPI NAND flash chip to the IPMI controller could be done. If it was earlier in the boot order than the "main" flash chip, it could boot the system and possibly run the "main" code in a virtual environment.
 
Last edited:

ornias

Neophyte Sage
Joined
Mar 6, 2020
Messages
1,370
Somewhat surprisingly, Bloomberg is digging up this story again.

I think the original stories did not have any names attached, this time they have a handful of people on the record saying some of this took place.
Eric, be VERY carefull reading.

They have no one on record confirming it. They used very sneaky writhing to frame totally different comments about the general issue that was presented to the interviewed people.

I've carefully read the article and it's just a more sneaky and carefull way of writhing precisely the same thing they did in the first article. Actually this is worse: Instead of being clear it is based on nothing-but thin air, this time they actually tried to burry the thin-air in sneaky use of language.
 

Arwen

Neophyte Sage
Joined
May 17, 2014
Messages
1,354
@ornias Ah, another stock scam. Or competitor trying to boost their sales.

Because of their prior story, I mostly ignore Bloomberg. Just more reason now.
 

ornias

Neophyte Sage
Joined
Mar 6, 2020
Messages
1,370
The first 3 allegations have no source. More importantly: It is writhen as if those proposed incidents are general knowlage.
The second of which, has nothing to do with chips.

From this they go all the way to:
"Each of these distinct attacks had two things in common: China and Super Micro Computer Inc. "
While no one in the previous "sources" confirmed either of those things.

But they mostly do things like this:
"according to 14 former law enforcement and intelligence officials familiar with the matter."
Do we really believe 14(!) people risked lifetime prison scentences for snitch about this "big deal" national security threat?! Thats bizar. Either that was leaked on purpose or they are bull****ting again.

"or FISA, according to five of the officials."
They had 5 people confirming what FISA means, not neccisarily the previous statements.
It's also funny how they made a link from the FISA name, to make it seem like they link to a warrant, which is obviously not the case.

"when agents started monitoring the communications of a small group of Supermicro workers,"
They didn't actually confirm if said investigation was fruitfull at all. Just that they started it.

"according to an adviser to two security firms that did the work."
It's highly unlikely that the same person with security clearance works at two contractors at the same time, gets asked questions about the same thing AND is able to leak it without sitting in a blacksite now.

But the most obvious fraud evidence is this portion:
"The Supermicro saga demonstrates a widespread risk in global supply chains, said Jay Tabb, a former senior FBI official who agreed to speak generally about China’s interference with the company’s products. "

The second part "who agreed to speak generally about China’s interference with the company’s products.", proves he would never had named SuperMicro.

What he did actually say was:
“Supermicro is the perfect illustration of how susceptible American companies are to potential nefarious tampering of any products they choose to have manufactured in China,”
Not that the supermikro thing was actually the case.

It's important to note here, that they left out the question asked (context) for the answer they wrote down.

"former U.S. officials who provided information for this story emphasized that the company itself has not been the target of any counterintelligence investigation."
Which contradicts their earlier statements about FISA warrants to keep tabs on SuperMicro employees. Thats called a counterintelligence against a company. Because companies consist of paperwork and employees.

" said Mike Janke, a former Navy SEAL who co-founded DataTribe, a venture capital firm."
So the advisor above, is a venture capitalist, who specialises in:
"a global cyber foundry that invests in and co-builds next-generation cybersecurity and data science companies"

Guess who gained from this BloomBerg Article?

"said the two companies are not allowed to speak publicly about that work but they did share details from their analysis with him."
Sorry, but this would never be allowed. Companies are DEFINATELY not allowed to discuss classified research with investors. Thats bull****.

Anyway:
After this they go on-and-on about other attackes using that modus operandi.
However: No one disputes these attacks are impossible or never done. People question the specific Super Micro hack.

" The company said Bloomberg had assembled “a mishmash of disparate and inaccurate allegations” that “draws farfetched conclusions.”"
I think is a nice way of describing this article.

TLDR:
Unless the few sources Named in this article disappear within a week, I take it all as complete stock manipulation BS.
 

naskit

Neophyte
Joined
Apr 19, 2021
Messages
7
Mod note: Moved here from a different thread
Is anyone here aware of the security vulnerabilities found on SuperMicro hardware? Their supply chain is corrupted and SM devices have been found to have secretly embedded chips in the Network Interface components making very suspicious connections back to unverified addresses. I do not trust SM because I have serious concerns over geo-political risk and data security.
 
Last edited by a moderator:

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
17,060
Given the discussion up-thread, what you say is false and backed by nothing more substantial than hand-waving.
 

danb35

Wizened Sage
Joined
Aug 16, 2011
Messages
12,070
Anyone here other than everyone who's commented in this thread, you mean? Funny, nobody's ever gone public with one of those SuperMicro boards that "have been found to have secretly embedded chips in the Network Interface components making very suspicious connections back to unverified addresses".
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
17,060
And two years is plenty of time to do one or all of the following:
  1. Produce an x-ray image of a fancy implant inside the PCB itself (quite the advanced manufacturing process)
  2. Produce a visible light photograph of a surface-mount implant
  3. Produce an analysis of a ROM dump of either the BMC or system firmware with a figurative big red arrow pointing to the evil code
Since exactly none of them happened, the story is as close to fantasy as bigfoot.
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,795
Mod note: Moved here from a different thread
Is anyone here aware of the security vulnerabilities found on SuperMicro hardware? Their supply chain is corrupted and SM devices have been found to have secretly embedded chips in the Network Interface components making very suspicious connections back to unverified addresses. I do not trust SM because I have serious concerns over geo-political risk and data security.
If you have "serious concerns over geo-political risk and data security", why in the world would you let your IPMI make random connections out to the Internet?

Conspiracy theories often run afoul of the most obvious problems.
 

naskit

Neophyte
Joined
Apr 19, 2021
Messages
7
Thanks everyone for your comments. Many interesting views. I will concede this: I am sick of my country being sold to CN by pollies for short term gain and back-pocketing...things have gone downhill in many ways, and 2020 proved to us how vulnerable we (in the west) have let ourselves become because of our neglect to watch the world-wide supply-chain as a whole...That aside, I am not a fan of Bloomberg for any reason. That was just where I found the article.

But my point was this: my brother designs the CAD software used to design Firmware programs for FPGAs etc, and I want to point out I did not mention anything about IPMI or the BMC. I work in network security and know something about TCP/IP, and in my mind it is totally feasible for there to be a NIC with an embedded (and hidden) chip that could copy packets and launch outbound connections to foreign addresses unbeknown to the user, or even the system. I don't know what exact components are used physically on the mobo, but if it's possible to copy traffic on a cable, I see no reason why packets could not be copied without interruption to the normal packet flow and without the mobo being any wiser to what is going on...

The whole reason I raised this was because I have lost (a lot) of trust in the worlds dominant supply chain, FreeNAS is all about data security (granted, we normally mean integrity, but still, privacy is almost of equal importance).

Why would anyone who cares about their data not being lost (to the GME) not also care about it being secure from unwelcome viewers?

It has become abundantly clear that various governments (and perhaps other ISPs and other orgs) are collecting personal data and online behaviour wholesale without warrants and without even informed consent. HW can be compromised further up the supply chain and could be aiding those nefarious data collection activities.

I just wanted to know if anyone else here had the same concerns about the integrity and security of the HW they were choosing for their NAS.
I would venture to disagree with those of you who think/assume there is nothing to be concerned about regarding HW integrity based on country of manufacture and the trustworthy nature of certain governments where HW is made. That's all. And you're free to disagree with me (because we do not live on one of those countries with such a controlling regime).
 
Last edited:

danb35

Wizened Sage
Joined
Aug 16, 2011
Messages
12,070
I think it's important to distinguish between two very different, though somewhat-related, assertions:
  • Our IT supply chain is vulnerable
  • Our IT supply chain has been compromised, in a specific way, at a specific time
It sounds like you're conflating the two. I don't think anyone can reasonably argue against the first, particularly when a great deal of it involves a country that isn't entirely friendly to the United States (or to the western world in general). But that isn't the assertion you made--you said, in a post you joined this forum to make, that SuperMicro hardware has been found to have specific vulnerabilities due to a specific compromise. And you've shown no evidence of that. You didn't even point to the Bloomberg articles, the only known source of such an assertion, which also contain no evidence to support it. Nor do you show any indication of understanding the claims that are made (no, you didn't mention IPMI or the BMC, but Bloomberg did), nor of any of the discussion that happened (even in this very thread, much less anywhere else) regarding these bombshell allegations.
 

ornias

Neophyte Sage
Joined
Mar 6, 2020
Messages
1,370
I see no reason why packets could not be copied without interruption to the normal packet flow and without the mobo being any wiser to what is going on...
Once people start overestimating their knowhow, discussions start to die.
This is not feasable without it, at least, being relatively easy to figure out when you know what to look for.

People aren't saying it isn't possible, people are saying no one proven Bloomberg was right that it WAS happening.


I just wanted to know if anyone else here had the same concerns about the integrity and security of the HW they were choosing for their NAS.
No. I pick my hardware carefully and don't rely on a single hardware device for data-transfer security.
My NIC's aren't directly connected to the internet and my router nic, does not have a way of accessing most of my internal datastreams.
 
Status
Not open for further replies.
Top