TrueNAS or hardware issue - Drives were modified after installation (HPA's, DCO's, remapped sectors)

[Love4Storage]

If your current working hypothesis is that your workstation is compromised--particularly in such a way that it's able to execute such an attack against your NAS--then it seems you're focusing on the wrong thing here. Any backups would themselves be untrustworthy, and any other device you'd attach could just as well be compromised.

This assumption is made because otherwise it's my NAS motherboard, but this has happened to two motherboards. Current one being Supermicro, the second being Tyan. I also believe something similar happened with an HP server, but I could have reused drives in this situation.

The workstation is the only way in, wouldn't you say? In this current instance with the supermicro board, if someone had control over the board, they wouldn't need SSH/root access and could easily create/modify the drives IF they had a connection externally. As mentioned previously, the NAS is a direct connect to the workstation and no connection to LAN.

I do not enable SSH on my NAS ever.

 

[Love4Storage]

However, I see no benefit to a malicious actor going and adjusting disk sizes to create a hidden partition, they would be interested in data, and with fileless malware that can be implanted to run at boot, they would just do that to extract data from your NAS if they wanted to, or encrypt it, or what ever from your workstation.

I totally agree with this logic, but the end goal is usually unique. Like to slow me down, hinder etc. The files in my NAS are not what they are after.

 

[MrGuvernment]

I totally agree with this logic, but the end goal is usually unique. Like to slow me down, hinder etc. The files in my NAS are not what they are after.

Not usually, the end goal is to steal data and make a profit in some form and /or use your resources for bot nets or other malicious attacks to originate from, whether selling your data direct, or seeing if you have any useful info or logins for something that could be of value. The end goal is profit for malicious actors, not to slow you down, they do not even want you to know they are there, why doing something like adding in a random partition of 40Gb makes no sense.

 

[Love4Storage]

Not usually, the end goal is to steal data and make a profit in some form and /or use your resources for bot nets or other malicious attacks to originate from, whether selling your data direct, or seeing if you have any useful info or logins for something that could be of value. The end goal is profit for malicious actors, not to slow you down, they do not even want you to know they are there, why doing something like adding in a random partition of 40Gb makes no sense.

It's entirely targeted. And you're right, I may not know they are there, I just pick up on the hints.

The 40GB area can be used to conceal code or executables. HPA/DCO's were used by manufacturer's to store data and can easily be used to store other things. It's somewhat of a trojan horse...? I know TrueNAS isn't the problem, but to be frank, until I understand how this is happening I will resort to other means to store/backup data. Safest thing to do at this point.

 

[NickF]

TLDR; My opinions mixed with some anecdotal information.

• I APPRECIATE THE EFFORTS IN TRYING TO UNDERSTAND AN ISSUE, ALWAYS ASK QUESTIONS
• WHEN TRYING TO UNDERSTAND A PROBLEM, START AT THE BOTTOM AND WORK YOUR WAY UP.
  • YOU STARTED FROM THE TOP AND WORKED YOUR WAY DOWN.
• PLEASE ASK QUESTIONS BEFORE JUMPING TO CONCLUSIONS.
• USING CONSISTENT UNITS OF MEASUREMENTS MATTERS.
• IT'S DANGEROUS TO BE CONFIDENT IN YOUR CONCLUSIONS WITHOUT ASKING QUESTIONS.
• BEFORE MAKING CLAIMS OF MALFEISANCE AND/OR QUESTION SECURITY, PLEASE DOUBLE CHECK YOUR WORK.
  • YOU WENT FROM 0-180 HERE BEFORE STOPPING TO CONSIDER ASKING QUESTIONS

I'm still not entirely convinced, by your data, of anything? In what limited data you've shared I don't see anything at all...Other than SSD sizes being weird (and potentially mis-marketed a bit)...
A 10 TB Hard Drive is really only a bit over 9 TiB. TiB are all that really matter.

SSDs (and flash storage in general) make this worse by having weird sizes in the other direction. They are sometimes marketed as 128GB but they actually mean 128GiB...

A fun way to prove this? Take a 8GB flash drive from two or three different manufacturers. Try and DD exactly 8GB to them. Then try and DD 8GiB to them. On top of the weird problems with GB vs GiB you'll likely find those flash drives are neither 8GB or 8GiB, but somewhere in between. Lets refer to this as the fudge factor.

How do I know all of this? I had to DD thousands of flash drives and SSDs over the last few years. Don't ask. But also don't take my word for it. Feel free to audit my claim. I don't know if this same phenomenon exists with HDDs though, never really thought about it till just now.

Because of history, and fun, we've had to create multiple definitions of what a "byte" is.

Then, add the filesystem? Iin this case, ZFS, will inherently use some space.
Let me provide a hypothetical example:

• In RAIDZ1, one of the disks' worth of space is reserved for parity. So, with 4 x 1TB drives, you'll effectively have 3TB of usable space.
• ZFS reserves a small portion (around 1/64th) of the disk space for its "slop space". This is used to ensure that ZFS doesn't run out of space for its administrative tasks.
  
  
• Code:

  The base-10 (decimal) system, where:
  1 kilobyte (KB) = 1,000 bytes
  1 megabyte (MB) = 1,000 KB = 1,000,000 bytes
  1 gigabyte (GB) = 1,000 MB = 1,000,000,000 bytes
  
  
  The base-2 (binary) system, where:
  1 kibibyte (KiB) = 1,024 bytes
  1 mebibyte (MiB) = 1,024 KiB = 1,048,576 bytes
  1 gibibyte (GiB) = 1,024 MiB = 1,073,741,824 bytes

For a 3TB setup, this would be around 51539607552 bytes.
Converting this to GiB (using the base-2 measurement): 46,875,000,000,000 bytes ÷ 1,073,741,824 bytes/GiB .... ish 40GiB

So, the "slop space" would be about 43,690.48 GiB for a 3TB (base-10) setup.

Why Don't you double check my math with your own disks :P I don't think you said, but I am assuming your 6 480GB Drives are in a RAIDZ2. Make sure you consider the fudge factor and use the right (or at least consistent) units of measurement.

HINT: The math will be damn close

(Samsung SM863 480GB) all 6 of them had DCO's (hidden areas much like an HPA - "hidden protected area") the size of 40GB's which were not originally on these drives. I have never overprovisioned the drives with any tools or purposefully created these DCO's, they just appeared sometime over the course of 6 months.

Everything seems to be explainable here. No whacky conspiracy theories IMO. I'll let others jump in with supporting information here that aren't just my personal anecdotes.

 

[danb35]

I know TrueNAS isn't the problem,

...but you still created a thread entitled "TrueNAS' bad security." You've made claims that just don't make any sense (like that these DCOs could somehow be created on a machine running a BSD or Linux operating system, but couldn't be seen by the tools of that operating system). You still haven't shown any real evidence of, well, anything you've claimed. And your stated hypotheses to explain what you think you're seeing are growing increasingly, um, odd--from TrueNAS itself being the problem, to your workstation being compromised, to the motherboard on your NAS being the attack vector (which is reminiscent of Bloomberg's discredited allegations).

Accepting at face value what you seem to be suggesting but won't outright say--that you're being "Mossaded upon" as Allan Jude would say, or that you're being directly targeted by a state-level (or near-state-level) actor--I doubt any of us can do much to help you. But I don't think you've yet shown reason to believe this is the case.

 

[Ericloewe]

If I was trying to be sneaky, a 40 GB slice of disk space would be far down the list. But let's examine the points raised:

1. New sneaky squirrel slices of disks are appearing
   1. There is little evidence that this is the case. A tool is reporting that they exist, with no indication of size. They might have always been there and they may be as pedestrian as a "write new firmware here" area to work around limitations of the ATA protocol.
   2. There is no indication that they actually contain anything
2. Disks being mislabeled by parted magic
   1. Both the incorrect size and incorrect identification are easily explained by general dodginess of the tool
   2. There is no indication that the disks have shrunk
   3. The reported manufacturer is not relevant information, in that it is not reported by the disks
3. Virus scan is flashing up like a Christmas tree
   1. The reported detections are less than convincing... "Skipped from scan" does not mean "malicious", and "password protected" calls into question the quality of the scan carried out.

 

[Love4Storage]

...but you still created a thread entitled "TrueNAS' bad security." You've made claims that just don't make any sense (like that these DCOs could somehow be created on a machine running a BSD or Linux operating system, but couldn't be seen by the tools of that operating system). You still haven't shown any real evidence of, well, anything you've claimed. And your stated hypotheses to explain what you think you're seeing are growing increasingly, um, odd--from TrueNAS itself being the problem, to your workstation being compromised, to the motherboard on your NAS being the attack vector (which is reminiscent of Bloomberg's discredited allegations).

Accepting at face value what you seem to be suggesting but won't outright say--that you're being "Mossaded upon" as Allan Jude would say, or that you're being directly targeted by a state-level (or near-state-level) actor--I doubt any of us can do much to help you. But I don't think you've yet shown reason to believe this is the case.

Fair enough, I've changed the title of the post. Equally, I was actually expecting this sort of reation more than anything. If anyone comes across this post in the future I guess the only impact I've made would be that they're not the only one.

I really didn't want to get into what I do, who I am, and who these people are. I guess it was short sighted of me posting this on a TrueNAS community site; but I was at a point where I didn't really know where to go. The main question for me was just: can I continue using TrueNAS, you've sufficiently answered my question.

[EDIT] I just read the "supermicro" link in your response @danb35 You need to chill out a bit. TrueNAS is truenas, but you do know what people have to say about the community. Do a simple search on truenas vs. unraid. I never understood what this meant till this post.

 

[danb35]

Equally, I was actually expecting this sort of reation more than anything.

You should have been expecting this--when you make claims, particularly claims that many (most with far more IT/infosec background than I) find somewhat implausible, you should expect to be asked for, and to provide, evidence to support them. Almost 30 posts into this thread, you have yet to do so. And now it seems you want to play the victim, I guess because we didn't unquestioningly accept your claims as true. If that's how you want to act, that's up to you, but it doesn't get you or us any closer to figuring out what's going on. If, on the other hand, you're trying to understand what's happening, maybe you could try answering the questions you've been asked throughout this thread. The question comes down to this: what were/are you trying to accomplish by posting here?

I really didn't want to get into what I do, who I am, and who these people are.

You can't have it both ways. You can't expect us to rely on your knowledge and expertise when you don't tell us anything about them. You can't expect us to understand the nature of the threat you think you're facing when you won't tell us about it. If you want to keep that information private, that's certainly your call, but then the only rational way for us to respond to you is as someone who has (or at least has demonstrated) no particular knowledge of, well, anything. You could demonstrate that knowledge, to a degree, by providing some evidence to support your claims--but you've thus far chosen not to do that.

You need to chill out a bit. TrueNAS is truenas, but you do know what people have to say about the community.

Oh no, someone said I was mean. Certainly I was out of line when I spoke sternly to someone who revived a dead thread to ask if anyone knew about the claims that were made in that very thread. /sarc

 

[NickF]

I really didn't want to get into what I do, who I am, and who these people are. I guess it was short sighted of me posting this on a TrueNAS community site; but I was at a point where I didn't really know where to go. The main question for me was just: can I continue using TrueNAS, you've sufficiently answered my question.

I don't understand the relevance here of any of that? You made broad, sweeping claims of a very specific problem you believed to be tied into some sort of conspiracy theory. A claim to which you provided no supporting evidence and only your suspicions.

I'm fine with you having suspicions and asking questions. Thats why we are here. But when you make a post clearly stating your conclusion before you've understood the details demonstrates a lack of responsibility and troubleshooting skills. This is especially dangerous when you are claiming security.

Next time start with your questions, not your conclusions. Don't let your overwhelming sense of urgency, warranted or not, consume your ability to think critically. There were many questions, all answered in this thread, that you could have been asking before coming to a conclusion. I'm not sure what was short-sighted about any of our answers here.

I think my post above fairly robustly explains....there is nothing to see here? Slow down my dude.

 

[HoneyBadger]

As a bit of a Carl Sagan fan, I'm reminded of his aphorism that "extraordinary claims require extraordinary evidence" - or perhaps related, Occam's Razor, stating that the simplest explanation is usually the correct one. "When you hear hoofbeats, think horses, not zebras."

@Love4Storage I'd be happy to review a debug file in confidence sent via PM.

With regards to security, both TrueNAS and OpenZFS are maintained as open-source, with their contents publicly viewable on GitHub:

https://github.com/truenas/

https://github.com/openzfs/

iXsystems also maintains the TrueNAS Security Advisories page to monitor for, identify, analyze, and categorize potential vulnerabilities.

https://security.truenas.com/