TrueNAS or hardware issue - Drives were modified after installation (HPA's, DCO's, remapped sectors)

[Love4Storage]

This is a story that some may believe, some may doubt, and some may have an answer to. This post is meant for the few (if any) who are able to answer this.

So, let's say I am a person of interest. I can not go into detail about this part, but I don't live a normal life.

PROBLEM
I am unsure of the security of either TrueNAS or the hardware I have used for TrueNAS. Starting with the conclusion, I recently found that the drives I used (Samsung SM863 480GB) all 6 of them had DCO's (hidden areas much like an HPA - "hidden protected area") the size of 40GB's which were not originally on these drives. I have never overprovisioned the drives with any tools or purposefully created these DCO's, they just appeared sometime over the course of 6 months.

BACKGROUND
My previous TrueNAS builds, something similar happened where 40GB hidden area's were created on a pool of 8 ssd sata drives. I had to use proprietory software (Blancco) which can see HPA's and DCO's - most software will not be able to see this hidden area. It is NOT a partition. I have used TrueNAS for bigger projects and have been attacked each time (either HPA's or DCO's would appear on SATA drives which could NOT be deleted or removed).

This happened with multiple different boards/systems: HP, Tyan and supermicro. For HP the HBA's were non-IT mode HBAs. For the Tyan system, an LSI HBA in IT mode was used. This also happened in a 13th gen Dell server. Lastly, and most recently was a supermicro X10SDV board with sata ports. I do not re-use USB's and when I create bootable USB's I make them read-only. USB's are easy attack vectors so I throw out USB's quite frequently after use.

After installing TrueNAS perhaps a week ago, I used some clean sata drives (SSDs) to create a pool. I transfered data over to the pools and CAM errors started occuring for around 24 hours. This stopped afterwards and no errors showed up on the consosle. The drives were tested later to find they did not have DCO's or HPA's, but were somehow modified (Blancco reports DCO: "unidentifiable", HPA: "unidentifiable" - when the baseline should be "doesn't exist") with remapped sectors.

Mod note: removed broken attachment

Stranger things have happened where drives that were used in TrueNAS would show up as different capacity (6TB drives = 2TB drives) or even HGST SAS SSD's would show as Seagates using parted magic. Parted magic uses SMART values to reflect this info so either the drives are lying or the SMART value was modified.

Mod note: removed broken attachment


QUESTIONS
1. Does TrueNAS have a feature that creates a hidden protected area that can not be removed with NWIPE, DBAN, DD, Blancco, or other low level formatting (flushing the nand flash via firmware or using DoD wiping)?
// These hidden areas can not be removed via BSD/linux commands and it is impossible to see what someone is hiding in this area.

2. Currently, Blancco is the only tool I know of that can see and attempt to delete these areas. Each time they're discovered, Blancco will fail to remove the DCO (or HPA). Has anyone had this experience? What tool did they use to wipe the drives or get rid of the hidden areas?

3. I almost always checksum anything I download and install onto a system so I'm ruling out MITM attacks. During virus scans of any TrueNAS ISO, it always encounters multiple files which are password protected, and can not be checked for malware/viruses. Why is this? The same is for pfsense, is it some BSD thing?

 

[danb35]

These hidden areas can not be removed via BSD/linux commands and it is impossible to see what someone is hiding in this area.

This seems, at best, questionable--TrueNAS runs on either FreeBSD (CORE) or Linux (SCALE), so anything it would (or could) have created, would have been created with "BSD/linux commands." That such commands subsequently couldn't detect what had happened strikes me as unlikely, though I wouldn't call myself a "person of interest" (whatever that's supposed to mean).

During virus scans of any TrueNAS ISO, it always encounters multiple files which are password protected, and can not be checked for malware/viruses. Why is this?

Surely it would be easier to answer this question if you identified the files in question, and perhaps the virus-scanning tool that's reporting this.

 

[Arwen]

The Linux tool hdparm should be able to both see and change any HPA. That said, it's possible that a vendor created special firmware that was not standard SATA compliant, and thus won't work with normal SATA tools.

I don't know if their is a FreeBSD tool that can modify HPAs.

As for an existing TrueNAS ZFS pool working after a HPA was created, no, it would not. ZFS does NOT allow shrinking of pool devices and any attempt to do so would cause pool corruption. Very noticeable, and likely causing data loss.


For those unfamiliar with HPAs, here is what the manual page for hdparm has to say about them;

-N Get/set max visible number of sectors, also known as the Host Protected Area setting. Without a
parameter, -N displays the current setting, which is reported as two values: the first gives the
current max sectors setting, and the second shows the native (real) hardware limit for the disk.
The difference between these two values indicates how many sectors of the disk are currently hidden
from the operating system, in the form of a Host Protected Area (HPA). This area is often used by
computer makers to hold diagnostic software, and/or a copy of the originally provided operating
system for recovery purposes. Another possible use is to hide the true capacity of a very large
disk from a BIOS/system that cannot normally cope with drives of that size (eg. most current {2010}
BIOSs cannot deal with drives larger than 2TB, so an HPA could be used to cause a 3TB drive to re-
port itself as a 2TB drive). To change the current max (VERY DANGEROUS, DATA LOSS IS EXTREMELY
LIKELY), a new value should be provided (in base10) immediately following the -N option. This
value is specified as a count of sectors, rather than the "max sector address" of the drive.
Drives have the concept of a temporary (volatile) setting which is lost on the next hardware reset,
as well as a more permanent (non-volatile) value which survives resets and power cycles. By de-
fault, -N affects only the temporary (volatile) setting. To change the permanent (non-volatile)
value, prepend a leading p character immediately before the first digit of the value. Drives are
supposed to allow only a single permanent change per session. A hardware reset (or power cycle) is
required before another permanent -N operation can succeed. Note that any attempt to set this
value may fail if the disk is being accessed by other software at the same time. This is because
setting the value requires a pair of back-to-back drive commands, but there is no way to prevent
some other command from being inserted between them by the kernel. So if it fails initially, just
try again. Kernel support for -N is buggy for many adapter types across many kernel versions, in
that an incorrect (too small) max size value is sometimes reported. As of the 2.6.27 kernel, this
does finally seem to be working on most hardware.

 

[Ericloewe]

I find it curious that you posted two functioning screenshots that show nothing particularly out of the ordinary (apart from one the tools misreporting a disk's size as 2 TB, which is rather conspicuously the limit imposed by the MBR partition table format), alongside two broken screenshots that would presumably show something more out of the ordinary.

 

[Ericloewe]

As for an existing TrueNAS ZFS pool working after a HPA was created, no, it would not. ZFS does NOT allow shrinking of pool devices and any attempt to do so would cause pool corruption. Very noticeable, and likely causing data loss.

There is some nuance to this, as ZFS pools are aligned to some sort of metaslab boundary. I won't pretend to know the details, but I recall this being up to a few GB. 40 GB sounds like a lot, though.
I'm sure zdb would also be very informative in this scenario, but I won't pretend to know how to use it.

 

[danb35]

The Linux tool hdparm should be able to both see and change any HPA.

...and it appears it can also manage DCOs as well. From its manpage:

Code:

       --dco-freeze
              DCO stands for Device Configuration Overlay, a way for vendors to selectively disable certain features of a drive.
              The --dco-freeze option will freeze/lock the current drive configuration, thereby preventing software (or malware)
              from changing any DCO settings until after the next power-on reset.

       --dco-identify
              Query and dump information regarding drive configuration settings which can be disabled by the vendor or  OEM  in‐
              staller.   These  settings show capabilities of the drive which might be disabled by the vendor for "enhanced com‐
              patibility".  When disabled, they are otherwise hidden and will not show in the -I identify output.  For  example,
              system  vendors sometimes disable 48_bit addressing on large drives, for compatibility (and loss of capacity) with
              a specific BIOS.  In such cases, --dco-identify will show that the drive is 48_bit capable, but -I will  not  show
              it, and nor will the drive accept 48_bit commands.

       --dco-restore
              Reset  all  drive  settings,  features,  and accessible capacities back to factory defaults and full capabilities.
              This command will fail if DCO is frozen/locked, or if a -Np maximum size restriction has also been set.   This  is
              EXTREMELY DANGEROUS and will very likely cause massive loss of data.  DO NOT USE THIS COMMAND.

Example output:

Code:

root@truenas[~]# hdparm --dco-identify /dev/sdg

/dev/sdg:
DCO Checksum verified.
DCO Revision: 0x0002
The following features can be selectively disabled via DCO:
    Transfer modes:
         mdma0 mdma1 mdma2
         udma0 udma1 udma2 udma3 udma4 udma5 udma6
    Real max sectors: 18446744072157735600
    ATA command/feature sets:
         SMART self_test error_log security PUIS HPA 48_bit
         streaming FUA selective_test
         WRITE_UNC_EXT
    SATA command/feature sets:
         NCQ NZ_buffer_offsets interface_power_management SSP

So, @Love4Storage, both HPAs and DCOs are defined in the ATA spec, even if not especially well-known. The Linux tool hdparm, which is included with TrueNAS SCALE, appears to be able to deal with both of them. It would be interesting to see what that tool reports for the disks you believe are affected.

 

[danb35]

The more I think about it, the more this sounds like a troll. OP who hasn't posted in almost a year, posts a thread with a deliberately provocative title, containing facially implausible allegations, omitting the supposed "proof", and hasn't been back since to address the very reasonable questions raised. I guess it has only been 36-ish hours, but...

 

[Love4Storage]

Surely it would be easier to answer this question if you identified the files in question, and perhaps the virus-scanning tool that's reporting this.

These are the files in question, usually it's 3 files; attachment of screenshot for result of scan:

Log File

TrueNAS-13.0-U5.3.iso=>(usr=>local=>lib=>python3.9=>test=>__pycache__=>test_zipfile.cpython-39.opt-2.pyc (Rescan 1))=>zero

TrueNAS-13.0-U5.3.iso=>TrueNAS=>Packages=>base-os-13.0-U5.3-ba7da45e7a5cb84a2cf26135b479472d.tgz=>(gzip)=>usr=>local=>lib=>python3.9=>test=>__pycache__=>test_zipfile.cpython-39.opt-1.pyc=>zero

TrueNAS-13.0-U5.3.iso=>(usr=>local=>lib=>python3.9=>test=>__pycache__=>test_zipfile.cpython-39.pyc (Rescan 2))=>zero

I use bitdefender. Does anyone else run scans to find some files within the ISO was skipped?

I was away for part of the weekend and was not able to respond. What I mean by person of interest: my work puts me in difficult situations with people in power. I use precaution in everything I do, but am forced to deal with things some people wouldn't put up with.

And no I am not trolling, I'm merely reaching out for help.

 

[Love4Storage]

As for an existing TrueNAS ZFS pool working after a HPA was created, no, it would not. ZFS does NOT allow shrinking of pool devices and any attempt to do so would cause pool corruption. Very noticeable, and likely causing data loss.

I've been using FreeNAS/TrueNAS for quite some time now. I understand this to be true.

Better question would be how would an HPA/DCO be created remotely assuming that the system is NOT connected to any LAN. I direct connect my NAS only to one device. Assuming my workstation is compromised, without having root & shell access how could this be engineered? The HPA's would have to be created prior to the pool being created, right? Not once the pool is created? Due to the delta in pool size.

So one way of checking to see if drives are compromised is if I make a pool first, check to see if the drives have been modified, then use them for storage afterwards?

 

[Love4Storage]

I find it curious that you posted two functioning screenshots that show nothing particularly out of the ordinary (apart from one the tools misreporting a disk's size as 2 TB, which is rather conspicuously the limit imposed by the MBR partition table format), alongside two broken screenshots that would presumably show something more out of the ordinary.

This was just to illustrate that this was happening. I'm sure it's not happening to everyone else; just that it was happening. Hoping to find how and what I could do about this. The focus is more on whether I can continue using truenas or if I had to resort to another tool for backups; ie. external drives, USB's or Synology/QNAP

 

[Love4Storage]

I'm sure zdb would also be very informative in this scenario, but I won't pretend to know how to use it.

What command line would I use? I understand it's to debug a pool but would I get anywhere with dumping the output here?

 

[Love4Storage]

So, @Love4Storage, both HPAs and DCOs are defined in the ATA spec, even if not especially well-known. The Linux tool hdparm, which is included with TrueNAS SCALE, appears to be able to deal with both of them. It would be interesting to see what that tool reports for the disks you believe are affected.

ABSOLUTELY! This occurs with SATA drives as I don't use SAS drives anymore (heat and noise). I try to stick with sata SSD's to be electricity bill friendly.

So one way of reducing the attack surface would be to not use SATA drives for storage?

 

[Love4Storage]

The more I think about it, the more this sounds like a troll. OP who hasn't posted in almost a year, posts a thread with a deliberately provocative title, containing facially implausible allegations, omitting the supposed "proof", and hasn't been back since to address the very reasonable questions raised. I guess it has only been 36-ish hours, but...

I was away for the weekend, I do appreciate that people have responded. To be honest, I didn't think people would be this helpful; or ready to help. I am no troll, I do appreciate truenas very much and have been using it for a decade or more. I do appreciate the community and the people around the software because it really is them that make truenas feasible.

 

[Love4Storage]

I just realized, some of my images were deleted by the mod. Or the files were broken?

Here is one where parted magic (paid version) reports HGST SAS SSD's as Seagates. These are Hitachi's... definately not Seagates:

 

[Ericloewe]

Or the files were broken?

They were, please feel free to (re)upload anything you deem useful. I deleted the originals so people wouldn't waste their time trying to open corrupted files.

 

[Ericloewe]

I just realized, some of my images were deleted by the mod. Or the files were broken?

Here is one where parted magic (paid version) reports HGST SAS SSD's as Seagates. These are Hitachi's... definately not Seagates: View attachment 69742

I would put that down to parted magic's database being incorrect. I think SMART data does not explicitly include the manufacturer, just the model number.

 

[danb35]

TrueNAS-13.0-U5.3.iso=>(usr=>local=>lib=>python3.9=>test=>__pycache__=>test_zipfile.cpython-39.opt-2.pyc (Rescan 1))=>zero

I can't find any such file on the .iso--there's no /usr directory there at all.

TrueNAS-13.0-U5.3.iso=>TrueNAS=>Packages=>base-os-13.0-U5.3-ba7da45e7a5cb84a2cf26135b479472d.tgz=>(gzip)=>usr=>local=>lib=>python3.9=>test=>__pycache__=>test_zipfile.cpython-39.opt-1.pyc=>zero

...but if I extract the base-os .tgz file, this directory structure and file are present. It is, as I'd have expected from the file extension, byte-compiled Python code:

Code:

root@truenas[.../local/lib/python3.9/test/__pycache__]# file test_zipfile.cpython-39.opt-1.pyc
test_zipfile.cpython-39.opt-1.pyc: python 3.9 byte-compiled

It appears that projects like https://github.com/rocky/python-decompile3 are able to decompile such code.

The focus is more on whether I can continue using truenas or if I had to resort to another tool for backups; ie. external drives, USB's or Synology/QNAP

If your current working hypothesis is that your workstation is compromised--particularly in such a way that it's able to execute such an attack against your NAS--then it seems you're focusing on the wrong thing here. Any backups would themselves be untrustworthy, and any other device you'd attach could just as well be compromised.

without having root & shell access how could this be engineered?

So you don't have SSH enabled on the NAS? Presumably you still have access to the web UI. And if that's the case, a clever attacker could presumably remotely execute shell commands that way. If the (hypothetical) attacker has compromised your workstation, he has access to any credentials you would have used to log in to the web UI. And since the web UI includes a (very crappy) shell, there's the shell access. Any malware could have been put onto the NAS by way of its normal file-sharing features from your workstation.

 

[wsoteros]

I was away for the weekend, I do appreciate that people have responded. To be honest, I didn't think people would be this helpful; or ready to help. I am no troll, I do appreciate truenas very much and have been using it for a decade or more. I do appreciate the community and the people around the software because it really is them that make truenas feasible.

Thank you so much for the kind words about the community.

We try our best to make the community a welcoming place!

 

[MrGuvernment]

As most of this is over my head, my first thought could this not be the over allocated space SSD's use for cache when onboard ram gets full?

I use bitdefender. Does anyone else run scans to find some files within the ISO was skipped?

You could extract said file and upload to virus total just to see, while we like to think AV is a solid solution, reality is in today's world malicious actors can often easily get passed most solutions and their tools go undetected (if this was a worse case scenario and a compromise)

However, I see no benefit to a malicious actor going and adjusting disk sizes to create a hidden partition, they would be interested in data, and with fileless malware that can be implanted to run at boot, they would just do that to extract data from your NAS if they wanted to, or encrypt it, or what ever from your workstation.

 

[WI_Hedgehog]

You know how all of these Chinese clone HBA and network cards are appearing on eBay new for unbelievably cheap prices?

“Nothing in life is free, you always pay in the end.” – Wayne Static

(perhaps that's not the OP's issue, but without more information it remains a possibility)