Can on board X9DR3-F Supermicro mini-sata (x4) be used for Truenas?

[ZiggyGT]

this is the error I get from the replication task Pool2-Pool3. I would appreciate any help you can provide

Replication "pool2 - Pool3" failed: cannot unmount '/var/db/system/samba4': pool or dataset is busy warning: cannot send 'pool2@auto-2022-11-06_19-10': signal received..​

 

[artlessknave]

something is preventing the umount of /var/db/system/samba4, which is very weird, because you should not be able to replicate that.
have you tried rebooting to see if it clears.....whatever the heck is causing that error?

 

[ZiggyGT]

something is preventing the umount of /var/db/system/samba4, which is very weird, because you should not be able to replicate that.
have you tried rebooting to see if it clears.....whatever the heck is causing that error?

I have rebooted. I get this same error every time the replication runs. Is there any information here that can help as to why? Pool3 is not in any samba share. I'll try again to reinitiate the replication

 

[artlessknave]

at first I had no idea, but a google search found a reddit

Reddit - Dive into anything

www.reddit.com

it's not the same error, but it's probably the same cause: exclude your system dataset from the replication. it'll be in use by smb.
should be pool2/.system.
you can also exclude it from your snapshots; it needs to exist but there is no real point backing it up.

you can also move it to your boot pool, or another location, but would have to delete all the .system snapshots.

 

[ZiggyGT]

I think it is describing my problem. Thanks! I will give it a try, I was a little careless when I set up the datasets so this is likely the problem.

 

[ZiggyGT]

I found out a lot about the motherboard after receiving it. I hate when the eBay seller does not post enough photos of the inside of a chassis. You can look up the data sheets but there is always some mystery. The X9DR3-F MB only Has 6 SATA ports and only one Min-SATA SCU port. The 825-7 chassis has 8 removable drive bays and 2 fixed drive bays. So Supermicro left 2 slots unconnected. I was able to snake a mini-SATA cable in to connect 4 of the drives. I connected the previously unused drive slots to the MB. That leaves a SATA port for the boot drive and perhaps a mirror or another drive. I had issue with the Bios complaining about something plugged into the previously unused/unconnected drive bays. This was resolved using the IDE bios setting instead of ACHI or RAID.

So to confirm my initial question, the plug and pray worked, The on board Mini-SAS does work with TRUENAS. Because only one of the mini-SAS port is actually present on the MB I cannot use it to connect to the 12 drive chassis with the embedded port expander. That is a bummer but I can add a raid card to connect that. That will burn a slot. So, the 3 PCIe slots are destined to be Raid, Dual 10Gb Mellanox X-3 and ... I am trying to stay with a single CPU.
Other changes I am making based on this thread:

• Create Backup system (right now to USB 3.0 drives, Need some help with this. The replication task is complete, but I cannot seem to create a SMB share it to check the contents.
• Add 4th 6TB drive to my Freenas setup so I can move from Z1 to Z2. Upgrade it to Truenas as now I have a backup copy.

 

[ZiggyGT]

Checked out your profile,

Quanta LB6M
Quanta LB4M

First switch I bought was a Quanta LB4M, It had 2 SPF+ ports so I could integrate the 10Gb stuff with my 1Gb stuff. It was so loud I could not stand it. Those tiny fans built into the supply were impossible to silence. I built a closet under the stairs in my basement to get it away from me. When I saw the Aruba switch, I sold the Quanta. The Aruba 2500 is very quiet and has 4 SFP+ ports and a nice web interface. 2 of those must be a challenge.

 

[artlessknave]

2 of those must be a challenge

naw, I have them racked in another room. I dont use any of the managed features, so a web interface is pointless. they give me wire speeds and basic switching. I haven't found anything to replace them that isn't many hundreds of dollars, much less ports, or also louder

Create Backup system (right now to USB 3.0 drives, Need some help with this. The replication task is complete, but I cannot seem to create a SMB share it to check the contents.

truenas and USB 3.0 is unreliable at best. there is no support or help for it.

That is a bummer but I can add a raid card to connect that. That will burn a slot.

you can probably put an expander in a PCIe slot with no CPU. it just needs power and a connection to a SAS, which ought to work with the onboard SAS.

RES2SV240, for example, can be powered by a PCIe slot OR a molex right from the PSU. you would be able to run 5 full sets of breakout cables or SFF's to backplanes from it.

 

[ZiggyGT]

naw, I have them racked in another room. I dont use any of the managed features, so a web interface is pointless. they give me wire speeds and basic switching. I haven't found anything to replace them that isn't many hundreds of dollars, much less ports, or also louder

truenas and USB 3.0 is unreliable at best. there is no support or help for it.

you can probably put an expander in a PCIe slot with no CPU. it just needs power and a connection to a SAS, which ought to work with the onboard SAS.

RES2SV240, for example, can be powered by a PCIe slot OR a molex right from the PSU. you would be able to run 5 full sets of breakout cables or SFF's to backplanes from it.

The Aruba 2500 is a lot quieter and is reasonably priced.
I thought "USB 3.0 is unreliable" is more dispersion on USB than about Truenas support. If I connect the USB to a Windows machine or directly to Truenas is there a difference in reliability? The 6TB drive has not arrived yet so I have not rebuilt my backup machine yet. I used the USB as a temporary backup until that is running again.

My second Chassis has an Expander in the 12 sled chassis. That RES2V240 is impressive and might save me a PCIe slot ( I assume the unused 2nd CPU PCIe slots are powered). The prices are all over the place but some are within reason. it is half height so will fit well into the build. I am not sure the motherboard Mini-SAS can support the Expander. I can test that with the expander in the backplane. I see if it works. I see so many negatives about port replicator and I do not understand the difference between an Expander and a port replicator.

 

[artlessknave]

USB than about Truenas support

USB drives are not great to begin with and USB is particularly unreliable with ZFS so TrueNAS doesn't bother to support it. the hub topology of USB, where devices are obfuscated from the OS by responding in a round-robin like hub-and-spoke topology means that USB drives are NOT available consistently, the way ZFS needs drives to be. ZFS needs to be able to reach all drives in a pool simultaneously, and USB makes this impossible (unless you have each drive on a root USB port and nothing else, which is unlikely)
the behaviour of USB drives will have similarities to how SMR mangles things.

port replicator

a port replicator is different, that's for SATA, and they are always crap, built with the cheapest parts and cheapest methods. they basically do to SATA the same thing as USB does - round robin hub-and-spoke sporadic drive availability.
a SAS expander is an entirely different beast. any SAS compliant controller will work with a SAS expander. the intel SAS in that board appears to be fully compliant. you can chain SAS expanders up to like 256-1024 or so devices. they are conceptually switches for SAS, giving you network switch like functionality.

My second Chassis

what, exactly, is your second chassis? can you give the part number, a pic, or the product page? "825-7" is not enough to look it up/

 

[ZiggyGT]

USB drives are not great to begin with and USB is particularly unreliable with ZFS so TrueNAS doesn't bother to support it. the hub topology of USB, where devices are obfuscated from the OS by responding in a round-robin like hub-and-spoke topology means that USB drives are NOT available consistently, the way ZFS needs drives to be. ZFS needs to be able to reach all drives in a pool simultaneously, and USB makes this impossible (unless you have each drive on a root USB port and nothing else, which is unlikely)
the behaviour of USB drives will have similarities to how SMR mangles things.

I only planned to use a pool with a single USB 3.0 drive for a backup. the concerns seems to be using an array of USB drives. I think I might be OK. But the ZFS concerns would be eliminated if I attach the USB to a client and backup there. I can also buy a single big internal drive and use that as an offline backup.

a port replicator is different, that's for SATA, and they are always crap, built with the cheapest parts and cheapest methods. they basically do to SATA the same thing as USB does - round robin hub-and-spoke sporadic drive availability.
a SAS expander is an entirely different beast. any SAS compliant controller will work with a SAS expander. the intel SAS in that board appears to be fully compliant. you can chain SAS expanders up to like 256-1024 or so devices. they are conceptually switches for SAS, giving you network switch like functionality.

Thanks for the explanation. I was nervous about the Expander. I see there is no reason to be.

what, exactly, is your second chassis? can you give the part number, a pic, or the product page? "825-7" is not enough to look it up/

Here is my lab description

 

[artlessknave]

ahh, so you have 2x 2Us. that makes more sense now. it wasn't making any sense as just 1 2U.
internal SAS cables are NOT designed for external use.

that leaves a few general paths (in order that I would recommend):

1. 1 external SAS card and just use the SATA/SAS ports for the first chassis. + PCI SAS adapter for the 2nd chassis.
2. expander that has both internal and external SAS ports (low profile might be difficult) + PCI SAS adapter for the 2nd chassis.
3. get a RES2V240 + PCI SAS adapters for both chassis.

since you are abandoning 1/2 the PCIe slots, you should have plenty of slots for expanders or SAS adapters.

Show : expander examples

low profile, but SAS3, so expensive. you would only be getting SAS1 out of it, but you would be getting full speeds possible. you would need 8087 > 8043 cables. the art of server hardware is usually updated to the newest firmware and fully tested.
Adaptec AEC-82885T 36-Port SAS-3 12Gbps Expander firmware B059
oh. found a cheaper one. expanders are less likely to be fakes - low volume. (I'm tempted to get one of these actually, though I dont really have a use for it...)
Adaptec 2283400-R AEC-82885T/LENOVO 36Port 12Gb/s SAS Expander Card 82885T
intel:
Intel RES3FV288 RAID Expander 36 Port 12Gb/s SAS PCI Express W/Full Height BRKT
high profile
HP 24-Bay 3GB SAS Expander Card 487738-001 468405-002

Show : PCI SAS Adapters

there are much cheaper ones but these are an example.

aom-sas3-16i16e for sale | eBay

Find great deals on eBay for aom-sas3-16i16e. Shop with confidence.

www.ebay.ca

as these are basically just cables soldered to a PCB, cheap ones are usually OK.
Dual Ports Mini SAS Internal SFF-8087 to External HD SFF-8088 PCI SAS Adapter

 

[ZiggyGT]

ahh, so you have 2x 2Us. that makes more sense now. it wasn't making any sense as just 1 2U.
internal SAS cables are NOT designed for external use.

that leaves a few general paths (in order that I would recommend):

1. 1 external SAS card and just use the SATA/SAS ports for the first chassis. + PCI SAS adapter for the 2nd chassis.
2. expander that has both internal and external SAS ports (low profile might be difficult) + PCI SAS adapter for the 2nd chassis.
3. get a RES2V240 + PCI SAS adapters for both chassis.

since you are abandoning 1/2 the PCIe slots, you should have plenty of slots for expanders or SAS adapters.

Show : expander examples

low profile, but SAS3, so expensive. you would only be getting SAS1 out of it, but you would be getting full speeds possible. you would need 8087 > 8043 cables. the art of server hardware is usually updated to the newest firmware and fully tested.
Adaptec AEC-82885T 36-Port SAS-3 12Gbps Expander firmware B059
oh. found a cheaper one. expanders are less likely to be fakes - low volume. (I'm tempted to get one of these actually, though I dont really have a use for it...)
Adaptec 2283400-R AEC-82885T/LENOVO 36Port 12Gb/s SAS Expander Card 82885T
intel:
Intel RES3FV288 RAID Expander 36 Port 12Gb/s SAS PCI Express W/Full Height BRKT
high profile
HP 24-Bay 3GB SAS Expander Card 487738-001 468405-002

Show : PCI SAS Adapters

there are much cheaper ones but these are an example.

aom-sas3-16i16e for sale | eBay

Find great deals on eBay for aom-sas3-16i16e. Shop with confidence.

www.ebay.ca

as these are basically just cables soldered to a PCB, cheap ones are usually OK.
Dual Ports Mini SAS Internal SFF-8087 to External HD SFF-8088 PCI SAS Adapter

I am a rookie, I did not expand the spoilers until after I went through looking at a bunch of options, then I saw what you posted. Again, thanks for the education. I did not know these kind of cables existed and how there are used. The bulkheads make things a lot more robust. If I did not already have the raid card, (I have 2, I bought those for a 16 drive earlier setup before I bought the 12 sled chassis with the Expander), the Intel RES25V240 would be very attractive. I will not be expanding beyond this many drives so the extra capability of the Intel expander is not needed. I have 4 extra SATA ports. The 12 Sled chassis is basically a JBOD so I could add 4 SSD drives in that chassis someday, so the double bulkhead is attractive..

 

[artlessknave]

raid card

RAID card does not equal HBA. when using an HBA, refer to it as an HBA, especially here where RAID + ZFS = bad.

 

[artlessknave]

I will not be expanding beyond this many drives

yes. we all say that. then get racks of chassis.

 

[ZiggyGT]

I have been known to succumb to that drug. Here is a photo of my son trying to install a boot loader into my computer rack in 1983. Wish I still had that relay rack.

 

[ZiggyGT]

Due to some limitations of the X9DR3-F, I acquired a X9DRH-7F. Happy to report that I was able to test a X9DRH-7F connected to a BPN-SAS2-826EL1 12 x 3.5" backplane and Truenas was able to see all the drives properly. It has 10 on board SATA ports so I can connect the boot drive on one of the fast SATA ports and all the other drives in the chassis to the MB SATA ports. I can cable both the MD mini-SAS ports to the external bulkhead. One for the 12 drive expander chassis and the other to a SSD 4 drive testbed. I'll test the X9DR3-F next for grins but that will replace my ailing S5520 MB.

 

[ZiggyGT]

RESOLVED: The X9DR3-F mother board in my 8 drive chassis does support an HBA mode.
I was able to connect the SM Expander backplane to the on board mini-SAS. The version MB I have has only one mini-SAS port, 4 SATA and 2 SATAII. I wanted more expansion so I looked for a MB with two onboard min-SAS ports.

I got a deal, $54 on a X9DRH-7F that has an on board raid so I decided that it should be the basis of my main server. I saw a lot of posts about reflashing it to HBA., but I found in a post here that it had an HBA mode. I moved my pools from the MB SATA connectors to the Epander backplane successfully. The MB has 10 SATA ports on the MB. That is enough to support the boot drive and all the drives in the chassis. I plumbed both the mini-SAS ports external. I was thinking this would be my main server. It is so damn noisy, I needed to find a solution.

I happened to go into a surplus shop nearby and walked out with a great deal on a second 12 drive expander chassis with a X9DRD-7FLN4 that is very quiet. So after all the work on the 8 drive chassis I decided to make the new 12 drive chassis my main server. Connecting one Expander backplane to an on Board mini-SAS and the second 12 drive chassis to the other on board min-SAS. The 8 drive chassis with be for backups.

So I found that all on the SM motherboards I have with and without raid can support HBA and are able to drive the Expander motherboards. I hope this info can help others.

 

[Ericloewe]

X9DR3-F

That uses the Intel PCH SAS option, which was somewhat popular in the Xeon E5 v1/v2 generation and disappeared in the E5 v3/v4 generation. It's similar to PCH SATA.

X9DRH-7F

That uses an LSI SAS2208 - of course, it's a SAS controller, but that's obvious. The problem is that it's typically only compatible with RAID firmware, which is a no-go. There may be a way to get IT firmware onto the thing, but I'll let someone with specific experience provide more information.

X9DRD-7FLN4

That one uses an LSI SAS2308, which is one of the go-to controllers as it supports IR (which is okay) and IT mode firmware (preferred). See for instance the once-ubiquitous X10SL7-F.

 

[ZiggyGT]

X9DRH-7F
That uses an LSI SAS2208 - of course, it's a SAS controller, but that's obvious. The problem is that it's typically only compatible with RAID firmware, which is a no-go. There may be a way to get IT firmware onto the thing, but I'll let someone with specific experience provide more information.

I read the description on how to flash the SAS2208 to the SAS2308 firmware. That looked doable but unsupported. There was a comment about an option in the SAS2208 firmware to select JBOD mode for the drives. I did that and below is what bootup looks like. I was able to move drives from the SAS2208 to the onboard SATA and vice versa with no issue. Do you think there is still an issue using the JBOD mode of the SAS2208 firmware? This is not the same as the technique to create dummy drives that apparently does have issues. If you compare what my bootup looks like and the after screen in the SAS2208/SAS2308 flash post https://www.truenas.com/community/t...log-zil-with-zfs-on-freenas.13633/post-527596 They look the same to me. Do you think there is a difference? I will use this in my backup server so I am really looking forward to some feedback. If I do not select the JBOD option I get the RAID config screens so I do not think I happen to have one that was modded.