sm X9 4u new build + 4u disk shelf + 2u disk shelf (48+ disks) ?s

[SMnasMAN]

I have been doing alot of reading and smalls testing of FreeNas the past few weeks (including the EXCELLENT intro pdf PP doc, i have never seen such a good, detailed intro into a community before!), im hoping for some recommendations or guidance on my new setup as im slowly buying parts now.

Load-wise, What i want from this FN build (plain FN only will be running on this build, no VMs/jails nor FN plugins):
+ hosting of ESX datastores ( for my 2x local 10gbit esxi "lab hosts", nothing production)
+ SMB/CIFS file shares for my windows work PCs (i work from home, PCs have 1gbit Ethernet)
+ backup space for larger offsite files, that will be sent to this freenas
+ maybe will have my home DVR recording to this freenas (via NFS or iSCSI)- maybe.

My disks and enclosures will be slowly moved from my current setup (a mess of Raid cards + windows 08r2 server linking to supermicro enclosures- a 4u x24bay , a 2nd 4u x24bay and a 2u x 12bay - it has been surprisingly fast/reliable for 4+ years now though, important data is backup'd to B2).

(all equipment will be on a 42u rack)
The freenas host will run on:

• a 4u 24bay ( direct attached backplane, 846-TQ)
• X9DR3-LN4F+ , 192gb ECC ram, 2x e5-2602v2 cpus.
• All HDDs will be 2tb / 3tb / 4tb HGST , mostly enterprise sata/sas disks (i already have ~48x + about 12x more sitting on shelf)

My main question is in regards to which Raidcards / Expanders should i be considering and how should i connect/layout the external 4u and 2u "disk shelf's" (which will physically be racked atop each other, all in a 42u rack) - or if i should be breaking this out into 2 or 3x independent freenas systems (im trying to avoid that to save on power/hw costs, but am open to it)

(im not necessarily looking to go the least expensive route, but rather the better, more tested route).

Im currently considering on the main 4u host (sys running FN):
(8x of the 24x bays will be connected to the X9 MB via the intel C606 chipset, like so:

• Local bay 1- my FN SSD boot
• Local bay 2,3- mirrored s3700 SSDs as ZIL (may change this to a p3700 nvme pcie)
• Local bay 4- for L2arc (on some ssd)
• Local bays 5,6,7,8- xTB HGST disks

for Local bays 9-24:
LSI SAS 9207-8i -> intel res2sv240 expander (and xTB HGST disks)

To uplink this host to the "Disk Shelfs", a LSI SAS 9207-8e:

On "disk shelf #1" ( 4u supermicro 24bay)
Disk Shelf #1 bays 1-24- xTB HGST disks
from host SFF-8088 #1 of the 9207-8e up to either a intel res2sv240 (OR buying a SM BPN-SAS2-846EL1 expander backplane instead of the res2sv240)

On "disk shelf #2" ( 2u supermicro 12bay)
Disk Shelf #2 bays 1-12- xTB HGST disks
from host , SFF-8088 #2 of the 9207-8e up to its BPN-SAS2-826EL1 expander backplane

2 main questions: (or please, address other elements too as i have not ordered the "disk infrastructure" yet):

1- Should i be looking at SAS 3000 gen cards (sas3) for performance ? (ie a lsi 9300-8i vs the 9207-8i im going with?) Even though all my Expanders will be SAS2 (or should i consider spending $ for SAS3 expanders also?)

2- Im guessing i should keep vDEVs and pools isolated to a single chasis/disk shelf? ie i should NEVER have a 2x VDEV Pool consisting of vdev #1 with disks from "the 2u disk shelf #1" and vdev #2 with disks from "the 4u disk shelf #2" , correct?
(ie what were to happen if *ONLY* "disk shelf #2" were to loose power, ie its PSU dies , but all else is running/fine)

(note, i do have alot of time for this, so i will be slowly adding drives to this system on a test bench and testing/benchmarking many scenarios for a few weeks before committing, for now im really just trying to get the hardware and concepts right, to avoid wasting $ on the wrong parts/setup). I will post my results/journey here, or in a new thread.

Thank you.

 

[Ender117]

I don't feel it's worth it to go SAS3 unless you have a bunch of SSDs. While in theory 24 HDDs is enough to saturate a 4*6Gbit link, it does not happen often. Besides your network is more likely to be the bottleneck.

With 60 disks that would pull quite some power. Have you consider to switch to higher capacity drives?

 

[SMnasMAN]

thanks, this is correct in that my current ~ 65 drives each are spec'd to pull:
~ 7w IDLE
~ 11.5w with r/w load
(most are- hgst hus724040ale641 )
however the sum of my UPSs seem to indicate *roughly* ~ 350w for the ~65 drives @ idle

I do plan on moving to larger TB drives to save on power, however that move wont start for some time, so for now im trying to save power where i can (on things other than the HDDs).

Do you think i should be looking at a single FN system like i outlined above, or should i lean towards 3x independent FN systems (ie 3x 4u 24 bays). (or can a single FN system linked to multiple disk shelves be pretty reliable? - im mainly asking if FN is or is not built for multiple disk shelves from a single system, as some enterprise SANs are)

thanks

EDIT: i should add, even if i hypothetically moved to larger TB drives *today* (ie 8 or 10tb) i still want this system built to have the 60+ drive bays ive spec'd out above (so that even if only using 24x bays with 10tb drives, today, tomorrow i can easily add more drives later without buying anything beyond the drives themselves)

 

[Ender117]

thanks, this is correct in that my current ~ 65 drives each are spec'd to pull:
~ 7w IDLE
~ 11.5w with r/w load
(most are- hgst hus724040ale641 )
however the sum of my UPSs seem to indicate *roughly* ~ 350w for the ~65 drives @ idle

You sure? My system of a R620 + 24bay shelf alone can pull ~300w idle, that makes a very efficient system.

Do you think i should be looking at a single FN system like i outlined above, or should i lean towards 3x independent FN systems (ie 3x 4u 24 bays). (or can a single FN system linked to multiple disk shelves be pretty reliable? - im mainly asking if FN is or is not built for multiple disk shelves from a single system, as some enterprise SANs are)

I am not a FreeNAS developer but I would be surprised if it have problem /w 3 shelves. I am sure someone more experienced than me can chime in here.

EDIT: i should add, even if i hypothetically moved to larger TB drives *today* (ie 8 or 10tb) i still want this system built to have the 60+ drive bays I've spec'd out above (so that even if only using 24x bays with 10tb drives, today, tomorrow i can easily add more drives later without buying anything beyond the drives themselves)

Sure, just keep the shelves and only power on when you need them. But at that time even larger drives might have come out.

 

[SMnasMAN]

> You sure? My system of a R620 + 24bay shelf alone can pull ~300w idle, that makes a very efficient system.
my 350w was just a rough estimate for the drives only - the entire system pulls 550-580w @ 15% cpu "idle load" , but its a sm X8 with 2x 8c xeons, 48gb and 5x HW raid cards (and 4x PSUs total, 2x on the system, and 1x on each of 2 x shelfs).

its exacly 59 drives, but 6x of them are SSDs, so i do think my ~350w is close , in this case. Thanks for your info / reply too.

 

[jgreco]

Do you think i should be looking at a single FN system like i outlined above, or should i lean towards 3x independent FN systems (ie 3x 4u 24 bays). (or can a single FN system linked to multiple disk shelves be pretty reliable? - im mainly asking if FN is or is not built for multiple disk shelves from a single system, as some enterprise SANs are)

That's not how enterprise SAN units work. They're not "built" for multiple disk shelves. It's just a practical concession in the implementation.

If you knock out power to a component disk shelf in an enterprise SAN, it will likely lose its ****. If you are really clever and have enough disk shelves, you may be able to "redundancy" your way out of an actual crash-tastrophe, and the software is designed to help you optimize towards that. If you have a full rack of shelves, this actually becomes pretty easy and you can build for various kinds of outages.

FreeNAS won't help you optimize towards that kind of solution because it doesn't know the topology of your system. However, it is happy to service disks in multiple shelves, and you can design redundancy to tolerate a failure if you wish, or you can create separate pools on each shelf, to help isolate an outage.

If you don't actually care what happens when a shelf loses power (or connectivity or whatever), then all of this is academic and yes you can make really large FreeNAS systems.

 

[Spearfoot]

...direct attached backplane, 846-TQ...

The TQ backplane means you have to connect each individual drive with its own cable; hook up 24 drives and you're going to have a real rat's nest!

I built a system very similar to what you have in mind (see 'my systems' below), except that I used a BPN-SAS-846A backplane driven by 3 x LSI 9210-8i cards. This uses only 6 cables to connect 24 drives.

Just my two cents... :)[/QUOTE]

 

[Chris Moore]

(ie what were to happen if *ONLY* "disk shelf #2" were to loose power, ie its PSU dies , but all else is running/fine)

Been there done that. If you have more drives than redundancy allows go offline at once, the pool becomes inaccessible. Instantly. In my case it was rapidly followed by a full server crash because I had swap in use that was located on the missing drives. In my system, it was a bad drive controller. I put a new controller in and brought the system back online and everything was right back the way it was just before the crash.
If you really want something that is "crash proof" you are going to need to put a lot more work into it or buy a TrueNAS from iXsystems with full fault tolerance. Their prices are not out of line with what you should expect to pay for that type of solution.

(note, i do have a lot of time for this, so i will be slowly adding drives to this system on a test bench and testing/benchmarking many scenarios for a few weeks before committing, for now im really just trying to get the hardware and concepts right, to avoid wasting $ on the wrong parts/setup). I will post my results/journey here, or in a new thread.

Not to rain on your parade but many of the answers you are looking for are already known. Just ask, no need to see if the wheel is still round.

 

[jgreco]

The TQ backplane means you have to connect each individual drive with its own cable; hook up 24 drives and you're going to have a real rat's nest!

I built a system very similar to what you have in mind (see 'my systems' below), except that I used a BPN-SAS-846A backplane driven by 3 x LSI 9210-8i cards. This uses only 6 cables to connect 24 drives.

Just my two cents... :)

That's also a good point. -TQ chassis are basically a cabling nightmare. You can definitely add an SAS expander to each chassis to make this work, and as long as you don't ever need to go digging through cables to remediate a marginal connection, there are ways to do this. The -A chassis substitute a 4 lane connector for four of the individual SAS connectors, which is better but still leaves you needing an SAS expander in the shelves (probably). The -E chassis are really the way to go as they incorporate the SAS expander into the backplane, so all you really do is run a single cable from the backplane to one of the slot backplates, where you install an external SAS connector, and bam, call it a day. Everything else has significantly higher cabling and add-on card requirements.

 

[SMnasMAN]

thanks for the replies!
jgreco/spearfoot- im aware of the differences in SM BPs (ie why i said "direct attached backplane, 846-TQ") , however this info is good for others viewing this post later. in my head ive been trying to justify the $ to sell my 2x direct attached BPs and go expander BPs strictly for cabling reasons. however i do also have an extra res2sv240 also, so ill prob just do the best i can with the cables and roll with the TQ and A i have for now (vs getting an el1 or el2), (i do have the SM right angle staggered length 8087-> 4x sata fanout so that helps a bit).

chrismorre: awesome info, exactly what i was hoping to get some insight into! (namely if FN is *usually* ok/data wise if a disk shelf suddenly loses power. ofcouse assuming that all vdevs of said vpool are intra-disk shelf ONLY. I will plan around this scenario in that my zpools will only consist of vdevs within a single disk shelf (ie no inter-diskshelf vdevs).

(btw, i will be doing some "giving back vs taking" to this forum/threads as i start to build the system in terms of posting benchmarks of various configs/hw and experiences , and picts. I have alot of time to test and play with this system so i plan to do that and to keep alot of my info on these fourms) , should be later this week as almost all parts are in route now. Ill post full exact specs and $$ spent, as im pretty happy with the prices/deals i was able to find (ebay)! tks

EDIT: also this is a great and relevant thread:
https://forums.freenas.org/index.php?threads/supermicro-x10srl-f-3-846-chassis-72-disks.30431/

 

[SMnasMAN]

I hope im doing something wrong as im pretty new to freenas, but not linux, and im not getting very far with FN.

So for the past week i have stress tested every part (and simultaneously) of this system under ubuntu17 , cent 6.7 and 2012r2 (just to verify my hardware is good b4 i start experimenting with freenas). I have alot of experience building and testing SM servers in particular.

so my HW is good (same as listed above in OP, but below is exact im using now, i specifically bought this hardware for FN after reading up):

• a 4u 24bay ( direct attached backplane, 846-TQ)
• X9DR3-LN4F+ , 128gb ECC ram (on SM QVL) , 2x e5-2602v2 cpus.
• 9207-8i (no expander in use yet , just direct attach to 8x bays)
• using 6x of the onboad intel c606 based sata ports
• 3tb HGST drives ("hua" type), i tested these extensively over past few days (and they were pulls from my own working server, here)

I install freenas onto a spare 300gb sata disk. boot up, and the first thing i wanted to do was to directly quick bench a few SINGLE drives in freenas (i havent done anything else in freenas, all disks are untouched, ie install FN, boot FN -> go to web terminal/shell). and type via FN terminal:

dd if=/dev/ada3 of=testfile bs=1M count=2500
(this disk is connected via my X9's intel c606's sata ports)

a few secs after, the system reboots itself. I figure i'll reinstall FN to a usb stick instead (as i saw that message at install about using flash).

FN comes up, same thing as above, but this time to a different disk (one connected via my 9207-8i HBA). Same thing, reboots.

I know logs are not persistant, so i dont even know where to look or start to debug this. my hardware / ram is good (im not saying its compatible though, just known not defective).

for now ill try setting up remote syslog and triggering the crash (there has to be a way to get logs out of FN? right?)

thanks


EDIT:
well i was able to get something , i created a volume successfully via web gui and then went to add an IP on a different nic (which worked but locked me out of web gui as that IP took over as def. gw route for some reason , so i went to console and when i choose reset all config back to default, i was able to frantically grab my phone and snap this image b4 the system rebooted again): Also watchdog is disabled in BIOS and on the motherboad jumper (set to 2-3).

 

[Spearfoot]

I hope im doing something wrong as im pretty new to freenas, but not linux, and im not getting very far with FN.

So for the past week i have stress tested every part (and simultaneously) of this system under ubuntu17 , cent 6.7 and 2012r2 (just to verify my hardware is good b4 i start experimenting with freenas). I have a lot of experience building and testing SM servers in particular.

so my HW is good (same as listed above in OP, but below is exact im using now, i specifically bought this hardware for FN after reading up):

• a 4u 24bay ( direct attached backplane, 846-TQ)
• X9DR3-LN4F+ , 128gb ECC ram (on SM QVL) , 2x e5-2602v2 cpus.
• 9207-8i (no expander in use yet , just direct attach to 8x bays)
• using 6x of the onboad intel c606 based sata ports
• 3tb HGST drives ("hua" type), i tested these extensively over past few days (and they were pulls from my own working server, here)

I install freenas onto a spare 300gb sata disk. boot up, and the first thing i wanted to do was to directly quick bench a few SINGLE drives in freenas (i havent done anything else in freenas, all disks are untouched, ie install FN, boot FN -> go to web terminal/shell). and type via FN terminal:

dd if=/dev/ada3 of=testfile bs=1M count=2500
(this disk is connected via my X9's intel c606's sata ports)

a few secs after, the system reboots itself. I figure i'll reinstall FN to a usb stick instead (as i saw that message at install about using flash).

FN comes up, same thing as above, but this time to a different disk (one connected via my 9207-8i HBA). Same thing, reboots.

I know logs are not persistant, so i don't even know where to look or start to debug this. my hardware / ram is good (im not saying its compatible though, just known not defective).

for now ill try setting up remote syslog and triggering the crash (there has to be a way to get logs out of FN? right?)

thanks


EDIT:
well i was able to get something , i created a volume successfully via web gui and then went to add an IP on a different nic (which worked but locked me out of web gui as that IP took over as def. gw route for some reason , so i went to console and when i choose reset all config back to default, i was able to frantically grab my phone and snap this image b4 the system rebooted again): Also watchdog is disabled in BIOS and on the motherboad jumper (set to 2-3).




View attachment 27003 View attachment 27004

Yikes!

Your motherboard is very similar the one in my main system (an X9DRi-LN4F+); they even share the same User's Manual. So I doubt there is any deal-breaking hardware incompatibility causing you grief.

Have you installed the latest BIOS on the board? I'd do that, which involves clearing the CMOS. That will start you off with a clean slate. There may be some BIOS settings that are biting you. I'd start off with the vanilla default settings and proceed from there.

I have a video of my BIOS settings. It's over 250MB in size(!), but I could try to send it to you if you think it'd help you out to have something to compare to your settings.

Good luck!

EDIT: What version of FreeNAS are you installing? Might be better to use 11.1-U6, which is the most stable available at the moment.

 

[SMnasMAN]

hi, thanks for your reply (im using 11.2 iso relnotes show dec 5 2018 date , and verified the hash of iso b4 burning) ! the first thing i did when i got this board in (about 10 or so days ago) was update the bios to the latest (v 3.3), and followed SMs bios update procedure (flash, then pull AC, reset cmos and power up, then load default optimized). My bios settings are almost those exact default optimized. The only changes were boot order, and turning off PXE on the onboard NIC (i dont use pxe boot ever).

I did try 3 other things after my post:
1- pulled all cards (ie the 9207-8i and my Cenro t520 sfp 10gb card) - and with 2 drives only on the c606 MB sata ports- same result

2- put in only the 9207-8i and connected 2 drives to that (so nothing was on the intel c606 mb sata ports) - same result.
(to refersh: "same result" = crashing during or right after FN os loads, or if it boots/loads, usually 2-10minutes after).

3- Tried 8gb (2x 4gb ECC sticks) of different known good memory (also on this x9 mb's QVL) - same result.

funny you said that about the video, i also made a recording of all the bios menu options, just for future reference (abit different than your video).

I think you maybe on to something with trying a prior FN version, that idea hit me last night and your post reinforces that idea. I will start trying that today. and will update if any progress. Thank you!

 

[SMnasMAN]

some progress! - installing 11.1rc6 (vs current 11.2), so far it has not crashed after about 20minutes of running / tests, however (and maybe its just freebsd and im not used to it), some things are still a bit off:

(iam only using 8gb of ram currently, i will test back with my other sticks of 128g)

(again, i have configured/added nothing in FN, i just install os, boot, turn on SSH and am using DD direct to disks tests for now)

using dd, my 2x disk will show normal ~160 MB/s if my test files are under ~500mb, but any dd tests over that size are showing horrible speeds (disks are direct attached to lsi 9207-8i no expanders):
(also i have extensively tested these same disks with other os tools, they are consistent 160-110 MB/s R or W the entire length of their 3 TB)

root@freenas:~ # dd if=/dev/da0 of=testfile bs=1M count=2500
2500+0 records in
2500+0 records out
2621440000 bytes transferred in 375.981628 secs (6972256 bytes/sec)


root@freenas:~ # dd if=/dev/da0 of=testfile bs=1M count=25
25+0 records in
25+0 records out
26214400 bytes transferred in 0.197543 secs (132702533 bytes/sec)


also when i hit enter (for the dd command), it will take like 10s for the results to show up (even though the dd test might only take 1 second), nothing else seems slow or sluggish though (ie web gui is fast, multi ssh sessions open quickly):

here is some info (and attached is full output from dmesg | grep mps ) -- Any ideas? (i will be testing more and adding back hw / ram and will update -- prob should have broke this out into a seperate forum thread :/ )
thanks

Code:

root@freenas:~ # dmesg | grep mps

mps0: <Avago Technologies (LSI) SAS2308> port 0x7000-0x70ff mem 0xdfa40000-0xdfa4ffff,0xdfa00000-0xdfa3ffff irq 32 at device 0.0 numa-domain 0 on pci5
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps0: SAS Address for SATA device = 6f6a852690b8dc8f
mps0: SAS Address from SATA device = 6f6a852690b8dc8f
mps0: SAS Address for SATA device = dc665c2490b8dbb4
mps0: SAS Address from SATA device = dc665c2490b8dbb4
da0 at mps0 bus 0 scbus2 target 1 lun 0
da1 at mps0 bus 0 scbus2 target 2 lun 0


--------------

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2308_2(D1)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2308_2(D1)   20.00.07.00    14.01.00.06    07.39.02.00     00:05:00:00


+--------------------------------------------------------------------------------+
+                         camcontrol devlist @1544564114                         +
+--------------------------------------------------------------------------------+
<ATA Hitachi HUS72403 A5F0>        at scbus2 target 1 lun 0 (pass0,da0)
<ATA Hitachi HUS72403 A5F0>        at scbus2 target 2 lun 0 (pass1,da1)
<ADATA USB Flash Drive 1100>       at scbus10 target 0 lun 0 (pass2,da2)
debug finished in 0 seconds for camcontrol devlist


+--------------------------------------------------------------------------------+
+                       camcontrol inquiry da0 @1544564114                       +
+--------------------------------------------------------------------------------+
pass0: <ATA Hitachi HUS72403 A5F0> Fixed Direct Access SPC-4 SCSI device
pass0: Serial Number P8GV151P
pass0: 600.000MB/s transfers, Command Queueing Enabled
debug finished in 0 seconds for camcontrol inquiry da0


+--------------------------------------------------------------------------------+
+                       camcontrol inquiry da1 @1544564114                       +
+--------------------------------------------------------------------------------+
pass1: <ATA Hitachi HUS72403 A5F0> Fixed Direct Access SPC-4 SCSI device
pass1: Serial Number P8H1D9ZR
pass1: 600.000MB/s transfers, Command Queueing Enabled
debug finished in 0 seconds for camcontrol inquiry da1

 

[SMnasMAN]

(see above post)
damn, so on 11.1 rc6 its def. acting up a bit, but not crashing anymore like on 11.2 .
so now ive rebooted a few times with the MB sata ports connected to 2x disks (+ the 9207-8i still connected to 2x disks).

im seeing the same horrible disk speeds to disks on the mb sata ports.

Code:

dd if=/dev/ada0 of=testfile bs=1M count=2500
the2500+0 records in
2500+0 records out
2621440000 bytes transferred in 184.708773 secs (14192287 bytes/sec)

root@freenas:~ # dd if=/dev/da0 of=testfile bs=1M count=2500
2500+0 records in
2500+0 records out
2621440000 bytes transferred in 321.553957 secs (8152411 bytes/sec)

then if i run 2x DD s (one to mb sata port diks, and one to disk on the 9207-8i), the system does start responding slowly via ssh (but load never goes about 0.06).

any ideas? or do i need to keep trying older FN versions maybe?

this really is not inspiring confidence in my new data platform with known good hardware, hw picked specifically for FN - (but this is why i/we test everything before committing).

thanks

 

[Spearfoot]

@SMnasMAN : How, exactly, did you burn-in your hardware?

Did you run memtest86 against all of your memory? For how many passes? You have the same amount of RAM I do, and it took several days for me to test mine thoroughly.

Did you run the mersienne prime or any other CPU torture tests?

Another thing to try... install plain Linux or FreeBSD and see if you still get odd behavior from your basic dd drive tests.

 

[SMnasMAN]

Thanks for the reply, please see answer a bit lower, but First off GOOD NEWS!

still on 11.1 rc6 and no crashes, and i may have an answer to why i was seeing slow dd speeds (i think some of the stress test tools i was using wrote over entire disk, including the gpt partition table, which was causing FN to show the slow speeds to individual disks, see this message from the console as i was creating a volume via FN volume manager!)-


and now this is what im now seeing in FN from a 4x disk stripe , with compression turned off (and encryption off):

root@freenas:/mnt/test2stripe # dd if=/dev/zero of=ddtest bs=8M count=2k
2048+0 records in
2048+0 records out
17179869184 bytes transferred in 29.917005 secs (574250980 bytes/sec)

The answer to all your questions is yes :) (and for many days, for each) - memtest86 pro v8.0 in uefi boot mode, for 51 hours (no errors displayed, nor ipmi event log ecc errors) // p95 in 2012r2 using up 105gb of 128g for about 24 hours // y-cruncher in 2012r2 using all mem and cpus + while running HDD Sentinel Pro's disk surface RW test to 3x HDDs for about 10 hours .

+ this concurrently, for ~39 hours (ubuntu 17 installed to a disk):
stress-ng --vm 8 --vm-bytes 75% --vm-method all --verify -t 10000m -v
+
(fio run concurrently on 3x direct hdds, so 3x instances)
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=te2st --filename=tes2t --bs=4k --iodepth=256 --size=12G --readwrite=randread --time_based --runtime=5000000
+
(below across my CHELSIO T520-CR (2x 10g sfp+) )
iperf3 -c -t 84000 192.168.8.2
and
iperf3 -s -p 666 (and -c from another system)

this is how my system looked for ~39 hours (no issues, nor errors)

more importantly, some good progress (and confidence)! I will keep testing and configing away and will update this thread.

and i will also try 11.2 again at some point, just to see if maybe the GPT table issue was causing thoses quick crashes (i doubt it, but will try). and based on what i find, if nessisary, will file a bug report re: 11.2

here is current setup from that dd test above, with another dd , if interested

Code:

root@freenas:/mnt/test2stripe # zpool status
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da3p2     ONLINE       0     0     0

errors: No known data errors

  pool: test2stripe
state: ONLINE
  scan: none requested
config:

        NAME                                          STATE     READ WRITE CKSUM
        test2stripe                                   ONLINE       0     0     0
          gptid/4a2698f8-fd94-11e8-8ead-00259084f1c8  ONLINE       0     0     0
          gptid/4f051a76-fd94-11e8-8ead-00259084f1c8  ONLINE       0     0     0
          gptid/53a0404e-fd94-11e8-8ead-00259084f1c8  ONLINE       0     0     0
          gptid/58e8fa53-fd94-11e8-8ead-00259084f1c8  ONLINE       0     0     0

errors: No known data errors
root@freenas:/mnt/test2stripe #
root@freenas:/mnt/test2stripe #

root@freenas:/mnt/test2stripe # dd if=/dev/zero of=ddte3st bs=8M count=2k
2048+0 records in
2048+0 records out
17179869184 bytes transferred in 30.534585 secs (562636399 bytes/sec)

 

[SMnasMAN]

Im thinking this has to be a bug with 11.2 stable, as ive been running /stressing the exact same system (16x disks) on 11.1 since this post (a little over 10 days now). for about 6 of those days ive had 24x7 ~7gbit of load (3.5up 3.5d) across the 10gb card to these disks, with all disk LEDs always flickering/illumnated (ie actual disk load, not arc/cache IO), the load is mostly SMB shares, but 1 NFS also. And not one hickup or any problem with 11.1. (im VERY MUCH loving freenas after this past 2 weeks!!)

So im now re-trying a fresh install of 11.2 (which i did try back when i was having issues), and right at first startup (post install) i got another fatal trap just like before (didnt get a chance to catch the number on this one).

Is there a guide or info on what i need to include for my bug report?

thanks

EDIT; just to test, im trying the latest beta of 11.3 right now. will updated (even though id only acutally run a stable when im done all this testing/playing)

EDIT2 - same thing on the dec22nd 11.3 FN, Fatal Trap 12 (see image) - about 5min after boot up (fwiw, all my disks did show up, but all had blank spaces on the drive size column, i was able to import a 4disk pool i had from 11.1 , and this too showed "error" on the disk size, but the pool was imported/browsable) will be filing bug form, and i should have the kdump from this 11.3 stick hopefully.

 

[SMnasMAN]

to update, i filed a bug here
https://redmine.ixsystems.com/issues/66130

(11.1 still rock solid after over 14 days of 24/7 stress testing w ~ 20 drives for now)

 

[SMnasMAN]

to update this a bit more (see the bug report), im having no issues with 11.2 + same v2 cpus (and other v2 cpus) on a x9dri-ln4f+ system (note the "i" vs "3" ). which is a VERY similar board to the x9dr3 (even uses the same SM bios). i think the dr3 uses a c606 chipset vs dri using c602 (i could be wrong as i cant look it up right now).

But on 2x different x9dr3 boards, each entirely independent setups (ie separate ram/psus/case ect , boards purchased from different sources over a year apart) , 11.2 panics at boot, or right after. So its something with 11.2 and the x9dr3 , as best i can tell. (both my x9dr3 have 0 issues running and actively utilizing other OSes or 11.1 for over a year). so for now i'll be using 11.1 (or a different board with 11.2). ixsys bug support was impressive, for a free os (i get that we are kind of their beta testers, but still! )