BUILD Large NAS

[DWELLER_ZA]

Awesome community, awesome software! Thank you all in advance


Background

I started off researching custom NAS solutions around 2 years ago, and decided to try FreeNAS. I haven't looked back, and it's high time for me to invest in a large storage capacity server


Intended Use

• Home use mostly (We run a company from home)
• Backup target for a couple of machines
• Long term file storage (with multiple proper off site backup destinations)
• Massive Media library

So the primary goal here is to use ZFS, RAIDZ and max on storage capacity.


Proposed Build

Platform: http://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR36N.cfm
CPU: Haswell Intel E5-2620V3
Memory: 4 x Samsung 16GB DDR4-2133 2Rx4 LP ECC REG RoHs (64GB Total)
SAS Controller(s): LSI 3108 AOC
Hard Drives: ?? 36 x Something, see questions
OS Hard Drive: 32GB SATA DOM
Networking: I'm going the 10GbaseT route here as I'm changing my core network to 10GbaseT


Questions

Do I need to swap out http://www.supermicro.com/products/accessories/addon/AOC-S3108L-H8iR.cfm for this one http://www.supermicro.com/products/accessories/addon/AOC-S3008L-L8i.cfm ?

Which drives do I go for here? I was thinking WD 6TB Red drives, but I can get the Seagate 8TB archive drives at the time price.

What would be the optimal configuration for a good balance in available size vs redundancy? (But leaning towards available size)

Am I speccing enough RAM?



Thank you

 

[depasseg]

I think the 3008 would be fine, performance wise, and I'm guessing it would be a tad cheaper if you wanted to save some cash.

Go with the 6TB. IIRC the Seagate 8TB archive drives are SMR, which aren't a good fit for a NAS (unless you plan to run something like Amazon Glacier (meaning you are only writing for long term archiving)).

As for RAM, it's on the low side, but I guess it depends on how much usable storage you plan to configure. What is the purpose of the box? Are all drives used for primary data, or do you plan to create a backup pool using some of the drives? More RAM is better. Given everything else you've spec'd, I'd go with 128GB.

And to take advantage of 10GbE, you will need to figure out your vdev configurations (speed vs capacity) and if planning to use NFS, consider adding a SLOG device.

 

[DWELLER_ZA]

Hi depasseg,

Thank you for your suggestions.

I will double the RAM. Is the rule of thumb still 1GB of RAM for each TB of storage even if you are talking about such a large storage server?

The primary purpose of the NAS is media storage.

I'm going for max storage on the primary pool. No secondary backup pool requirement at this stage, seeing as we have offsite backups in place.

Access methods include FTP, NFS and CIFS

What would you suggest as a config for the vdevs?

 

[depasseg]

I will double the RAM. Is the rule of thumb still 1GB of RAM for each TB of storage even if you are talking about such a large storage server?

No, it kinda trails off. 64 is better, and if you have issues, be prepared to double again. Do you plan to run any jails?

What would you suggest as a config for the vdevs?

What performance do you need? With 36 drives, you could do 4 x 9 drive RAID-Z2. That gives you a ton of sequential throughput, but the IOPS of 4 drives. And it gives you a decent amount of drive protection.

Sync writes will be very slow unless you add a SLOG, so consider your NFS workload. FTP and CIFS won't be an issue in that regard. CIFS will be CPU Freq limited.

 

[DWELLER_ZA]

Do you plan to run any jails?

Nope

Sync writes will be very slow unless you add a SLOG

SLOGs are new to me. Could you recommend one to me?

 

[Dice]

If you're not intending to run virtualization nor a bunch of jails, with limited amounts of users accessing the machine u'd probably be fine without getting a SLOG.

Cheers /

 

[DWELLER_ZA]

Thank you for the article Dice, the NAS will always only have a couple of users using it. Will investigate the SLOG in any event

 

[danb35]

What would you suggest as a config for the vdevs?

I'd lean toward six-disk RAIDZ2 vdevs, and fill up the server as you need them, buying drives six at a time. Unless you really need all that capacity right away, it's likely to be more economical to add drives as you go.

BTW, nice hardware choice--mine is similar, but a generation or so older. I hope I don't run out of drive bays any time soon!

 

[DWELLER_ZA]

Thank you! :)

I'm purchasing either all 36 drives from the word go, or 18 larger capacity ones. (Haven't decided yet)

 

[Dice]

the NAS will always only have a couple of users using it. Will investigate the SLOG in any event

In this case, I'm quite convinced there is no point in getting a SLOG device.
Plenty of RAM, no VM's, very few users.
Meanwhile it might be "cool" to add a SLOG for the sake of spending more cash on a killer system, it is not necessarily only a <good thing>.
You are introducing another link in the chain of products that must work to keep your pool safe, without no performance benefits in your use case.
I'd advice against any sort of SLOG in the described scenario.

18 larger capacity ones.

Stay away from Shingled drives (such as seagate 8TB's archive drives - they do not do well on ZFS due to their design).

Cheers /

 

[DWELLER_ZA]

Hi Gents,

I've received my server and have one issue, how do I handle S.M.A.R.T. within FreeNAS?

I put my LSI 3108 into JBOD and setup my 4 x 9 vdevs within FreeNAS and noticed the following error when starting the S.M.A.R.T. service:

notifier: Starting smartd.
root: /usr/local/etc/rc.d/smartd: WARNING: failed to start smartd
notifier: /usr/local/etc/rc.d/smartd: WARNING: failed to start smartd

I've read that this is due to LSI not passing the S.M.A.R.T. info through to FreeNAS.

Any ideas or advice?

 

[JeroenvdBerg]

Hi Gents,

I've received my server and have one issue, how do I handle S.M.A.R.T. within FreeNAS?

I put my LSI 3108 into JBOD and setup my 4 x 9 vdevs within FreeNAS and noticed the following error when starting the S.M.A.R.T. service:

notifier: Starting smartd.
root: /usr/local/etc/rc.d/smartd: WARNING: failed to start smartd
notifier: /usr/local/etc/rc.d/smartd: WARNING: failed to start smartd

I've read that this is due to LSI not passing the S.M.A.R.T. info through to FreeNAS.

Any ideas or advice?

PICS! I want to see this bad boy :) did you use 6TB or 8TB drives?

As for your issue:

smartctl --smart=on /dev/da0
(and that per drive)

See also: https://webcache.googleusercontent....start-smartd.25563/+&cd=1&hl=nl&ct=clnk&gl=nl

Or:

Finally I found the way to read SMART through these MegaRAID cards. The first thing you'll have to do is to list IDs of all your physical disks:

server:~# megacli -PDlist -a0 | grep '^Device Id:'
Device Id: 0
Device Id: 1
Device Id: 2
Device Id: 3

Then you can add this kind of lines to /etc/smartd.conf (don't forget to comment the DEVICESCAN one):

# LSI MegaRAID
/dev/sda -d sat+megaraid,0 -a -s L/../../3/02
/dev/sda -d sat+megaraid,1 -a -s L/../../3/03
/dev/sda -d sat+megaraid,2 -a -s L/../../3/04
/dev/sda -d sat+megaraid,3 -a -s L/../../3/05

Please note you need a recent version of smartmontools. 5.38 from Debian Lenny won't work, 5.39.1+svn3124 from Squeeze does.

See also:
http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS


Does this work?
smartctl -a -d megaraid,0 /dev/sda
(if your drive is sda)

See also: https://www.smartmontools.org/ticket/208


Or:
https://forums.freenas.org/index.ph...o-start-at-boot-but-manually-starts-ok.12647/

Please post the part of your syslog where it try's to start smartd.

 

[DWELLER_ZA]

Thank you for the post, here are some pics:

It will be moved to our server rack tomorrow.

I'm unable to sort the S.M.A.R.T. issues out from your post.
Prepare for information overload! :)
Let me know if I missed anything or if anybody requires anything else.

[root@NAS] /usr/local/etc# tail /var/log/messages
Aug 28 19:27:27 NAS notifier: /usr/local/etc/rc.d/smartd: WARNING: failed to start smartd
Aug 28 19:33:24 NAS notifier: smartd not running? (check /var/run/smartd.pid).
Aug 28 19:33:24 NAS notifier: Cannot 'restart' smartd. Set smartd_enable to YES in /etc/rc.conf or use 'onerestart' instead of 'restart'.
Aug 28 19:33:29 NAS notifier: smartd not running? (check /var/run/smartd.pid).
Aug 28 19:33:30 NAS notifier: Cannot 'restart' smartd. Set smartd_enable to YES in /etc/rc.conf or use 'onerestart' instead of 'restart'.
Aug 28 19:34:03 NAS notifier: smartd not running? (check /var/run/smartd.pid).
Aug 28 19:34:03 NAS notifier: Cannot 'restart' smartd. Set smartd_enable to YES in /etc/rc.conf or use 'onerestart' instead of 'restart'.
Aug 28 19:34:09 NAS notifier: smartd not running? (check /var/run/smartd.pid).
Aug 28 19:34:09 NAS notifier: Cannot 'restart' smartd. Set smartd_enable to YES in /etc/rc.conf or use 'onerestart' instead of 'restart'.
Aug 28 19:36:45 NAS root: /usr/local/etc/rc.d/smartd: WARNING: failed to start smartd

[root@NAS] /usr/local/etc# more /usr/local/etc/smartd.conf
################################################
# smartd.conf generated by /etc/ix.rc.d/ix-smartd
################################################
/dev/mfisyspd7 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd8 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd9 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd10 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd11 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd12 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd13 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd14 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd15 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd0 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd1 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd2 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd3 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd4 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd5 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd6 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd16 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd17 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd18 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd19 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd20 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd21 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd22 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd23 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd24 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd25 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd26 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd27 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd28 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd29 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd30 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd31 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd32 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd33 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd34 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py
/dev/mfisyspd35 -a -n never -W 0,0,0 -m my@email.com -M exec /usr/local/www/freenasUI/tools/smart_alert.py

[root@NAS] /usr/local/etc# smartctl -a /dev/mfisyspd0 -d scsi
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

Smartctl open device: /dev/mfisyspd0 failed: No such file or directory

[root@NAS] /# smartctl --scan
/dev/ses0 -d scsi # /dev/ses0, SCSI device
/dev/ses1 -d scsi # /dev/ses1, SCSI device
/dev/pass2 -d scsi # /dev/pass2, SCSI device
/dev/pass3 -d scsi # /dev/pass3, SCSI device
/dev/pass4 -d scsi # /dev/pass4, SCSI device
/dev/pass5 -d scsi # /dev/pass5, SCSI device
/dev/pass6 -d scsi # /dev/pass6, SCSI device
/dev/pass7 -d scsi # /dev/pass7, SCSI device
/dev/pass8 -d scsi # /dev/pass8, SCSI device
/dev/pass9 -d scsi # /dev/pass9, SCSI device
/dev/pass10 -d scsi # /dev/pass10, SCSI device
/dev/pass11 -d scsi # /dev/pass11, SCSI device
/dev/pass12 -d scsi # /dev/pass12, SCSI device
/dev/pass13 -d scsi # /dev/pass13, SCSI device
/dev/pass14 -d scsi # /dev/pass14, SCSI device
/dev/pass15 -d scsi # /dev/pass15, SCSI device
/dev/pass16 -d scsi # /dev/pass16, SCSI device
/dev/pass17 -d scsi # /dev/pass17, SCSI device
/dev/pass18 -d scsi # /dev/pass18, SCSI device
/dev/pass19 -d scsi # /dev/pass19, SCSI device
/dev/pass20 -d scsi # /dev/pass20, SCSI device
/dev/pass21 -d scsi # /dev/pass21, SCSI device
/dev/pass22 -d scsi # /dev/pass22, SCSI device
/dev/pass23 -d scsi # /dev/pass23, SCSI device
/dev/pass24 -d scsi # /dev/pass24, SCSI device
/dev/pass25 -d scsi # /dev/pass25, SCSI device
/dev/pass26 -d scsi # /dev/pass26, SCSI device
/dev/pass27 -d scsi # /dev/pass27, SCSI device
/dev/pass28 -d scsi # /dev/pass28, SCSI device
/dev/pass29 -d scsi # /dev/pass29, SCSI device
/dev/pass30 -d scsi # /dev/pass30, SCSI device
/dev/pass31 -d scsi # /dev/pass31, SCSI device
/dev/pass32 -d scsi # /dev/pass32, SCSI device
/dev/pass33 -d scsi # /dev/pass33, SCSI device
/dev/pass34 -d scsi # /dev/pass34, SCSI device
/dev/pass35 -d scsi # /dev/pass35, SCSI device
/dev/pass36 -d scsi # /dev/pass36, SCSI device
/dev/pass37 -d scsi # /dev/pass37, SCSI device

[root@NAS] /# smartctl -a /dev/pass2 -d scsi
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca260c7ca44
Serial number: XXXXXXXX
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Sun Aug 28 19:28:01 2016 SAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Disabled or Not Supported

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 38 C
Drive Trip Temperature: 0 C

Error Counter logging not supported


[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No self-tests have been logged

[root@NAS] /# gpart show
=> 34 15628053101 mfisyspd0 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd1 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd2 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd3 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd4 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd5 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd6 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd7 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd8 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd9 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd10 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd11 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd12 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd13 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd14 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd15 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd16 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd17 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd18 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd19 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd20 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd21 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd22 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd23 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd24 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd25 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd26 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd27 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd28 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd29 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd30 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd31 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd32 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd33 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd34 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 15628053101 mfisyspd35 GPT (7.3T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)
15628053128 7 - free - (3.5K)

=> 34 781422701 nvd0 GPT (373G)
34 222 - free - (111K)
256 204800 1 efi (100M)
205056 781217672 2 freebsd-zfs (373G)
781422728 7 - free - (3.5K)

[root@NAS] /# mfiutil show drives
mfi0 Physical Drives:
2 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S14
3 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S3
4 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S10
5 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S15
6 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S17
7 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S23
8 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S0
9 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S4
10 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S5
11 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S12
12 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S6
13 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S2
14 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S8
15 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S9
16 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S5
17 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S1
18 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S7
19 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S4
20 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S11
21 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S21
22 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S20
23 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S9
24 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S16
25 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S10
26 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S18
27 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S8
28 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S3
29 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S19
30 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S7
31 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S13
32 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S2
33 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S6
34 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E1:S22
35 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S1
36 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S11
37 ( 7452G) JBOD <WDC WD80EFZX-68U 0A83 serial=XXXXXXX> SATA E2:S0

[root@NAS] /# mfiutil show adapter
mfi0 Adapter:
Product Name: AVAGO 3108 MegaRAID
Serial Number:
Firmware: 24.15.0-0018
RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
Battery Backup: present
NVRAM: 32K
Onboard Memory: 2048M
Minimum Stripe: 64K
Maximum Stripe: 1M

[root@NAS] /usr/bin# MegaCli -PDlist -a0 | grep '^Device Id:'
Device Id: 8
Device Id: 17
Device Id: 13
Device Id: 3
Device Id: 9
Device Id: 10
Device Id: 12
Device Id: 18
Device Id: 14
Device Id: 23
Device Id: 4
Device Id: 20
Device Id: 11
Device Id: 31
Device Id: 2
Device Id: 5
Device Id: 24
Device Id: 6
Device Id: 26
Device Id: 29
Device Id: 22
Device Id: 21
Device Id: 34
Device Id: 7
Device Id: 37
Device Id: 35
Device Id: 32
Device Id: 28
Device Id: 19
Device Id: 16
Device Id: 33
Device Id: 30
Device Id: 27
Device Id: 15
Device Id: 25
Device Id: 36

 

[DWELLER_ZA]

[root@NAS] /usr/bin# MegaCli -AdpAllInfo -aAll

Adapter #0

==============================================================================
Versions
================
Product Name : AVAGO 3108 MegaRAID
Serial No :
FW Package Build: 24.15.0-0018

Mfg. Data
================
Mfg. Date : 00/00/00
Rework Date : 00/00/00
Revision No :
Battery FRU : N/A

Image Versions in Flash:
================
BIOS Version : 6.31.03.0_4.17.08.00_0x06140200
Ctrl-R Version : 5.16-0300
Preboot CLI Version: 01.07-05:#%0000
FW Version : 4.650.00-6223
NVDATA Version : 3.1602.00-0003
Boot Block Version : 3.07.00.00-0003

Pending Images in Flash
================
None

PCI Info
================
Controller Id : 0000
Vendor Id : 1000
Device Id : 005d
SubVendorId : 15d9
SubDeviceId : 0809

Host Interface : PCIE

ChipRevision : C0

Link Speed : 0
Number of Frontend Port: 0
Device Interface : PCIE

Number of Backend Port: 8
Port : Address
0 500304801ee354bf
1 0000000000000000
2 0000000000000000
3 0000000000000000
4 0000000000000000
5 0000000000000000
6 0000000000000000
7 0000000000000000

HW Configuration
================
SAS Address : 500304801aad7d01
BBU : Present
Alarm : Present
NVRAM : Present
Serial Debugger : Present
Memory : Present
Flash : Present
Memory Size : 2048MB
TPM : Absent
On board Expander: Absent
Upgrade Key : Absent
Temperature sensor for ROC : Present
Temperature sensor for controller : Absent

ROC temperature : 56 degree Celsius

Settings
================
Current Time : 17:3:59 8/28, 2016
Predictive Fail Poll Interval : 300sec
Interrupt Throttle Active Count : 16
Interrupt Throttle Completion : 50us
Rebuild Rate : 30%
PR Rate : 30%
BGI Rate : 30%
Check Consistency Rate : 30%
Reconstruction Rate : 30%
Cache Flush Interval : 4s
Max Drives to Spinup at One Time : 2
Delay Among Spinup Groups : 12s
Physical Drive Coercion Mode : Disabled
Cluster Mode : Disabled
Alarm : Enabled
Auto Rebuild : Enabled
Battery Warning : Enabled
Ecc Bucket Size : 15
Ecc Bucket Leak Rate : 1440 Minutes
Restore HotSpare on Insertion : Disabled
Expose Enclosure Devices : Enabled
Maintain PD Fail History : Enabled
Host Request Reordering : Enabled
Auto Detect BackPlane Enabled : SGPIO/i2c SEP
Load Balance Mode : Auto
Use FDE Only : No
Security Key Assigned : No
Security Key Failed : No
Security Key Not Backedup : No
Default LD PowerSave Policy : Controller Defined
Maximum number of direct attached drives to spin up in 1 min : 10
Auto Enhanced Import : No
Any Offline VD Cache Preserved : No
Allow Boot with Preserved Cache : No
Disable Online Controller Reset : No
PFK in NVRAM : Yes
Use disk activity for locate : No
POST delay : 90 seconds
BIOS Error Handling : Stop On Errors
Current Boot Mode :Normal
Capabilities
================
RAID Level Supported : RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Supported Drives : SAS, SATA

Allowed Mixing:

Mix in Enclosure Allowed
Mix of SAS/SATA of HDD type in VD Allowed

Status
================
ECC Bucket Count : 0

Limitations
================
Max Arms Per VD : 32
Max Spans Per VD : 8
Max Arrays : 128
Max Number of VDs : 64
Max Parallel Commands : 928
Max SGE Count : 60
Max Data Transfer Size : 8192 sectors
Max Strips PerIO : 128
Max LD per array : 16
Min Strip Size : 64 KB
Max Strip Size : 1.0 MB
Max Configurable CacheCade Size: 0 GB
Current Size of CacheCade : 0 GB
Current Size of FW Cache : 1954 MB

Device Present
================
Virtual Drives : 0
Degraded : 0
Offline : 0
Physical Devices : 39
Disks : 36
Critical Disks : 0
Failed Disks : 0

Supported Adapter Operations
================
Rebuild Rate : Yes
CC Rate : Yes
BGI Rate : Yes
Reconstruct Rate : Yes
Patrol Read Rate : Yes
Alarm Control : Yes
Cluster Support : No
BBU : Yes
Spanning : Yes
Dedicated Hot Spare : Yes
Revertible Hot Spares : Yes
Foreign Config Import : Yes
Self Diagnostic : Yes
Allow Mixed Redundancy on Array : No
Global Hot Spares : Yes
Deny SCSI Passthrough : No
Deny SMP Passthrough : No
Deny STP Passthrough : No
Support Security : No
Snapshot Enabled : No
Support the OCE without adding drives : Yes
Support PFK : Yes
Support PI : Yes
Support Boot Time PFK Change : No
Disable Online PFK Change : No
Support LDPI Type1 : No
Support LDPI Type2 : No
Support LDPI Type3 : No
PFK TrailTime Remaining : 0 days 0 hours
Support Shield State : Yes
Block SSD Write Disk Cache Change: Yes
Support Online FW Update : Yes

Supported VD Operations
================
Read Policy : Yes
Write Policy : Yes
IO Policy : Yes
Access Policy : Yes
Disk Cache Policy : Yes
Reconstruction : Yes
Deny Locate : No
Deny CC : No
Allow Ctrl Encryption: No
Enable LDBBM : Yes
Support Breakmirror : Yes
Power Savings : No

Supported PD Operations
================
Force Online : Yes
Force Offline : Yes
Force Rebuild : Yes
Deny Force Failed : No
Deny Force Good/Bad : No
Deny Missing Replace : No
Deny Clear : No
Deny Locate : No
Support Temperature : Yes
NCQ : Yes
Disable Copyback : No
Enable JBOD : Yes
Enable Copyback on SMART : No
Enable Copyback to SSD on SMART Error : Yes
Enable SSD Patrol Read : No
PR Correct Unconfigured Areas : Yes
Enable Spin Down of UnConfigured Drives : Yes
Disable Spin Down of hot spares : No
Spin Down time : 30
T10 Power State : No
Error Counters
================
Memory Correctable Errors : 0
Memory Uncorrectable Errors : 0

Cluster Information
================
Cluster Permitted : No
Cluster Active : No

Default Settings
================
Phy Polarity : 0
Phy PolaritySplit : 0
Background Rate : 30
Strip Size : 256kB
Flush Time : 4 seconds
Write Policy : WB
Read Policy : Adaptive
Cache When BBU Bad : Disabled
Cached IO : No
SMART Mode : Mode 6
Alarm Disable : Yes
Coercion Mode : None
ZCR Config : Unknown
Dirty LED Shows Drive Activity : No
BIOS Continue on Error : 0
Spin Down Mode : None
Allowed Device Type : SAS/SATA Mix
Allow Mix in Enclosure : Yes
Allow HDD SAS/SATA Mix in VD : Yes
Allow SSD SAS/SATA Mix in VD : No
Allow HDD/SSD Mix in VD : No
Allow SATA in Cluster : No
Max Chained Enclosures : 16
Disable Ctrl-R : No
Enable Web BIOS : No
Direct PD Mapping : No
BIOS Enumerate VDs : Yes
Restore Hot Spare on Insertion : No
Expose Enclosure Devices : Yes
Maintain PD Fail History : Yes
Disable Puncturing : No
Zero Based Enclosure Enumeration : No
PreBoot CLI Enabled : No
LED Show Drive Activity : Yes
Cluster Disable : Yes
SAS Disable : No
Auto Detect BackPlane Enable : SGPIO/i2c SEP
Use FDE Only : No
Enable Led Header : Yes
Delay during POST : 0
EnableCrashDump : Yes
Disable Online Controller Reset : No
EnableLDBBM : Yes
Un-Certified Hard Disk Drives : Allow
Treat Single span R1E as R10 : No
Max LD per array : 16
Power Saving option : Don't Auto spin down Configured Drives
Max power savings option is not allowed for LDs. Only T10 power conditions are to be used.
Default spin down time in minutes: 30
Enable JBOD : No
TTY Log In Flash : No
Auto Enhanced Import : No
BreakMirror RAID Support : Yes
Disable Join Mirror : Yes
Enable Shield State : Yes
Time taken to detect CME : 60s

Exit Code: 0x00

 

[Mirfster]

Hi Gents,

I've received my server and have one issue, how do I handle S.M.A.R.T. within FreeNAS?

I put my LSI 3108 into JBOD and setup my 4 x 9 vdevs within FreeNAS and noticed the following error when starting the S.M.A.R.T. service:

notifier: Starting smartd.
root: /usr/local/etc/rc.d/smartd: WARNING: failed to start smartd
notifier: /usr/local/etc/rc.d/smartd: WARNING: failed to start smartd

I've read that this is due to LSI not passing the S.M.A.R.T. info through to FreeNAS.

Any ideas or advice?

Correct me if I am wrong, but isn't that an actual Hardware Raid as opposed to a HBA (thinking LSI 3008)? I don't have first hand experience with the card though.

Can you elaborate on what you did to put it into JBOD? Only reason I am asking is because some Hardware Raid card (think most) do not fully/properly do JBOD.

Not being able to obtain SMART info would be a big concern if it were me...

 

[DWELLER_ZA]

Hi Mirfster, thanks for the reply.

Originally I posed the question, exactly for that reason:

Do I need to swap out http://www.supermicro.com/products/accessories/addon/AOC-S3108L-H8iR.cfm for this one http://www.supermicro.com/products/accessories/addon/AOC-S3008L-L8i.cfm ?

I think the 3008 would be fine, performance wise, and I'm guessing it would be a tad cheaper if you wanted to save some cash.

I must have misunderstood depasseg, or misread the sentence.

Can you elaborate on what you did to put it into JBOD?

I went into the MegaRAID configuration utility on boot up by pressing CTRL-R.
I cleared the configuration that was on there. There was an option to enable JBOD, which I enabled. Rebooted.

Not being able to obtain SMART info would be a big concern if it were me...

This is exactly why I'm not taking the server live until this issue is sorted.
I've emailed my supplier to see what we can do regarding swapping the 3108 for a 3008.

I'd just like some confirmation that this is indeed my problem and that I will need to swap the card. (Not specifically from you :))

 

[Mirfster]

Here is a bit of info on SuperMicro's page:
http://www.supermicro.com/products/nfo/storage_cards.cfm

Weird thing is that I would *think* that based on the "Port" and "RAID" Columns the "AOC-S3008L-L8e" is the one you want. However, in my experiences when it ends with "e" that is for "External" and usually "i" indicates "Internal". Kinda confusing...

 

[DWELLER_ZA]

Thank you for the effort, it's sort of the same conclusion I came too. Hopefully my supplier will be able to swap the cards for me :)

 

[DWELLER_ZA]

Swapped the 3108 for the 3008, flashed to IT mode, and S.M.A.R.T. working now. YEY, thanks everybody

 

[Mirfster]

Glad to hear you were able to get the cards swapped and things are working as desired.