Large SMB Transfers Causing SMB and HTTP Services to Stop

[FlexGunship]

Okay, I've seen a lot of bad posts, so I'm going to do my best to create a great one!

Server hardware:

• Motherboard: eVGA nForce 750i SLI (https://www.evga.com/support/manuals/files/122-YW-E173.pdf)
• CPU: Intel Core 2 Quad Q6600 (http://ark.intel.com/products/29765/Intel-Core2-Quad-Processor-Q6600-8M-Cache-2_40-GHz-1066-MHz-FSB)
• RAM: 8GB (4x 2GB) DDR2-6400 Kingston (non-buffered, non-ECC) (Note: I know I should have 16GB, but I just can't find compatible 4GB sticks for less than $40 each)
• Boot media: 160GB WD Blue 7200RPM 8MB SATAII 3.0Gb/s
• Array 1: (2x) 2TB WD Red 5400 RPM 64MB SATAIII 6.0Gb/s + (1x) Seagate Barracuda Green 2TB 5900 RPM 64MB SATAIII 6.0Gb/s In RaidZ (wizard recommendation; 5.1 TB usable)
• Hard drive controllers for boot drive and Seagate: onboard
• Hard drive controllers for 2x WD Red drives: Syba PCIe card Marvell 88SE9230 in 16x PCIe slot (http://a.co/c5wd4wa)
• Network card: re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> (taken directly from 'messages' file)
• FreeNAS Version: FreeNAS-11-MASTER-201706220409 (876bb13) (I just upgraded earlier today hoping to resolve this problem)

Description of problems:

1. Apart from S.M.A.R.T. no services can be set to "start on boot". This means I always need to log into the web UI to click "start now" for SMB and SSH
2. After about 30 - 150GB of data is transferred to the server, the HTTP service (web ui) and SMB services both stop. I can no longer reach the FreeNAS server and I cannot access the web UI. The server has NOT hard crashed because it remains "pingable" until I perform a hard reset

Below is, what seems to be, the most relevant data from the /var/log/messages file. Please note the panic at the end.

Code:

Jun 21 16:40:57 athena kernel: re0: link state changed to DOWN
Jun 21 16:40:57 athena kernel: re0: link state changed to DOWN
Jun 21 16:40:57 athena kernel: re0: link state changed to UP
Jun 21 16:40:57 athena kernel: re0: link state changed to UP
Jun 21 16:40:57 athena kernel: nfe0: link state changed to DOWN
Jun 21 16:40:57 athena kernel: nfe0: link state changed to DOWN
Jun 21 16:40:59 athena ntpd[2135]: ntpd 4.2.8p10-a (1): Starting
Jun 21 16:41:04 athena root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Jun 21 16:41:11 athena su: in openpam_check_error_code(): pam_sm_setcred(): unexpected return value 12
Jun 21 16:41:11 athena root: /etc/rc: WARNING: failed precmd routine for minio
Jun 21 16:49:05 athena smbd: in openpam_check_error_code(): pam_sm_acct_mgmt(): unexpected return value 12
Jun 22 00:00:00 athena syslog-ng[1724]: Configuration reload request received, reloading configuration;
Jun 22 07:14:05 athena syslog-ng[1588]: syslog-ng starting up; version='3.7.3'
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (ada2:ata2:0:0:0): READ_DMA. ACB: c8 00 c8 c7 20 40 00 00 00 00 0d 00
Jun 22 07:14:05 athena (ada2:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (ada2:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retry was blocked
Jun 22 07:14:05 athena ada3 at ata3 bus 0 scbus11 target 0 lun 0
Jun 22 07:14:05 athena ada3: <ST2000DL003-9VT166 CC32> s/n 5YD3ZG9E detached
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retry was blocked
Jun 22 07:14:05 athena ada2 at ata2 bus 0 scbus10 target 0 lun 0
Jun 22 07:14:05 athena ada2: <WDC WD1600YD-01NVB1 10.02E01> s/n WD-WCANM4547411 detached
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retry was blocked
Jun 22 07:14:05 athena ada5 at ata5 bus 0 scbus13 target 0 lun 0
Jun 22 07:14:05 athena ada5: <Hitachi HUA721075KLA330 GK8OA70M> s/n GTE200P8G1LN7E detached
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe2:ata5:0:0:0): Error 5, Retries exhausted
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe0:ata3:0:0:0): Retrying command
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): CAM status: Command timeout
Jun 22 07:14:05 athena (aprobe1:ata2:0:0:0): Retrying command
Jun 22 07:14:05 athena panic: I/O to pool 'bigboy' appears to be hung on vdev guid 15772048961736765878 at '/dev/gptid/021b52ee-56cc-11e7-a55e-0014d124ac68'.
Jun 22 07:14:05 athena cpuid = 3
Jun 22 07:14:05 athena KDB: stack backtrace:
Jun 22 07:14:05 athena db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe027d36e970
Jun 22 07:14:05 athena vpanic() at vpanic+0x186/frame 0xfffffe027d36e9f0
Jun 22 07:14:05 athena panic() at panic+0x43/frame 0xfffffe027d36ea50
Jun 22 07:14:05 athena vdev_deadman() at vdev_deadman+0x194/frame 0xfffffe027d36eaa0
Jun 22 07:14:05 athena vdev_deadman() at vdev_deadman+0x41/frame 0xfffffe027d36eaf0
Jun 22 07:14:05 athena spa_deadman() at spa_deadman+0x86/frame 0xfffffe027d36eb20
Jun 22 07:14:05 athena taskqueue_run_locked() at taskqueue_run_locked+0x127/frame 0xfffffe027d36eb80
Jun 22 07:14:05 athena taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xfffffe027d36ebb0
Jun 22 07:14:05 athena fork_exit() at fork_exit+0x85/frame 0xfffffe027d36ebf0
Jun 22 07:14:05 athena fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe027d36ebf0
Jun 22 07:14:05 athena --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Jun 22 07:14:05 athena KDB: enter: panic
Jun 22 07:14:05 athena Copyright (c) 1992-2017 The FreeBSD Project.
Jun 22 07:14:05 athena Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jun 22 07:14:05 athena	 The Regents of the University of California. All rights reserved.
Jun 22 07:14:05 athena FreeBSD is a registered trademark of The FreeBSD Foundation.
Jun 22 07:14:05 athena FreeBSD 11.1-BETA1 #0 9e57002a9(freenas/11-stable): Wed Jun 21 05:43:00 UTC 2017
Jun 22 07:14:05 athena root@gauntlet:/freenas-clean/freenas/_BE/objs/freenas-clean/freenas/_BE/os/sys/FreeNAS.amd64 amd64
Jun 22 07:14:05 athena FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
Jun 22 07:14:05 athena CPU: Intel(R) Core(TM)2 Quad CPU	Q6600  @ 2.40GHz (2400.03-MHz K8-class CPU)
Jun 22 07:14:05 athena Origin="GenuineIntel"  Id=0x6fb  Family=0x6  Model=0xf  Stepping=11
Jun 22 07:14:05 athena Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jun 22 07:14:05 athena Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
Jun 22 07:14:05 athena AMD Features=0x20100800<SYSCALL,NX,LM>
Jun 22 07:14:05 athena AMD Features2=0x1<LAHF>
Jun 22 07:14:05 athena VT-x: HLT,PAUSE
Jun 22 07:14:05 athena TSC: P-state invariant, performance statistics
Jun 22 07:14:05 athena real memory  = 9663676416 (9216 MB)
Jun 22 07:14:05 athena avail memory = 8246071296 (7864 MB)

I was running Ubuntu 14.10 server before this, and Windows Home Server 2012 before that. I'm NOT an expert in any of this, but it's also not my first rodeo. I can probably get whatever extra data is needed to work this.

Notes:

1. I know I should have more RAM, but a single 4Gb stick of this old stuff costs $40, I just can't justify that; I'd probably migrate back to Ubuntu server
2. The array of drives I have is definitely odd, I agree... but it's what FreeNAS recommended for "media" and I'm not a BSD expert so I just went with the recommendation. There are no critical files here (yet), I have additional drives I plan to mirror with important data after this is all stable.

 

[Jailer]

After about 30 - 150GB of data is transferred to the server, the HTTP service (web ui) and SMB services both stop

That would be from the craptastic Realtek NIC on the motherboard crapping out under load. No surprise there, they are junk, especially under load.

The nforce chipset ATA controllers are dubious at best. If you insist on running this hardware for FreeNAS expect much pain, suffering and frustration and most likely data loss in your future. Seriously.

 

[FlexGunship]

That would be from the craptastic Realtek NIC on the motherboard crapping out under load. No surprise there, they are junk, especially under load.

The nforce chipset ATA controllers are dubious at best. If you insist on running this hardware for FreeNAS expect much pain, suffering and frustration and most likely data loss in your future. Seriously.

Thanks. I'm not using the onboard actually. So maybe I have the wrong data. I specifically installed a stand-alone NIC for gigabit capability. Maybe I've copied the wrong data then. Can you recommend a NIC?

Regarding the nForce controller: is there a reason why FreeNAS/FreeBSD would have difficulty in using this controller versus a Linux or Windows based OS? Is my other HDD controller acceptable? I'll just connect everything to that if so.

EDIT: It's worth noting that most folks using FreeNAS aren't going to be running something like your rig. I used to install FreeNAS on old PCs I got from my college back when it was... hmmm... FreeNAS 5, maybe? 2007, anyway.

If FreeNAS isn't the friendly package is used to be, it might not be for me, then. That'd be a real bummer. Are there other NAS-specific distros that are more hardware-of-last-year friendly?

 

[Jailer]

The nforce controller was never good regardless of OS.

If you are just getting your feet wet to learn FreeNAS then go ahead and use what you have. But if this is going to be a long term solution you really need to give up on this hardware and get something more appropriate if you care about your data.

 

[FlexGunship]

The nforce controller was never good regardless of OS.

If you are just getting your feet wet to learn FreeNAS then go ahead and use what you have. But if this is going to be a long term solution you really need to give up on this hardware and get something more appropriate if you care about your data.

I think I'll just go back to a more stable OS then.

Do you think anyone else would have some insight? It surprises me to only get a "your hardware isn't good enough" type response regarding a network appliance distro. Especially after meeting all of the minimum requirements.

 

[Jailer]

Read the hardware primer thread in the resource section at the top of the page for more information. Just because you meet minimum specs doesn't mean it will be a good reliable system.

 

[FlexGunship]

Read the hardware primer thread in the resource section at the top of the page for more information. Just because you meet minimum specs doesn't mean it will be a good reliable system.

Just read through it. Seems like FreeNAS has gone a different route in recent years. I'll probably just switch back to Ubuntu server.

Why is using a SATA controller card discouraged (page 16), but also using integrated SATA controllers discouraged (your post)? Is there a third option I'm not aware of?

 

[Jailer]

Using the add in cards or integrated SATA controllers is not discouraged. Unsupported and/or poor quality controllers, such as the nforce controllers, are not recommended. The IBM M1015 for example is an add in card that is very highly recommended.

You need an HBA or good quality controllers that passes the drives through to FreeNAS for ZFS to be able to do it's magic. Unfortunately most of the cheap add in cards are port multipliers or RAID cards that don't pass the drives through cleanly and cause more trouble then they are worth.

 

[FlexGunship]

Using the add in cards or integrated SATA controllers is not discouraged. Unsupported and/or poor quality controllers, such as the nforce controllers, are not recommended. The IBM M1015 for example is an add in card that is very highly recommended.

You need an HBA or good quality controllers that passes the drives through to FreeNAS for ZFS to be able to do it's magic. Unfortunately most of the cheap add in cards are port multipliers or RAID cards that don't pass the drives through cleanly and cause more trouble then they are worth.

So, going by your earlier observations (which seems to be correct) it looks like the network adapter is going up and down. I looked it up and it's a TP-Link, which does use Realtek. Cheap, it was $10 from Amazon.

This one: http://a.co/ji7ETd9 is Intel. I realize other components could fail next; but do you believe this would eliminate this one problem so I could work the others?

Also, any thoughts on why I can't set ""SMB" to "Start on boot"? Is that a common issue with the current build?

 

[Jailer]

Is that a common issue with the current build?

No I think it's an odd edge case issue due to your marginal hardware.

 

[FlexGunship]

No I think it's an odd edge case issue due to your marginal hardware.

Code:

Jun 22 16:01:51 athena kernel: re0: link state changed to DOWN
Jun 22 16:01:51 athena kernel: re0: link state changed to DOWN
Jun 22 16:01:51 athena kernel: re0: link state changed to UP
Jun 22 16:01:51 athena kernel: re0: link state changed to UP
Jun 22 16:01:51 athena kernel: nfe0: link state changed to DOWN
Jun 22 16:01:51 athena kernel: nfe0: link state changed to DOWN
Jun 22 16:01:53 athena ntpd[1999]: ntpd 4.2.8p10-a (1): Starting
Jun 22 16:01:58 athena root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Jun 22 16:02:05 athena root: /etc/rc: WARNING: failed precmd routine for minio
Jun 22 16:02:05 athena su: in openpam_check_error_code(): pam_sm_setcred(): unexpected return value 12
Jun 22 16:33:11 athena smbd: in openpam_check_error_code(): pam_sm_acct_mgmt(): unexpected return value 12
Jun 22 18:27:04 athena syslog-ng[1588]: syslog-ng starting up; version='3.7.3'
Jun 22 18:27:04 athena Copyright (c) 1992-2017 The FreeBSD Project.
Jun 22 18:27:04 athena Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

I think I've isolated it to this. I created a separate volume which was a single drive (impossible to "over-tax") and found that there's a vmware related issue (I have no VMs installed, configured, or running).

I don't think it's related to the NIC. Only re0 is used. Only nfe0 shows "DOWN" state.

I plan to open a bug relating to vmware_guestd. Would you agree this is the root issue right now?

EDIT: I also confirmed that it's not related to "overloading" an interface or controller. I did a batch copy of many small files but paused it after every ~1GB. It still failed about 50GB of transfers.

 

[Jailer]

Do you have FreeNAS installed in esxi? If so that's a very important piece of information you left out of your first post.

 

[FlexGunship]

Do you have FreeNAS installed in esxi? If so that's a very important piece of information you left out of your first post.

No. This is a native install on actual metal. No virtualization above or below FreeNAS.

 

[FlexGunship]

To anyone with similar issues searching for answers, I've resolved problem number (1) above:

Problem:

1. Apart from S.M.A.R.T. no services can be set to "start on boot". This means I always need to log into the web UI to click "start now" for SMB and SSH

The resolution to this was to perform a clean install of the latest build of FreeNAS. I'm unsure of why or how this became an issue, but the only other item I changes was to NOT ENABLE autotune (which was enabled since first run of the earlier build). I'm not 100% sure that this was related, but it seems smart to keep it turned off.

TL;DR: if you have services that won't "start on boot" then reinstall FreeNAS, reimport your volumes, and DO NOT ENABLE AUTOTUNE.

A factory reset might also work, but I didn't try.

 

[FlexGunship]

I apologize for the triple post, but I resolved issue number (2), also:

• 2. After about 30 - 150GB of data is transferred to the server, the HTTP service (web ui) and SMB services both stop. I can no longer reach the FreeNAS server and I cannot access the web UI. The server has NOT hard crashed because it remains "pingable" until I perform a hard reset

This was related to the nForce 750i chipset. It was perfectly capable under normal loads with 4 drives (for 6 years), but under FreeNAS loads, apparently, it would give out. Even a single drive connected to the onboard controller would eventually cause FreeNAS to crash during a transfer. This might hint at an underlying instability in FreeNAS or FreeBSD... but for now, the solution is to just use an external SATA controller card.

With all drives connected via PCIe I have no more issues.

I went ahead and found a Dell PowerEdge T410 on Craigslist for $300, so I'll replace my current server with that in a few days (after I quiet it down).

 

[Jailer]

If you haven't already committed to the T410 the HP ML10 gen9 from tigerdirect is a fantastic bargain at $200 ($179 right now with coupon code). Add some more memory and you're all set.

 

[FlexGunship]

If you haven't already committed to the T410 the HP ML10 gen9 from tigerdirect is a fantastic bargain at $200 ($179 right now with coupon code). Add some more memory and you're all set.

Definitely appreciate the suggestion. I haven't committed to the T410 other than verbally. But it comes with 3 new 1TB iron wolf drives, 12Gb of memory, and dual E5607 Xeons. Any particular upsides with the HP ML10? The i3 seems much slower. Is there better support for that processor? Benefits of known working hardware?

EDIT: I'm going to follow your advice, Jailer, and get the HP. Could you direct me towards an equally good deal on memory? Best I could find seems to be to buy a SECOND server and take the 4GB out of it.

 

[Jailer]

This will get you to the minimum spec of 8GB but personally I wouldn't buy less than a 16GB stick.