TN Core locked up hard

[jcizzo]

started memtest last night, been running for 12 hours now. all is clear. i'll let it run through the weekend since i'm going away.

the only thing that changed was that i reinstalled truenas core the other day. i had it previously installed to test hardware and figure out how i want to lay out my datasets and basically learn what i can before committing to everything and migrating all my files over to it... all worked fine with the old install (same truenas version). the only thing that changed is the old SSDs i was using to test; i swapped them out for a pair of larger and new ssds. samsung 870 evos. that's it.. all seemed to be perfectly stable.

maybe i just need to re-gpart the OS drives and reinstall from scratch again..?

the odd observation i've made is that when the drives get kicked out, checking on the drives i find one to be very warm, the other to be cool. happened on all instances.

 

[Davvo]

maybe i just need to re-gpart the OS drives and reinstall from scratch again..?

The installation takes care of everything, you should not partition your boot drive.

 

[jcizzo]

The installation takes care of everything, you should not partition your boot drive.

yes, it takes care of everything. what i meant was, use gparted prior to installation to ensure all partitions have been fully wiped and there's nothing on the drive that could remain behind. i actually have experienced that, where a system was unstable then i gparted it (or used ms diskmgr, or whatever it is) to removed everything at a low level. upon reinstallation, all ran properly. dunno why i didn't think of this the other day..

the only other experience i have with zfs is on my firewalls (pfsense), so this is my foray into it for use as a nas.

btw, ran memtest all weekend long while i was away. no problems.

 

[jcizzo]

The installation takes care of everything, you should not partition your boot drive.

that brings me to a question regarding partitioning of the os drives? why can't we edit the partitions during the installation? during a pfsense installation, i can go in and choose the size of my swap partition. so, while it's probably irrelevant, just by choice i can make the swap space the same size as my ram. i'd prefer to do that as well with truenas.. again, probably not needed but i'd prefer to have the option.

 

[Davvo]

that brings me to a question regarding partitioning of the os drives? why can't we edit the partitions during the installation?

Standard answer: TN is an appliance.

 

[Etorix]

so, while it's probably irrelevant, just by choice i can make the swap space the same size as my ram.

swap >= RAM is a Linux thing. FreeBSD is a bit smarter than that and lives well with swap partitions sized just to accommodate an excess over the current RAM (so can live with swap < RAM). That, and using 2 GB of the data drives for swap and general manoeuvre space (e.g. dealing with replacement drive a tad smaller than the original) means that CORE can install in very small drives. Enjoy it.

 

[jcizzo]

my truenas box has an intel x710-da2 nic for the 10G side.. is there a way to force it to only 1Gb/s? i can't edit anything of the sort from within the interfaces tab.

 

[jcizzo]

i tested a theory of mine, that the 2 ssds are being overwhelmed.. buffer overrun.. the data stream coming over the 10Gig line is too much for the SSDs to either keep up, or between the incoming 10G stream and trying to write it out to the SSD's (since they're only 6G), the cpu just can't keep up. the cpu is an i3-7100t (dual core) with HT'ing disabled. i disabled HT because i noticed that i picked up about 100Mb/s across the wire during file transfers..

-the folder i'm copying contains about 109Gigs of movies.
-when i copy it over the 10G line, the cpu spikes to 70+%, the zfs cache fills after several seconds (i have 32Gigs of ram).
-after several seconds the throughput drops from 1Gb/s to around 500Mb/s.. from there its just a matter of time before the whole thing conks out.. cpu drops to 0%, file transfer hangs, and the only way outta the mess is to drill into the ipmi interface and restart, as doing a restart via the truenas gui doesn't work.

anywho, so the results: that 109Gig folder copies across the 1Gb lan side without issue. the cpu stays around 10%.. maybe pops to 20% from time to time, but all works properly.

i performed the test using an NVME. i copied the same movie folder but to the nvme and i did so across the 10Gb line this time.. went through without a hitch!

so, copying the 109Gig movie folder to the SSD's works properly across the 1Gb lan interface, and copying that same folder works properly across the 10Gb interface but ONLY to the NVME drive.

does anyone think that a 4 core cpu would cure this problem? i can't believe i bought these 10gig nics and i can't take advantage of them..
i'd like to drop the interface speed of the 10G nics to 1Gb/s just to test. if anyone can tell me how to do that i'd appreciate it.

thoughts/comments?

 

[NugentS]

Plug it into a 1Gb port?

 

[Davvo]

i have 32Gigs of ram

Feels small for 10Gbps.

the cpu just can't keep up. the cpu is an i3-7100t (dual core) with HT'ing disabled

SMB requires higher frequencies over core number: if you want to use that 10Gbps you need the speed (also being capped by SATA isn't helpful).

 

[Davvo]

so, copying the 109Gig movie folder to the SSD's works properly across the 1Gb lan interface, and copying that same folder works properly across the 10Gb interface but ONLY to the NVME drive.

samsung 870 evos

Those should be the issue, I vaguely remember their cache being problematic. Can someone please confirm this?

 

[jcizzo]

Plug it into a 1Gb port?

yes, i did that but i'd also like to test the nics.

I've realized that the 10G nics are overkill for my application.. i just move a lotta movies and thought i could take advantage of it but i've realized that i'll never achieve 10G throughput because the spinners will never be able to keep up. so, i'm thinking of dumbing it down to 2.5Gb with some 2.5sfp's.. sucks.. splurged on those intel nics only to have them serve no purpose.

 

[jcizzo]

Those should be the issue, I vaguely remember their cache being problematic. Can someone please confirm this?

interesting..

one thing i've noticed is after one of the drives gets kicked out, if i remove one it's always very hot. i've never known ssds to get hot.

 

[winnielinnie]

i've never known ssds to get hot.

SATA SSDs, those about the size of a 2.5" laptop drive? Maybe not.

But NVMe in an M.2 form-factor? Yes, they can get quite hot.

As "gimmicky" as it sounds, those "M.2 heatsinks" with thermal pads work wonders.

The only catch is that you need to make sure there's room. (In my case, my bulky CPU cooler actually "covers" an M.2 slot on the motherboard, so one of my NVMe's is installed without a heatsink, and it consistently runs about 10-C to 15-C hotter than the one with a heatsink, depending on the load.)

 

[Whattteva]

SATA SSDs, those about the size of a 2.5" laptop drive? Maybe not.

But NVMe in an M.2 form-factor? Yes, they can get quite hot.

As "gimmicky" as it sounds, those "M.2 heatsinks" with thermal pads work wonders.

The only catch is that you need to make sure there's room. (In my case, my bulky CPU cooler actually "covers" an M.2 slot on the motherboard, so one of my NVMe's is installed without a heatsink, and it consistently runs about 10-C to 15-C hotter than the one with a heatsink, depending on the load.)

Yeah, this is consistent with my anecdotal experience. The 2.5" SSD's don't even feel warm when touched, but those M.2 NVMe ones though, are a different story.

 

[NugentS]

yes, i did that but i'd also like to test the nics.

I've realized that the 10G nics are overkill for my application.. i just move a lotta movies and thought i could take advantage of it but i've realized that i'll never achieve 10G throughput because the spinners will never be able to keep up. so, i'm thinking of dumbing it down to 2.5Gb with some 2.5sfp's.. sucks.. splurged on those intel nics only to have them serve no purpose.

I meant plug the 10Gb NiC into a 1Gb port on a switch

 

[jcizzo]

SATA SSDs, those about the size of a 2.5" laptop drive? Maybe not.

But NVMe in an M.2 form-factor? Yes, they can get quite hot.

As "gimmicky" as it sounds, those "M.2 heatsinks" with thermal pads work wonders.

The only catch is that you need to make sure there's room. (In my case, my bulky CPU cooler actually "covers" an M.2 slot on the motherboard, so one of my NVMe's is installed without a heatsink, and it consistently runs about 10-C to 15-C hotter than the one with a heatsink, depending on the load.)

yes, the 2.5 sata ssds.. yeah, never known them to get hot.. the m.2 heatsinks ABSOLUTELY work wonders! as long as they're properly designed. i've played with a couple that didn't work because the heatpipes didn't spread along the drive far enough and never came into contact with the controller, so they couldn't wick away the heat.. the ineo's (dual heat pipes with a fan, although the fans only last for 6 months before going belly-up. i take the fan off. as long as there's some air circulation, it seems to be more than adequate) and the jeyi finned heatsinks are excellent at conducting heat..

 

[jcizzo]

I meant plug the 10Gb NiC into a 1Gb port on a switch

i found a few sfp 2G transceivers in my parts bin.. gonna give those a shot with fiber. the dac cables aren't long enough to reach my switch.

 

[jcizzo]

i found a few sfp 2G transceivers in my parts bin.. gonna give those a shot with fiber. the dac cables aren't long enough to reach my switch.

well those sfp's seem to not like the x710's.. no activity. plugged them into the sfp ports on my switch and they lit up so..

i thought the x710s could do other speeds other than just 1 and 10Gb/s? anyone with experience?

 

[HoneyBadger]

well those sfp's seem to not like the x710's.. no activity. plugged them into the sfp ports on my switch and they lit up so..

If they actually say 2Gbps they may be FC (Fibre Channel) modules and not Ethernet. It's possible the X710 also has a list of approved/allowed modules.