10Gbe - 8Gbps with iperf, 1.3Mbps with NFS

[LearnLearnLearn]

The Samsung MZ-75E1T0 850 EVO 1TB seems to imply that even with encryption enabled, it gets over 500 for R/W.
There is a non pro and a pro version but they say that even the non pro version is mainly a pro version.
If this can work, they are 1TB at least.

Also, the SLOG PCI card that I bought has two slots. I was kind of thinking maybe I cold get another smaller optane and use that as the OS which would give me two additional slots for 1TB drives.

 

[Samuel Tai]

The Samsungs are a good choice. Many here have had good success with them.

 

[LearnLearnLearn]

Do you feel this exact model would work out then, even though it seems to have the encryption built in.

 

[Samuel Tai]

The more recent drives have much faster enryption processors, so I think you'll have good chances. If it can really do 500 MBps with encryption, at that capacity, you'll be set.

 

[LearnLearnLearn]

I have no idea, it's just what I found while reading up on them. Still, I guess they would be better than the non SSD no matter what.

 

[LearnLearnLearn]

Do any folks run the OS from a single optane? They sound very reliable and I'll have a spare m.2 slot.

 

[Samuel Tai]

While not an Optane, I run my system off a single M.2. I've found M.2s reliable enough not to need mirroring for my boot pool.

 

[LearnLearnLearn]

ok well, that could mean two more 2.5 slots then.

 

[LearnLearnLearn]

I'll buy another optane for the second slot. I'll have a 32GB for SLOG and 16GB should be plenty for the OS I think.

 

[LearnLearnLearn]

Anyone here know how to use one of these?
There is nothing on the card stating which connector is port 1 or 2.
There is no information explaining why there is a SATA data connector.
Does the first m.2 card get power and data flow through the PCIe connection while maybe the second one only gets data but needs the SATA data connector for data transfer?

There is one SATA connector on the motherboard (for tape drive) but no power so if I could power the second optane from the PCIe card, then I could use that one for the Os.

The seller only repeats sales info, not how one uses this.

 

[Samuel Tai]

Since PCI 2.0 x4 only supports 1x NVMe device, I suspect you've got a board that supports 1x M.2 NVMe drive, with the upper slot only supporting 1x M.2 SATA drive, with a pass-through SATA connector for a motherboard SATA port.

 

[LearnLearnLearn]

Is there anything you don't know about storage? :). I'm impressed.

Ok, so, I could use this for both the Optane and a second optane for the OS, wired to the mobo using the SATA cable.
I think this is what you are confirming.

I found one of these on my desk which I might be able to use for the OS if I don't need the speed of an Optane.

Amazon.com

What do you think of these? DELAIHE 870 EVO 1TB SSD MZ-77E1T0B/AM
No idea why they have Samsung part numbers but a different name but looking online, it almost looks like this is a division of Samsung.
These are affordable at this point in the budget as I've had no luck finding very good quality ones.

 

[Samuel Tai]

I'd be careful of that vendor. See the reviews at https://www.newegg.com/Seller-Store/DELAIHE-DLH-Official-Store/about.

 

[LearnLearnLearn]

Sorry, I edited the above comment with more info about the PCIe card. If I can find a reliable seller, do you feel those drives could be worth the cost?

 

[Samuel Tai]

I'd be leery of fakes. Stick with the known good vendors.

 

[LearnLearnLearn]

Then my only choice will be to buy smaller SSD's so I can use the storage mainly to run production shared htlp/php pages and move vms around. I have free slots in the blades so I could stick some drives in those for longer term storage.

Just need to find the right price because all this is adding nearly another grand to this setup and I'm already over budget on this one.

 

[LearnLearnLearn]

I was able to pick up 10 of these so I sure hope they will do the trick.
The specs are nice, they have a SAS interface and no encryption capabilities.

sTec S842E800M2 800GB Enterprise Serial Attached SCSI SAS Solid State Drive SSD

sTec S842E800M2 / 94100-02054-LI6BBCTU 800GB SFF 2.5inch Enterprise SAS SSD Endurance Write Intensive - Refurbished

www.disctech.com

I installed the PCI card with the two modules on it and the TN installer sees both storage devices. The 64GB kingdata is being seen as a DogFish SSD. Never heard of that.

I re-installed the OS onto the 64GB device and have the drives on order so now have to wait to move on.
Those are 800GB drives and I'll have 10 slots.
Since I went through the process of adding a SLOG and now running the OS on one of the m.2 modules, I'd like to maximize the amount of storage and performance since I'll have a pretty nice setup from what I started with. Maybe being delayed will end up being ok.

My only concern is losing the OS but from what I've read, if that happens, no storage or access will be lost, I just need to re-install the OS and reload the saved configuration and all is back.

One question is, do I really need all this extra memory and could it do fine with just one CPU instead of two?
I have a limited power budget for each 10U of rack space I use and could use saving a little power where ever I can at this point.
The system has two E5-2660 CPU's installed at the moment. I could save around 90 watts by removing one.

 

[NugentS]

Do any folks run the OS from a single optane? They sound very reliable and I'll have a spare m.2 slot.

I do, I run my QNAS (signature) off 2 * M10 Optanes (16GB) in USB to M.2 adapters. I use the Optanes as they were the same price as similar non Optane 16GB devices

 

[LearnLearnLearn]

Thanks for the input.
I feel confident about running the OS on the m.2 device since there seems to be a nice recovery path should that device fail without losing data.
I'll set up the SLOG device next. I have re-entered the tunables, hope I didn't miss something.

Code:

# midclt call tunable.query | jq
[
  {
    "id": 3,
    "value": "0",
    "type": "LOADER",
    "comment": "?",
    "enabled": true,
    "var": "hw.pci.honor_msi_blacklist"
  },
  {
    "id": 4,
    "value": "2048",
    "type": "LOADER",
    "comment": "256",
    "enabled": true,
    "var": "hw.vmx.rxndesc"
  },
  {
    "id": 5,
    "value": "16",
    "type": "LOADER",
    "comment": "8",
    "enabled": true,
    "var": "hw.vmx.rxnqueue"
  },
  {
    "id": 6,
    "value": "1024",
    "type": "LOADER",
    "comment": "512",
    "enabled": true,
    "var": "hw.vmx.txndesc"
  },
  {
    "id": 7,
    "value": "16777216",
    "type": "SYSCTL",
    "comment": "2097152",
    "enabled": true,
    "var": "kern.ipc.maxsockbuf"
  },
  {
    "id": 8,
    "value": "1024",
    "type": "SYSCTL",
    "comment": "128",
    "enabled": true,
    "var": "kern.ipc.soacceptqueue"
  },
  {
    "id": 9,
    "value": "2048",
    "type": "SYSCTL",
    "comment": "256",
    "enabled": true,
    "var": "net.inet.ip.intr_queue_maxlen"
  },
  {
    "id": 10,
    "value": "4194304",
    "type": "SYSCTL",
    "comment": "16384",
    "enabled": true,
    "var": "net.inet.tcp.recvbuf_inc"
  },
  {
    "id": 11,
    "value": "16777216",
    "type": "SYSCTL",
    "comment": "2097152",
    "enabled": true,
    "var": "net.inet.tcp.recvbuf_max"
  },
  {
    "id": 12,
    "value": "4194304",
    "type": "SYSCTL",
    "comment": "65536",
    "enabled": true,
    "var": "net.inet.tcp.recvspace"
  },
  {
    "id": 13,
    "value": "4194304",
    "type": "SYSCTL",
    "comment": "8192",
    "enabled": true,
    "var": "net.inet.tcp.sendbuf_inc"
  },
  {
    "id": 14,
    "value": "16777216",
    "type": "SYSCTL",
    "comment": "2097152",
    "enabled": true,
    "var": "net.inet.tcp.sendbuf_max"
  },
  {
    "id": 15,
    "value": "4194304",
    "type": "SYSCTL",
    "comment": "32768",
    "enabled": true,
    "var": "net.inet.tcp.sendspace"
  },
  {
    "id": 16,
    "value": "0",
    "type": "SYSCTL",
    "comment": "Preclude unnecessary ARP info logging",
    "enabled": true,
    "var": "net.link.ether.inet.log_arp_movements"
  },
  {
    "id": 17,
    "value": "2048",
    "type": "SYSCTL",
    "comment": "256",
    "enabled": true,
    "var": "net.route.netisr_maxqlen"
  },
  {
    "id": 18,
    "value": "yes",
    "type": "LOADER",
    "comment": "",
    "enabled": true,
    "var": "cc_cubic_load"
  },
  {
    "id": 19,
    "value": "dctcp",
    "type": "SYSCTL",
    "comment": "",
    "enabled": true,
    "var": "net.inet.tcp.cc.algorithm"
  },
  {
    "id": 20,
    "value": "yes",
    "type": "LOADER",
    "comment": "",
    "enabled": true,
    "var": "cc_dctcp_load"
  }
]

 

[LearnLearnLearn]

SLOG...

# sedutil-cli --scan
**snip**
/dev/nvme0 No INTEL MEMPEK1J032GA K4110400

Nice, there's my Optane device. Let's follow the article I found here.

SLOG benchmarking and finding the best SLOG

I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you need for a "great" SLOG: 1. Fast throughput 2...

www.truenas.com

# diskinfo -wS /dev/nvme0
diskinfo: /dev/nvme0: ioctl(DIOCGMEDIASIZE) failed, probably not a disk.

Uh? Looking online, this error seems to means that truenas thinks this is a nonexistent drive yet it shows it in the drives list.
check the device to see if there's a problem and that looks fine.

# smartctl -a /dev/nvme0
**snips**
=== START OF INFORMATION SECTION ===
Model Number: INTEL MEMPEK1J032GA
NVMe Version: <1.2
Namespace 1 Size/Capacity: 29,260,513,280 [29.2 GB]
Namespace 1 Formatted LBA Size: 512

SMART/Health Information (NVMe Log 0x02)
No Errors Logged

# camcontrol devlist
<SEAGATE ST91000640SS AS08> at scbus0 target 8 lun 0 (pass0,da0)
<SEAGATE ST91000640SS AS08> at scbus0 target 9 lun 0 (pass1,da1)
<SEAGATE ST91000640SS AS08> at scbus0 target 10 lun 0 (pass2,da2)
<SEAGATE ST91000640SS AS08> at scbus0 target 11 lun 0 (pass3,da3)
<IBM-ESXS MBE2147RC SC18> at scbus0 target 12 lun 0 (pass4,da4)
<SEAGATE ST91000640SS AS08> at scbus0 target 13 lun 0 (pass5,da5)
<IBM-ESXS MBE2147RC SC17> at scbus0 target 14 lun 0 (pass6,da6)
<SEAGATE ST91000640SS AS08> at scbus0 target 15 lun 0 (pass7,da7)
<SEAGATE ST91000640SS AS08> at scbus0 target 16 lun 0 (pass8,da8)
<SEAGATE ST91000640SS AS08> at scbus0 target 17 lun 0 (pass9,da9)
<Dogfish SSD 64GB T0114A0> at scbus5 target 0 lun 0 (pass10,ada0)
<AHCI SGPIO Enclosure 2.00 0001> at scbus7 target 0 lun 0 (pass11,ses0)
Not sure what this last entry is but it's not the Optane which doesn't seem to be in the list.

Ugh... monday... need more coffee!