PCI SSD Options

[jgreco]

Given the numbers we've seen, it should be worth it in your case.

Try doing an "iostat daXX 1" on your L2ARC device and see how much traffic is hitting it. That's mostly reads not hitting your pool.

 

[kspare]

ok ill keep an eye on it tomorrow when people log on,

 

[kspare]

i rebooted an idle terminal server and got this:

tin tout KB/t tps MB/s us ni sy in id
0 44 54.28 135 7.15 0 0 2 0 98
0 132 51.14 65 3.24 0 0 2 0 98
0 44 53.91 86 4.52 0 0 4 0 95
0 44 67.59 98 6.46 0 0 4 1 95
0 44 63.03 259 15.93 0 0 5 0 94
0 44 61.82 262 15.80 0 0 4 1 95
0 44 63.37 127 7.85 0 0 3 1 97
0 44 95.71 293 27.36 0 0 4 1 96
0 44 79.74 416 32.36 0 0 7 1 93
0 44 65.41 420 26.80 0 0 4 1 95
0 44 70.46 210 14.43 0 0 2 0 97
0 44 64.68 148 9.34 0 0 2 1 98
0 44 63.54 52 3.22 0 0 1 0 99
0 44 67.43 21 1.38 0 0 2 0 98

it seems like its working, not huge numbers either....

 

[jgreco]

You want ARC to be serving the heavy hitters. L2ARC is bonus load reduction on the pool.

 

[kspare]

ok, so if I was to upgrade my ram to 256gb and my cpus, the next upgrade would be the new intel pci ssd, 400gb for Slog and the bigger for L2Arc?

 

[diehard]

ok, so if I was to upgrade my ram to 256gb and my cpus, the next upgrade would be the new intel pci ssd, 400gb for Slog and the bigger for L2Arc?

Sounds great to me. Depends on your usage, a faster SLOG might help you out a lot more than the extra RAM.

 

[jgreco]

Sounds great to me. Depends on your usage, a faster SLOG might help you out a lot more than the extra RAM.

If you have a suggestion for a faster SLOG device than the Intel 750 or P37oo SSD's that doesn't also bankrupt you, we're all ears.

 

[diehard]

If you have a suggestion for a faster SLOG device than the Intel 750 or P37oo SSD's that doesn't also bankrupt you, we're all ears.

Sorry i meant him upgrading to a faster SLOG like one of those NVMe drives. I kinda assume that since he needs an SLOG that he already has something?

 

[jgreco]

The 750 and P3700's are NVMe ...

 

[diehard]

The 750 and P3700's are NVMe ...

Yeah i know i meant "one of those" as in the ones you mentioned. Lol my English is failing me today, sorry.

 

[kspare]

We are setting up a new customer, this is what it's doing ont he l2arc when outlook is syncing on 10 accounts:

tin tout KB/t tps MB/s us ni sy in id
0 96 93.14 646 58.78 0 0 5 1 94
0 221 74.59 376 27.37 0 0 6 1 93
0 88 78.39 310 23.71 0 0 5 1 94
0 132 84.91 388 32.14 0 0 3 1 96
0 88 82.20 317 25.42 0 0 2 0 97
0 132 76.40 344 25.64 0 0 6 1 93
0 88 79.84 501 39.02 0 0 17 2 81
0 132 75.19 581 42.69 0 0 4 1 94
0 88 80.32 558 43.80 0 0 8 1 91
0 132 75.66 485 35.80 0 0 4 1 95
0 88 76.90 469 35.19 0 0 8 1 91
0 132 71.16 488 33.88 0 0 3 1 97
0 88 73.03 468 33.34 0 0 6 1 93
0 132 84.59 366 30.21 0 0 8 1 91
0 88 85.00 323 26.79 0 0 10 2 88
0 132 91.69 397 35.51 0 0 8 1 91
0 88 95.23 556 51.75 0 0 10 1 89
0 220 84.89 336 27.83 0 0 3 1 97
0 88 86.88 541 45.94 0 0 15 1 84
0 132 82.22 268 21.50 0 0 7 1 92
0 88 86.14 810 68.15 0 0 9 1 89
0 132 80.06 314 24.53 0 0 3 1 96
0 88 91.57 803 71.83 0 0 17 1 82
0 132 90.34 427 37.63 0 0 17 2 82

 

[kspare]

This is the current arc start while doing all that:

[*]Put your data type(s) here...
[*]4.20TiB / 18.3TiB (ESXI_1)
[*]2.87GiB / 464GiB (freenas-boot)
[*]46.20GiB (MRU: 37.28GiB, MFU: 8.92GiB) / 64.00GiB
[*]Hit ratio -> 84.08% (higher is better)
[*]Prefetch -> 23.12% (higher is better)
[*]Hit MFU:MRU -> 69.30%:27.84% (higher ratio is better)
[*]Hit MRU Ghost -> 0.21% (lower is better)
[*]Hit MFU Ghost -> 1.00% (lower is better)

 

[jgreco]

We are setting up a new customer, this is what it's doing ont he l2arc when outlook is syncing on 10 accounts:

tin tout KB/t tps MB/s us ni sy in id
0 96 93.14 646 58.78 0 0 5 1 94

If that's the effect of just one customer, it seems likely that you'd see much greater hit rates with more. Not that reducing pool traffic by 50MB/sec is bad.

 

[kspare]

it was a pretty heavy load when we got everyone logged in and their mailboxes were syncing up.

By more do you mean more memory or a larger l2arc?

 

[jgreco]

Actually I meant more activity (more customers), but it could be yes to either of those as well. That's the 64GB system?

 

[kspare]

I have two of these bad boys on order.

http://www.newegg.com/Product/Produ...167300&cm_re=intel_750-_-20-167-300-_-Product

One for SLOG and One for L2 Arc.

 

[cyberjock]

No, you ordered two.. one for you and one for me. :)

 

[kspare]

ha! Funny guy! They are on backorder until may 11th, so still a long wait.

In the mean time, I took out all the ssd's so no slog and no l2cache and it seems to run better, even though I got alot of hits on the l2 cache.

We keep having drives fail out of our freenas box, not sure why yet so we're going to add some more hot spares in the space where we had the ssd's.

Next to order is more memory 64gb to 192gb and then cpus.

So now we will run 22 2tb drives in mirroed vdevs with 2 hot spares (which don't rebuild automagically)

 

[kspare]

I just got my shipping notification from new egg, the intel 750s are on the way!

 

[diehard]

I just got my shipping notification from new egg, the intel 750s are on the way!

Keep us updated on how they work out