so i have a extra intel p3605 1.6tb PCIe nvme card, and a few extra slots, so im considering adding it as a L2ARC, but wanted to get some feedback as to if i should even add an L2ARC. (i was all ready to do it, and even have tested it on a separate FN test server i keep, but after reading quite a bit, im having second thoughts as to if i should even run a l2arc).
The server specs/pool types are in my SIG below, it does have 256g of fast ecc ram. My usecase for the files on this FN box (physical box btw):
MY PLAN IS TO SPLIT THE L2ARC (ie ~400gb each for 3x of these pools)- please, i would like this discussion to mainly focus on if i should add an L2ARC, not on the wisdom (or lack of) of splitting/sharing a single 1.6tb drive l2arc. Thanks
pool name="he8x8TBz2": (has split optane slog)
SMB file access during work hours (from 4x windows pcs, here at my home office)
Backups via SMB and NFS - (ie veeam for some vms onsite, some offsite)
maybe 5x low use VMs running via NFS from it.
about 20tb of video files, that a plex VM accesses via SMB share
pool name="ssd3x6x480GBz2": (has split optane slog)
~ 15x VMs running via NFS from it. (all service type VMs, many windows 2012r2)
2x large PRTG VMs store their files here via NFS - (PRTG = snmp network motioning app)
2x heavy disk IO "torrent" VMs running 24/7 (use this ssd pool via NFS as temp/scratch download storage, then move out to other HDD pools via SMB when download completed)
pool name="red4tbENCz1": (has split optane slog)
3x milestone NVRs (cctv) archive their video to it daily (about 75gb per day)
light daily Backups via SMB - (ie some FTP / syncbackpro)
1x low use VMs running via NFS from it.
pool name="he5x8TBz1": (no slog)
ONLY ~ 20tb of video files, that a plex VM accesses via SMB share
random output of arc summary:
zpool status:
Some stats on the p3605;
(i know diskinfo is a write/slog relevant test, so not really relevant for a L2ARC drive, but i figured i would post at the bottom anyway)
The server specs/pool types are in my SIG below, it does have 256g of fast ecc ram. My usecase for the files on this FN box (physical box btw):
MY PLAN IS TO SPLIT THE L2ARC (ie ~400gb each for 3x of these pools)- please, i would like this discussion to mainly focus on if i should add an L2ARC, not on the wisdom (or lack of) of splitting/sharing a single 1.6tb drive l2arc. Thanks
pool name="he8x8TBz2": (has split optane slog)
SMB file access during work hours (from 4x windows pcs, here at my home office)
Backups via SMB and NFS - (ie veeam for some vms onsite, some offsite)
maybe 5x low use VMs running via NFS from it.
about 20tb of video files, that a plex VM accesses via SMB share
pool name="ssd3x6x480GBz2": (has split optane slog)
~ 15x VMs running via NFS from it. (all service type VMs, many windows 2012r2)
2x large PRTG VMs store their files here via NFS - (PRTG = snmp network motioning app)
2x heavy disk IO "torrent" VMs running 24/7 (use this ssd pool via NFS as temp/scratch download storage, then move out to other HDD pools via SMB when download completed)
pool name="red4tbENCz1": (has split optane slog)
3x milestone NVRs (cctv) archive their video to it daily (about 75gb per day)
light daily Backups via SMB - (ie some FTP / syncbackpro)
1x low use VMs running via NFS from it.
pool name="he5x8TBz1": (no slog)
ONLY ~ 20tb of video files, that a plex VM accesses via SMB share
random output of arc summary:
Code:
oot@freenas:~# arcstat.py
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
03:28:22 46G 2.5G 5 1.8G 3 748M 87 583M 1 182G 181G
# arc_summary.py
System Memory:
0.09% 225.63 MiB Active, 5.13% 12.80 GiB Inact
92.35% 230.43 GiB Wired, 0.00% 0 Bytes Cache
2.19% 5.46 GiB Free, 0.24% 617.35 MiB Gap
Real Installed: 256.00 GiB
Real Available: 99.97% 255.93 GiB
Real Managed: 97.49% 249.51 GiB
Logical Total: 256.00 GiB
Logical Used: 92.87% 237.74 GiB
Logical Free: 7.13% 18.26 GiB
Kernel Memory: 3.77 GiB
Data: 98.90% 3.73 GiB
Text: 1.10% 42.53 MiB
Kernel Memory Map: 249.51 GiB
Size: 7.81% 19.48 GiB
Free: 92.19% 230.03 GiB
Page: 1
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Storage pool Version: 5000
Filesystem Version: 5
Memory Throttle Count: 0
ARC Misc:
Deleted: 1.64b
Mutex Misses: 1.82m
Evict Skips: 1.82m
ARC Size: 73.21% 181.94 GiB
Target Size: (Adaptive) 73.23% 181.99 GiB
Min Size (Hard Limit): 12.50% 31.06 GiB
Max Size (High Water): 8:1 248.51 GiB
ARC Size Breakdown:
Recently Used Cache Size: 94.96% 172.80 GiB
Frequently Used Cache Size: 5.04% 9.18 GiB
ARC Hash Breakdown:
Elements Max: 7.23m
Elements Current: 79.66% 5.76m
Collisions: 314.86m
Chain Max: 6
Chains: 440.55k
Page: 2
------------------------------------------------------------------------
ARC Total accesses: 46.83b
Cache Hit Ratio: 94.56% 44.28b
Cache Miss Ratio: 5.44% 2.55b
Actual Hit Ratio: 94.34% 44.18b
Data Demand Efficiency: 53.34% 2.62b
Data Prefetch Efficiency: 12.03% 844.63m
CACHE HITS BY CACHE LIST:
Anonymously Used: 0.19% 85.99m
Most Recently Used: 4.48% 1.98b
Most Frequently Used: 95.29% 42.19b
Most Recently Used Ghost: 0.01% 3.38m
Most Frequently Used Ghost: 0.03% 14.49m
CACHE HITS BY DATA TYPE:
Demand Data: 3.15% 1.40b
Prefetch Data: 0.23% 101.60m
Demand Metadata: 96.60% 42.77b
Prefetch Metadata: 0.02% 8.12m
CACHE MISSES BY DATA TYPE:
Demand Data: 47.93% 1.22b
Prefetch Data: 29.16% 743.03m
Demand Metadata: 22.68% 578.00m
Prefetch Metadata: 0.23% 5.84m
Page: 3
------------------------------------------------------------------------
Page: 4
------------------------------------------------------------------------
DMU Prefetch Efficiency: 7.83b
Hit Ratio: 9.43% 738.60m
Miss Ratio: 90.57% 7.10b
Page: 5
------------------------------------------------------------------------
zpool status:
Code:
root@freenas:~ # zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:04 with 0 errors on Sat Sep 14 03:45:04 2019
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
errors: No known data errors
pool: he5x8TBz1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
he5x8TBz1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/f9ad6c75-d748-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/fac4063a-d748-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/febf393e-d748-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/02df6e7a-d749-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/044e4834-d749-11e9-bae4-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
pool: he8x8TBz2
state: ONLINE
scan: scrub repaired 0 in 1 days 07:56:31 with 0 errors on Wed Sep 11 12:55:02 2019
config:
NAME STATE READ WRITE CKSUM
he8x8TBz2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/ab4a8be7-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/ac4d1939-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/ad4d7a28-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/ae4b5c46-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/af54390e-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/b05a4b41-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/b15cf9b3-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/b2675d33-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/e1eba74e-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/e34d7ec3-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/e4a114e3-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/e608021e-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/e727453f-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/eb0401da-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/eed1a7b2-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/f013e8e4-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
logs
gptid/ed098586-c7bc-11e9-975e-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
pool: hus9x4TBz2
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
hus9x4TBz2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/9a817388-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/9d201743-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/a077cd9e-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/a800a13f-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/b4bd6736-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/c2588d0b-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/cfea05aa-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/dabb68f5-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/e76a9705-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
pool: red4tbENCz1
state: ONLINE
scan: resilvered 487G in 0 days 01:41:38 with 0 errors on Mon Sep 2 01:56:54 2019
config:
NAME STATE READ WRITE CKSUM
red4tbENCz1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/85a9cf3a-cd40-11e9-975e-00259084f1c8.eli ONLINE 0 0 0
gptid/7afcb936-c450-11e9-bbf0-00259084f1c8.eli ONLINE 0 0 0
gptid/7ca5439e-c450-11e9-bbf0-00259084f1c8.eli ONLINE 0 0 0
gptid/7e4df0e4-c450-11e9-bbf0-00259084f1c8.eli ONLINE 0 0 0
gptid/7f8f24d1-c450-11e9-bbf0-00259084f1c8.eli ONLINE 0 0 0
logs
gptid/e82c363e-c7bc-11e9-975e-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
pool: ssd3x6x480GBz2
state: ONLINE
scan: scrub repaired 0 in 0 days 00:29:33 with 0 errors on Mon Sep 16 02:03:06 2019
config:
NAME STATE READ WRITE CKSUM
ssd3x6x480GBz2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/b238bbd8-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b2e8a1de-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b382351d-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b421e71c-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b4d79d37-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b580104e-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/b670b4d0-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b714597f-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b7d8bca7-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b88979e7-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b94e1216-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b9f0c93b-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/bb0fa8d3-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/bbb3f9cf-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/bc74e749-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/bd35b67b-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/bde00cf9-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/be9f1fbc-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
logs
gptid/f166ac78-c7bc-11e9-975e-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
root@freenas:~ #
Some stats on the p3605;
(i know diskinfo is a write/slog relevant test, so not really relevant for a L2ARC drive, but i figured i would post at the bottom anyway)
Code:
root@freenas:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
1600321314816 # mediasize in bytes (1.5T)
3125627568 # mediasize in sectors
131072 # stripesize
0 # stripeoffset
INTEL SSDPEDME016T4S # Disk descr.
CVMD4414004X1P6KGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Synchronous random writes:
0.5 kbytes: 22.4 usec/IO = 21.8 Mbytes/s
1 kbytes: 22.1 usec/IO = 44.3 Mbytes/s
2 kbytes: 20.4 usec/IO = 95.9 Mbytes/s
4 kbytes: 15.4 usec/IO = 253.8 Mbytes/s
8 kbytes: 20.5 usec/IO = 380.7 Mbytes/s
16 kbytes: 26.9 usec/IO = 581.8 Mbytes/s
32 kbytes: 36.4 usec/IO = 858.3 Mbytes/s
64 kbytes: 49.5 usec/IO = 1263.3 Mbytes/s
128 kbytes: 94.9 usec/IO = 1316.8 Mbytes/s
256 kbytes: 168.1 usec/IO = 1487.0 Mbytes/s
512 kbytes: 316.9 usec/IO = 1577.7 Mbytes/s
1024 kbytes: 648.5 usec/IO = 1542.0 Mbytes/s
2048 kbytes: 1288.0 usec/IO = 1552.8 Mbytes/s
4096 kbytes: 2586.7 usec/IO = 1546.4 Mbytes/s
8192 kbytes: 5110.6 usec/IO = 1565.4 Mbytes/s
root@freenas:~ #
root@freenas:~ # smartctl -a /dev/nvme0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: INTEL SSDPEDME016T4S
Serial Number: CVMD4414004X1P6KGN
Firmware Version: 8DV1RA13
PCI Vendor ID: 0x8086
PCI Vendor Subsystem ID: 0x108e
IEEE OUI Identifier: 0x5cd2e4
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,600,321,314,816 [1.60 TB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Sat Sep 21 16:47:45 2019 CDT
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0006): Wr_Unc DS_Mngmt
Maximum Data Transfer Size: 32 Pages
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 25.00W - - 0 0 0 0 0 0
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2
1 - 512 8 2
2 - 512 16 2
3 - 4096 0 0
4 - 4096 8 0
5 - 4096 64 0
6 - 4096 128 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning: 0x00
Temperature: 30 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 1%
Data Units Read: 2,757,905 [1.41 TB]
Data Units Written: 1,053,105 [539 GB]
Host Read Commands: 6,052,800,385
Host Write Commands: 8,210,809,402
Controller Busy Time: 0
Power Cycles: 388
Power On Hours: 19,775
Unsafe Shutdowns: 1
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged
Last edited:
