tngri
Dabbler
- Joined
- Jun 7, 2017
- Messages
- 39
board: $MOTHERBOARD memory/RAM: $RAM drives: $DRIVES power unit: $PSU case: $CASE etc.
I'd use a mix of different brands. Exos drives have been good for us.Ok after resilvering, two more sata disk with sector error / smart fail even on short test. same after cable/sata port switching
I got only one more so I may order new ones because goind further... Just waiting for long smart test result on all other drivers now
I also removed the chelsio eth adapter for now, just in case, because it's hot and maybe to close from my sata cables ...
I used WD Red 4TB, I think I'll stick on them because I not sure 8TB etc are good, same heat etc... what do you think guys ?
Ho sorry I got it now sorry, no write cache with ZFS, only kind of optimisation for writing sequentially a queue of write transactions right ?You mixed up what I wrote about read and write. With SMB (async write by default) write speeds higher than read ones are not that uncommon.
testing: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=64 fio-3.27 Starting 1 process testing: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [m(1)][100.0%][r=294MiB/s,w=288MiB/s][r=293,w=287 IOPS][eta 00m:00s] testing: (groupid=0, jobs=1): err= 0: pid=5080: Wed Sep 15 11:22:47 2021 read: IOPS=497, BW=498MiB/s (522MB/s)(5091MiB/10225msec) bw ( KiB/s): min=188205, max=926087, per=100.00%, avg=514924.85, stdev=243211.70, samples=20 iops : min= 183, max= 904, avg=502.50, stdev=237.58, samples=20 write: IOPS=503, BW=504MiB/s (528MB/s)(5149MiB/10225msec); 0 zone resets bw ( KiB/s): min=212490, max=999521, per=100.00%, avg=517233.70, stdev=255919.68, samples=20 iops : min= 207, max= 976, avg=504.80, stdev=249.96, samples=20 cpu : usr=8.37%, sys=4.04%, ctx=8408, majf=0, minf=1 IO depths : 1=0.1%, 2=0.4%, 4=2.1%, 8=8.7%, 16=19.3%, 32=64.0%, >=64=5.3% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.9%, 8=0.7%, 16=0.8%, 32=0.4%, 64=1.2%, >=64=0.0% issued rwts: total=5091,5149,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=498MiB/s (522MB/s), 498MiB/s-498MiB/s (522MB/s-522MB/s), io=5091MiB (5338MB), run=10225-10225msec WRITE: bw=504MiB/s (528MB/s), 504MiB/s-504MiB/s (528MB/s-528MB/s), io=5149MiB (5399MB), run=10225-10225msec
PS C:\WINDOWS\system32> iperf2.exe -c 10.11.12.13 ------------------------------------------------------------ Client connecting to 10.11.12.13, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [344] local 10.11.12.113 port 56210 connected with 10.11.12.13 port 5001 [ ID] Interval Transfer Bandwidth [344] 0.0-10.0 sec 384 KBytes 314 Kbits/sec
ant@mastodon:~ % iperf -c 10.11.12.113 -w 64k ------------------------------------------------------------ Client connecting to 10.11.12.113, TCP port 5001 TCP window size: 62.7 KByte (WARNING: requested 62.5 KByte) ------------------------------------------------------------ [ 3] local 10.11.12.13 port 64926 connected with 10.11.12.113 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.2 sec 2.14 MBytes 1.77 Mbits/sec ant@mastodon:~ % iperf -c totr -w 64k error: Name does not resolve ant@mastodon:~ % iperf -c totor -w 64k ------------------------------------------------------------ Client connecting to totor, TCP port 5001 TCP window size: 62.7 KByte (WARNING: requested 62.5 KByte) ------------------------------------------------------------ [ 3] local 192.168.1.80 port 44356 connected with 192.168.1.200 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 2.15 MBytes 1.79 Mbits/sec
# windows to freenas iperf3.exe -c 10.11.12.13 Connecting to host 10.11.12.13, port 5201 [ 4] local 10.11.12.113 port 61730 connected to 10.11.12.13 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 621 MBytes 5.21 Gbits/sec [ 4] 1.00-2.00 sec 714 MBytes 5.99 Gbits/sec [ 4] 2.00-3.00 sec 734 MBytes 6.16 Gbits/sec [ 4] 3.00-4.00 sec 719 MBytes 6.03 Gbits/sec [ 4] 4.00-5.00 sec 726 MBytes 6.09 Gbits/sec [ 4] 5.00-6.00 sec 754 MBytes 6.33 Gbits/sec [ 4] 6.00-7.00 sec 747 MBytes 6.27 Gbits/sec [ 4] 7.00-8.00 sec 740 MBytes 6.21 Gbits/sec [ 4] 8.00-9.00 sec 740 MBytes 6.21 Gbits/sec [ 4] 9.00-10.00 sec 749 MBytes 6.28 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 7.07 GBytes 6.08 Gbits/sec sender [ 4] 0.00-10.00 sec 7.07 GBytes 6.08 Gbits/sec receiver iperf Done. # -------------------------------------------- # freenas to windows ant@mastodon:~ % iperf3 -c 10.11.12.113 ^C- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr iperf3: interrupt - the client has terminated ant@mastodon:~ % iperf3 -c 10.11.12.113 Connecting to host 10.11.12.113, port 5201 [ 5] local 10.11.12.13 port 56539 connected to 10.11.12.113 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 59.3 MBytes 498 Mbits/sec 0 208 KBytes [ 5] 1.00-2.00 sec 70.5 MBytes 591 Mbits/sec 0 208 KBytes [ 5] 2.00-3.00 sec 67.7 MBytes 568 Mbits/sec 0 208 KBytes [ 5] 3.00-4.00 sec 73.9 MBytes 620 Mbits/sec 0 208 KBytes [ 5] 4.00-5.00 sec 74.8 MBytes 628 Mbits/sec 0 208 KBytes [ 5] 5.00-6.00 sec 73.0 MBytes 612 Mbits/sec 0 208 KBytes [ 5] 6.00-7.00 sec 74.2 MBytes 622 Mbits/sec 0 208 KBytes [ 5] 7.00-8.00 sec 73.9 MBytes 620 Mbits/sec 0 208 KBytes [ 5] 8.00-9.00 sec 72.2 MBytes 606 Mbits/sec 0 208 KBytes [ 5] 9.00-10.00 sec 70.5 MBytes 591 Mbits/sec 0 208 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 710 MBytes 596 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 710 MBytes 596 Mbits/sec receiver
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 2861588MB (5860533168 512 byte sectors) ada2: quirks=0x1<4K> ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device ada3: Serial Number WD-WCC4N6DU12DF ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 2861588MB (5860533168 512 byte sectors) ada3: quirks=0x1<4K> ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 ada4: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device ada4: Serial Number WD-WCC4N0TCY4DN ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 2861588MB (5860533168 512 byte sectors) ada4: quirks=0x1<4K> ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 ada5: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device ada5: Serial Number WD-WCC4N3KV6HFY ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 2861588MB (5860533168 512 byte sectors) ada5: quirks=0x1<4K> ada6 at ahcich14 bus 0 scbus14 target 0 lun 0 ada6: <INTEL SSDSA2M160G2GC 2CV102M3> ATA-7 SATA 2.x device ada6: Serial Number CVPO9480015S160AGN ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 152626MB (312579695 512 byte sectors) ada6: quirks=0x1<4K> ada6 at ahcich14 bus 0 scbus14 target 0 lun 0 ada6: <INTEL SSDSA2M160G2GC 2CV102M3> s/n CVPO9480015S160AGN detached (ada6:ahcich14:0:0:0): Periph destroyed ada6 at ahcich15 bus 0 scbus15 target 0 lun 0 ada6: <WDC WD10EZEX-00WN4A0 01.01A01> ACS-3 ATA SATA 3.x device ada6: Serial Number WD-WCC6Y5NY158U ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 953868MB (1953523055 512 byte sectors) ada6 at ahcich15 bus 0 scbus15 target 0 lun 0 ada6: <WDC WD10EZEX-00WN4A0 01.01A01> s/n WD-WCC6Y5NY158U detached (ada6:ahcich15:0:0:0): Periph destroyed ada6 at ahcich15 bus 0 scbus15 target 0 lun 0 ada6: <WDC WD10EZEX-00WN4A0 01.01A01> ACS-3 ATA SATA 3.x device ada6: Serial Number WD-WCC6Y5NY158U ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 953868MB (1953523055 512 byte sectors) ada6 at ahcich15 bus 0 scbus15 target 0 lun 0 ada6: <WDC WD10EZEX-00WN4A0 01.01A01> s/n WD-WCC6Y5NY158U detached (ada6:ahcich15:0:0:0): Periph destroyed ada6 at ahcich15 bus 0 scbus15 target 0 lun 0 ada6: <ST2000DM006-2DM164 CC26> ACS-2 ATA SATA 3.x device ada6: Serial Number Z560LLML ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 1907729MB (3907029168 512 byte sectors) ada6: quirks=0x1<4K> ada6 at ahcich15 bus 0 scbus15 target 0 lun 0 ada6: <ST2000DM006-2DM164 CC26> s/n Z560LLML detached (ada6:ahcich15:0:0:0): Periph destroyed ada6 at ahcich15 bus 0 scbus15 target 0 lun 0 ada6: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device ada6: Serial Number WD-WCC4N7SVP36F ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 2861588MB (5860533168 512 byte sectors) ada6: quirks=0x1<4K> ada6 at ahcich15 bus 0 scbus15 target 0 lun 0 ada6: <WDC WD30EFRX-68EUZN0 82.00A82> s/n WD-WCC4N7SVP36F detached (ada6:ahcich15:0:0:0): Periph destroyed
whole_disk: 1 DTL: 283 create_txg: 4 com.delphix:vdev_zap_leaf: 38 children[1]: type: 'disk' id: 1 guid: 15437823396949053150 path: '/dev/gptid/64859b1d-132c-11ec-b185-0007430bdb28' DTL: 13960 create_txg: 4 com.delphix:vdev_zap_leaf: 13627 children[1]: type: 'mirror' id: 1 guid: 13450808671679462686 metaslab_array: 96 metaslab_shift: 34 ashift: 12 asize: 2998440558592 is_log: 0 create_txg: 25 com.delphix:vdev_zap_top: 93 children[0]: type: 'disk' id: 0 guid: 3779443773719678560 path: '/dev/gptid/dd0fbcf3-a93a-11e9-95a1-d05099c188d1' whole_disk: 1 DTL: 280 create_txg: 25 com.delphix:vdev_zap_leaf: 94 children[1]: type: 'disk' id: 1 guid: 7759890288198451326 path: '/dev/gptid/df012241-a93a-11e9-95a1-d05099c188d1' whole_disk: 1 DTL: 277 create_txg: 25 com.delphix:vdev_zap_leaf: 95 children[2]: type: 'mirror' id: 2 guid: 4337055466478410576 metaslab_array: 104 metaslab_shift: 34 ashift: 12 asize: 2998440558592 is_log: 0 create_txg: 33 com.delphix:vdev_zap_top: 101 children[0]: type: 'disk' id: 0 guid: 321227728348730692 path: '/dev/gptid/ec9e4ce9-a93a-11e9-95a1-d05099c188d1' whole_disk: 1 DTL: 275 create_txg: 33 com.delphix:vdev_zap_leaf: 102 children[1]: type: 'disk' id: 1 guid: 5048171274672213817 path: '/dev/gptid/1996590b-42c8-11ea-87b4-d05099d3036a' whole_disk: 1 DTL: 274 create_txg: 33 com.delphix:vdev_zap_leaf: 957 features_for_read: com.delphix:hole_birth com.delphix:embedded_data
fruit:metadata = stream nfs4:chown = true [Ressources] ea support = No path = /mnt/tank/ressources read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [videos] ea support = No path = /mnt/tank/videos read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [gaming] ea support = No path = /mnt/tank/gaming read only = No vfs objects = fruit streams_xattr shadow_copy_zfs ixnas aio_fbsd fruit:resource = stream fruit:metadata = stream nfs4:chown = true [home] ea support = No path = /mnt/tank/home read only = No vfs objects = fruit streams_xattr shadow_copy_zfs ixnas crossrename recycle aio_fbsd crossrename:sizelimit = 2048 recycle:subdir_mode = 0700 recycle:directory_mode = 0777 recycle:touch = yes recycle:versions = yes recycle:keeptree = yes recycle:repository = .recycle/%U fruit:resource = stream fruit:metadata = stream nfs4:chown = true [Travail] ea support = No kernel share modes = No path = /mnt/tank/Travail posix locking = No read only = No vfs objects = fruit streams_xattr shadow_copy_zfs ixnas aio_fbsd fruit:resource = stream fruit:metadata = stream nfs4:chown = true [backups] ea support = No kernel share modes = No path = /mnt/tank/backups posix locking = No read only = No vfs objects = fruit streams_xattr shadow_copy_zfs ixnas aio_fbsd fruit:resource = stream fruit:metadata = stream nfs4:chown = true
Load smb config files from /usr/local/etc/smb4.conf Loaded services file OK. Server role: ROLE_STANDALONE # Global parameters [global] aio max threads = 2 bind interfaces only = Yes disable spoolss = Yes dns proxy = No enable web service discovery = Yes kernel change notify = No load printers = No logging = file max log size = 5120 nsupdate command = /usr/local/bin/samba-nsupdate -g registry shares = Yes restrict anonymous = 2 server role = standalone server server string = TrueNAS Server unix extensions = No idmap config *: range = 90000001-100000000 fruit:nfs_aces = No idmap config * : backend = tdb directory name cache size = 0 dos filemode = Yes
# Global parameters [global] aio max threads = 2 bind interfaces only = Yes disable spoolss = Yes dns proxy = No enable web service discovery = Yes interfaces = 127.0.0.1 10.0.0.13 192.168.1.80 kernel change notify = No load printers = No logging = file max log size = 5120 netbios aliases = mastodon nsupdate command = /usr/local/bin/samba-nsupdate -g registry shares = Yes restrict anonymous = 2 server role = standalone server server string = FreeNAS Server unix extensions = No username map = /usr/local/etc/smbusername.map username map cache time = 60 idmap config *: range = 90000001-100000000 fruit:nfs_aces = No idmap config * : backend = tdb directory name cache size = 0 dos filemode = Yes [Son] ea support = No path = /mnt/tank/son read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [emulation] ea support = No path = /mnt/tank/emulation read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [Papier] ea support = No path = /mnt/tank/papier read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [Formation] ea support = No path = /mnt/tank/formations read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [Softs] ea support = No path = /mnt/tank/soft read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [downloads] ea support = No path = /mnt/tank/downloads read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [Ressources] ea support = No path = /mnt/tank/ressources read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [videos] ea support = No path = /mnt/tank/videos read only = No vfs objects = zfs_space fruit streams_xattr zfsacl fruit:resource = stream fruit:metadata = stream nfs4:chown = true [gaming] ea support = No path = /mnt/tank/gaming read only = No vfs objects = fruit streams_xattr shadow_copy_zfs ixnas aio_fbsd fruit:resource = stream fruit:metadata = stream nfs4:chown = true [home] ea support = No path = /mnt/tank/home read only = No vfs objects = fruit streams_xattr shadow_copy_zfs ixnas crossrename recycle aio_fbsd crossrename:sizelimit = 2048 recycle:subdir_mode = 0700 recycle:directory_mode = 0777 recycle:touch = yes recycle:versions = yes recycle:keeptree = yes recycle:repository = .recycle/%U fruit:resource = stream fruit:metadata = stream nfs4:chown = true [Travail] ea support = No kernel share modes = No path = /mnt/tank/Travail posix locking = No read only = No vfs objects = fruit streams_xattr shadow_copy_zfs ixnas aio_fbsd fruit:resource = stream fruit:metadata = stream nfs4:chown = true [backups] ea support = No kernel share modes = No path = /mnt/tank/backups posix locking = No read only = No vfs objects = fruit streams_xattr shadow_copy_zfs ixnas aio_fbsd fruit:resource = stream fruit:metadata = stream nfs4:chown = true