.conf files for large mem machines?

[aghadjip]

Hopefully this is the right area to ask this. I was wondering what options everyone was using for sysctl.conf and loader.conf for large (16g+) mem machines.

I came from Freenas7 and had quite a few modifications in these files but i only had 4 gigs of memory then.

Performance is quite good so far, so im wondering if I should leave well enough alone.

Cifs transfers at 70 MB/s, iscsi disk io is at 68 MB/s, nfs needs some work, very spiky averaging 38.1 MB/s using dd

Eventually i plan to add a ssd cache to the zpool (6 x 1 tb) to see if that improves anything at all.

Top info
Mem: 143M Active, 47M Inact, 12G Wired, 744K Cache, 232M Buf, 3383M Free

 

[William Grzybowski]

Did you test the speeds locally? dd, iozone, etc?

 

[aghadjip]

DD on the NAS itself

[root@fuel] /mnt/ZFS_Pool1/www# dd if=/dev/zero of=/mnt/ZFS_Pool1/www/ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 16.516020 secs (1269768378 bytes/sec)

[root@fuel] /mnt/ZFS_Pool1/www# dd of=/dev/null if=/mnt/ZFS_Pool1/www/ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 14.265375 secs (1470099440 bytes/sec)


DD on linux VM root drive hosted via iscsi(ctrl c'ed didnt want to wait)

[4]face[/home/aghadjip]: dd if=/dev/zero of=/home/aghadjip/ddfile bs=2048k count=10000
^C7664+0 records in
7664+0 records out
16072572928 bytes (16 GB) copied, 243.341 s, 66.0 MB/s


[5]face[/home/aghadjip]: dd of=/dev/null if=/home/aghadjip/ddfile bs=2048k count=10000
^C2460+0 records in
2459+0 records out
5156896768 bytes (5.2 GB) copied, 70.2995 s, 73.4 MB/s

same VM to NFS store

[2]face[/mnt/www]: df .
Filesystem 1K-blocks Used Available Use% Mounted on
10.0.10.15:/mnt/ZFS_Pool1/www
26214400 5646016 20568384 22% /mnt/www
[3]face[/mnt/www]: sudo dd if=/dev/zero of=/mnt/www/ddfile bs=2048k count=10000
[sudo] password for aghadjip:
^C3109+0 records in
3109+0 records out
6520045568 bytes (6.5 GB) copied, 84.1965 s, 77.4 MB/s

[4]face[/mnt/www]: sudo dd of=/dev/null if=/mnt/www/ddfile bs=2048k count=10000
^C785+0 records in
784+0 records out
1644167168 bytes (1.6 GB) copied, 23.6919 s, 69.4 MB/s

Sorry getting impatient waiting for dd.

Now i have an NFS stored VM (windows) that is getting about half the performance. Measured with HDtune pro Image attached.

Cifs reads and writes are attached.

Writes is slightly faster

Im thinking there is definitely room for improvement.

-Andrew

 

[William Grzybowski]

Indeed...
I'm not happy with the local speed test... are you using compression on zfs root or dataset? If so disable it and do it again, otherwise try increasing the filesize or using another tool (like iozone)
Also, test network speed using iperf

 

[aghadjip]

Thanks,

Does not one run 16gig mem or greater? I was getting awesome throughput on freenas 7 with 4 gig of memory and tuned. If anyone has 8+ memory id love to see everyones configs.

William,

Ive had compression on but not for the iscsi mount. I have 2 dual gig nics in my esx box and the nas is lacp lagged... I dont think we will turn up anything there. I was hoping for some insight into tuning zfs for large memory as im sure there are people here who have setups similar or better than mine. I guess i can start scouring the web to see what people have setup. Appreciate the help.

-Andrew