Pulled a disk for testing, web UI not responding.

[diskdiddler]

I'm new to all this but here goes.

Code:

[root@freenas] /dev# dd if=/dev/ad14 of=/dev/null bs=4096k count=1000
1000+0 records in
1000+0 records out
4194304000 bytes transferred in 19.661605 secs (213324600 bytes/sec)

That's a successful read, correct?

 

[diskdiddler]

Well I'm at a loss at this point, the array 100% worked at betwen 60->80mb/s sustained for single large files when I first set it up, I wasn't happy with it but I could deal with it.
I've just spent 90 minutes testing, I can not get it to write a file over 40mb/s at all - no idea why. Deleted array, removed jails, re-created the array, switched even to STRIPING and I'm getting 20 to 30mb/s

Code:

[root@freenas] ~#  dd if=/dev/random of=/mnt/ARRAY/ham.test bs=4096k count=700
700+0 records in
700+0 records out
2936012800 bytes transferred in 71.249542 secs (41207462 bytes/sec)
[root@freenas] ~# dd if=/mnt/ARRAY/ham.test of=/dev/null
5734400+0 records in
5734400+0 records out
2936012800 bytes transferred in 33.384088 secs (87946473 bytes/sec)
[root@freenas] ~# 

If I'm reading that right, 40MB/s writes, 87MB/s reads on a "raid0" array and CIFS can't be impacting that at all.

 

[cyberjock]

Notice that dd is at 97%. That means that the dd command alone is using a full core.

 

[cyberjock]

Wow.. do more reading.

First, inputting from a disk you want to output too is a fail.

Second, inputting from /dev/random is a fail.

So no, none of your tests mean anything at all except you want to move bits from here to there and back. :P

 

[diskdiddler]

I was following some information I'd found in other posts.
Here's me trying to copy from disk 0 to disk 1 (4 and 6?)
Is that right?

Code:

[root@freenas] /# dd if=/dev/ad4 of=/dev/ad6 bs=4096k count=1000
1000+0 records in
1000+0 records out
4194304000 bytes transferred in 44.867344 secs (93482333 bytes/sec)
[root@freenas] /#

Looks like 93MB/s it can read and write at, yeah?
EDIT: also seeing information from posters across the internet claiming they can saturate their link so I'll be damned if I know.

 

[diskdiddler]

This is me copying at 25MB/s - I've re-installed FreeNAS entirely on the USB key. I'm totally baffled.
It's clearly not maxing out the CPU yet 25MB/s sustained on large files - encryption off, compression off.

Code:

last pid:  8308;  load averages:  1.34,  0.76,  0.41    up 0+00:25:10  03:04:40
36 processes:  1 running, 35 sleeping
CPU:  2.5% user,  0.0% nice, 19.4% system,  1.2% interrupt, 76.9% idle
Mem: 168M Active, 93M Inact, 5960M Wired, 65M Buf, 1565M Free
ARC: 5606M Total, 12M MFU, 5518M MRU, 53M Anon, 13M Header, 10M Other
Swap: 12G Total, 12G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
7477 root          1  32    0   323M 20892K select  1   1:27 17.48% smbd
2474 root          6  22    0   299M   109M usem    1   0:14  0.00% python2.7
4908 root         12  20    0   136M 16884K uwait   1   0:01  0.00% collectd
7925 root          4  52    0   168M 46240K select  1   0:01  0.00% python2.7
3127 root          1  52    0   158M 46452K ttyin   0   0:01  0.00% python2.7
2560 www           1  20    0 26044K  5464K kqread  1   0:00  0.00% nginx
4558 root          1  20    0 12036K  1848K select  1   0:00  0.00% syslogd
2073 root          1  20    0 22220K  3984K select  1   0:00  0.00% ntpd
1887 root          1 -52   r0 12044K  8024K nanslp  1   0:00  0.00% watchdogd
5455 root          1  20    0   275M 17760K select  1   0:00  0.00% smbd
8303 root          1  20    0 16556K  2556K CPU0    0   0:00  0.00% top

 

[diskdiddler]

Ok look, sorry for badgering like a crazy person, I've been questioning myself for the last 90 minutes if I definitely saw 75MB/s - but I just went to the reporting page and I definitely remember seeing around 30 to 40MB/s per disk when I was getting 75MB/s
Now take a peek

Definitely something has changed and I'm baffled what it could be?

 

[joeschmuck]

What you haven't said was the files you are copying, it it one large single file or many smaller files? Of course the speed will change dramatically between the two. Also you pool configuration makes a difference as well so what is that now? Do a "zpool status" and post it.

Also, as Cyberjock said, your CPU is a bit underpowered for FreeNAS 9 but it will get the job done, eventually. And the RAM being 8GB means you don't have much RAM to spare for the ARC which is crucial in the NAS performance.

I too would like to see how you will keep the system cool in a cabinet. Maybe you could make a sound isolation box out of some MDF and egg crate foam and place it over the server, but don't forget to provide positive ventilation.

 

[diskdiddler]

Sorry it was a single large file - many of them for testing - file was uncompressable.
I've opted out, sorry and thanks. I googled and the reports from people online are conflicting but it seems to me like I shouldn't be expecting 70MB/s speeds, which is really a bare minumum

 

[joeschmuck]

You could try NAS4Free, I believe the system requirements are a bit lighter, or run FreeNAS 8.3.2-Release, that should be good too.

 

[cyberjock]

I googled and the reports from people online are conflicting but it seems to me like I shouldn't be expecting 70MB/s speeds, which is really a bare minumum

Definitely not with that hardware. The N40L "works" if time isn't a major factor. If it is, you should be looking at more powerful hardware. People that want low power pay for it in performance.. as you have seen first hand.

 

[gpsguy]

The real question is whether you are running on of the hacked BIOS' or not. Assuming disk #5 and #6 are connected to the optical drive connector and eSATA connection, they run in an IDE emulation mode. To disable that and allow AHCI Support on those connections, you need to install a hacked BIOS.

I believe all 6 ports are SATA3 (any way to check?)

For those drives, you'll need more RAM. Regardless of what HP says, you can put 16Gb of RAM in the box.

 

[diskdiddler]

The real question is whether you are running on of the hacked BIOS' or not. Assuming disk #5 and #6 are connected to the optical drive connector and eSATA connection, they run in an IDE emulation mode. To disable that and allow AHCI Support on those connections, you need to install a hacked BIOS.



For those drives, you'll need more RAM. Regardless of what HP says, you can put 16Gb of RAM in the box.

Nope definitely run the hacked BIOS, the disks can do read tests individually at 200MB/s apparently (surprisingly fast)
I've posted on another forum and someone surprisingly said, when they replaced their USB stick performance went up, that doesn't seem logical to me. Almost as if FreeNAS is locking up writing a log file perhaps when handling my ZFS writes to the array? It's plausible but I figured something booting from USB would load linux / bsd into ram and logs would be in ram until committed to the USB drive later / shutdown?

I don't have another spare port to plug in an SSD to test though.
I was considering trying IDE rather than AHCI emulation - but I've got people on other forums claiming vastly superior speeds when writing with FreeNAS :{

As for the RAM, yes, I know the box can support more RAM and it would be nice but 8gb on a 17TB array isn't the end of the world, I've seen many posts from people with 4 and even 2gb of ram still getting decent performance out of FreeNAS. I'm absolutely not convinced the problem is specifically the ram in this instance. If someone can tell me /precisely/ how to benchmark to 100% confirm it's a ram limitation, I'll consider it but just not convinced (sorry)

P.S I know this is MOSTLY irrelevant but I did a DBAN on the disks writing 0's, I got 70MB's sustained writing to all 6 disks at once that's 420MB/s. Now if you factor in the overhead of FreeNAS / ZFS, writing "real data" and what have you, even one FIFTH of the 420MB/s would be 84MB/s
What I really need is a fellow ZFS2, 6 disk N40L owner to chime in with their performance numbers to be honest.

 

[gpsguy]

If you look at section 1.3.2 of the manual you'd see that "For systems with large disk capacity (greater than 8 TB), a general rule of thumb is 1 GB of RAM for every 1 TB of storage."

Folks with 2-4Gb of RAM won't be running ZFS on FreeNAS 9.x. Maybe FreeNAS 0.7 (now known as NAS4Free). ZFS on FreeNAS (even with a small pool), needs a minimum of 8Gb of RAM.

I've seen many posts from people with 4 and even 2gb of ram still getting decent performance out of FreeNAS.

If you search around, you *might* be able to find someone who has posted some performance numbers with a 6 disk RAIDz2 system on a N40L. But, more than likely, it'll be a smaller pool, with possibly 2Tb drives. 4+Tb drives weren't available when the N40L was in it's heyday.

Typical performance for most user runs in the 40's, with spinning rust in the server and on the connected workstation.

 

[diskdiddler]

Is there anything else I can do to isolate the bottleneck?

I'm pretty sure I'm seeing 15% CPU usage when writing files at 25MB/s over CIFS - so it's not the CPU, right?
What about memory? Anything I can check there on the memory chart? Anything which would be indicative that it's definitely or definitely not memory?
Would you all like a copy of the graphs as it's writing or anything?

I'm 98% sure I saw 75MB/s writes initially, it was only noise that had me pissed. I solved the noise issue in an awkward way and then the speed issue occurred.

 

[diskdiddler]

Could this 4kb / 512b thing impact my performance?
https://www.icts.uiowa.edu/confluence/pages/viewpage.action?pageId=67307552

I'm about to produce some graphs for you guys.

 

[diskdiddler]

I'M BACK IN BUSINESS JERRY
(Those 2 were run 180 seconds apart, the issue appears to be a very odd switch issue)
Living my life like it's golden.

 

[joeschmuck]

Make sure you take into account all your hardware. The last throughput issue I helped someone troubleshoot turned out to be his main computer's internal laptop hard drive couldn't support the throughput, the FreeNAS system was running fine the entire time.

It's a bit late to bring this up but I wish everyone who has a network problem would simply make a direct connection to the computer and ensure the computer can handle the throughput vice blaming FreeNAS.

But I am glad you have located your problem.

 

[diskdiddler]

It's a bit late to bring this up but I wish everyone who has a network problem would simply make a direct connection to the computer and ensure the computer can handle the throughput vice blaming FreeNAS.

But I am glad you have located your problem.

Firstly.
The server going through a gigabit switch in the lounge, 80MB/s
Same server, same cable, going through a different gigabit switch, 20MB/s
Bear in mind here just how many people tried to claim it's RAM, CPU and what not. Those kinds of posts do not help anyone if you're trying to defend it operating at less than 1/3 the speed it's capable of as normal. I'd expect members here to have an /approximate/ idea of the performance FreeNAS can offer.

Anyhow now that it's all over with, time to delve deeper. 2 of my 6 disks have hit 57c unfortunately :/ it's within spec but not ideal. Anyone run disks around 55 to 65 for extended periods?

 

[gpsguy]

Do you have a fast workstation running a SSD?

The server going through a gigabit switch in the lounge, 80MB/s

40c is generally the target threshold for hard drive temperatures. At 55-65c they will probably die prematurely. But, who am i to tell you.

57c unfortunately :/ it's within spec but not ideal. Anyone run disks around 55 to 65 for extended periods?