Performance assistance please !

[willhorner]

I am only getting write performance of 20 - 30MB/S copy from 1 cif share to another.
system is I5 - 2400
32 gigs RAM
5 x 3 TB - in 1 raidz1 vol
8 x 1.5 TB - 1 raidz1 vol
7 x 1.5 TB - in 1 raidz1 vol

gig network card and switch


i'm sure there's a ton of other info you need , but i am truly a noob and i'm not even sure what else to provide.
out of respect for your time i've searched a number of the forums. the result of which was upgrading from 16 to 32 gigs of ram mentioned above. upgrading from a q6600 to the I5. some of the recommened tweeks were beyond my knowledge level. Before i installed freenas I ran server 2008 with raid 5 and regularly got 80+megs per second from these drives. More disturbing than the xfer rate is the fact that when browsing the disks video streams will freeze up, which bandwidth-wise shouldn't even come close to being an issue. Ill provide any info you need, and appreciate any help you can give.

 

[cyberjock]

I'd NEVER do more than 4 drives in a RAIDZ1. Those should be RAIDZ2. If you have a single disk failure and any other disk has even one bad sector you will lose data.. hence the need for a RAIDZ2.

If your system is freezing up, that may be a sign of a hardware issue or a failing drive. Have you examined all of the SMART data on your drives? If one disk is having issues reading a sector it can "freeze" the zpool up pretty badly. TLER can prevent the freezing, but bad sectors can ruin a zpool's performance.

Your hardware is spec'd higher than mine, so unless you've implemented some tweaks that aren't smart(99% of people that add tweaks have them wrong) you definitely have a hardware issue. My first guess would be a failing disk drive. If you don't know what settings do you shouldn't be checking/unchecking boxes in the settings either. I left everything at defaults and I get over 100MB/sec on all of my servers.

BTW, trying to compare your RAID5 performance to a ZFS pool isn't a "like for like". ZFS is much more CPU intensive and works totally different. You can achieve great RAID5 performance in Windows and have a zpool that can't get 50MB/sec because of your CPU and RAM.

 

[jgreco]

I'd NEVER do more than 4 drives in a RAIDZ1. Those should be RAIDZ2. If you have a single disk failure and any other disk has even one bad sector you will lose data.. hence the need for a RAIDZ2.

That's not quite correct. If you have a single disk failure and any other disk has bad sectors that were unused, you're still fine. However, if the sector was in use, then you've got data loss. Of course, you can't rely on that... errors are bad and need to be addressed.

 

[willhorner]

well, I appreciate the feedback, and any and all help is appreciated, but data integrity isn't really a concern right now. I have all the data backed up off line so data loss from failure isn't really a concern. If disk errors cause the loss of speed then that is a great pointer, although i'm not sure how a disk failure could cause all 3 arrays to have the same poor performance unless each array has a failing or damaged drive. ill check that asap. I don't expect the drives to be as fast as under straight raid, but they should be faster than 30 MB/S right, i've seen people reporting speeds close to 3 times that from far lower spec'ed setups. I have to have something misconfgured somewhere.

 

[cyberjock]

Yeah, I'd expect that you'd be able to saturate multiple Gb LAN ports simultaneously. You way overspec me and I can saturate 2xGb LAN simultaneously.

 

[gpsguy]

How are your disks connected to the server? You mention that you'd been using RAID5 before - are these disks still connected to a hardware RAID controller?

Tell us what version of FreeNAS that you're running. What does "zpool status -v" say?

 

[willhorner]

they are connected through non-raid pci-e 4 port sata controllers.
freenas ver. 8.3.1

pool: media1
state: ONLINE
scan: scrub repaired 0 in 21h4m with 0 errors on Mon Nov 5 06:47:55 2012
config:

NAME STATE READ WRITE CKS
UM
media1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/04458c40-2113-11e2-b4e3-00223fe1572b ONLINE 0 0 0
gptid/04a3b8c3-2113-11e2-b4e3-00223fe1572b ONLINE 0 0 0
gptid/0508b559-2113-11e2-b4e3-00223fe1572b ONLINE 0 0 0
gptid/05922943-2113-11e2-b4e3-00223fe1572b ONLINE 0 0 0
gptid/0600f261-2113-11e2-b4e3-00223fe1572b ONLINE 0 0 0
gptid/55835b67-2590-11e2-9b2b-00223fe1572b ONLINE 0 0 0

errors: No known data errors

pool: media2
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKS
UM
media2 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/fc5fb266-9b42-11e2-afeb-00e04c86425e ONLINE 0 0 0
gptid/fcbd758e-9b42-11e2-afeb-00e04c86425e ONLINE 0 0 0
gptid/fd16c3dc-9b42-11e2-afeb-00e04c86425e ONLINE 0 0 0
gptid/fd6e7cbb-9b42-11e2-afeb-00e04c86425e ONLINE 0 0 0
gptid/fdc00267-9b42-11e2-afeb-00e04c86425e ONLINE 0 0 0
gptid/fe115c5a-9b42-11e2-afeb-00e04c86425e ONLINE 0 0 0
gptid/fe640ffc-9b42-11e2-afeb-00e04c86425e ONLINE 0 0 0
gptid/ff258182-9b42-11e2-afeb-00e04c86425e ONLINE 0 0 0

errors: No known data errors

pool: media3
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKS
UM
media3 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/f40f76e1-24aa-11e2-a28a-00223fe1572b ONLINE 0 0 0
gptid/f4e7a2aa-24aa-11e2-a28a-00223fe1572b ONLINE 0 0 0
gptid/f5ba9fd9-24aa-11e2-a28a-00223fe1572b ONLINE 0 0 0
gptid/f67c3b58-24aa-11e2-a28a-00223fe1572b ONLINE 0 0 0
gptid/f724f0c3-24aa-11e2-a28a-00223fe1572b ONLINE 0 0 0

errors: No known data errors

 

[cyberjock]

That's not quite correct. If you have a single disk failure and any other disk has bad sectors that were unused, you're still fine. However, if the sector was in use, then you've got data loss. Of course, you can't rely on that... errors are bad and need to be addressed.

But isn't it a foregone conclusion that if the sector has no data then it won't be part of the resilvering, so you wouldn't even know if it was bad(or maybe even care at that particular moment)? Personally, I wouldn't care at that moment as I wouldn't want to see my one good drive drop off the system because it couldn't read an empty sector. I'd much prefer that it not read the empty sectors so I don't have the opportunity to see bad sectors.

I think that the best way(which I don't consider to be particularly good) to check all of the sectors on the disk is a long SMART test. But since it's not a write/read/verify I don't put much faith in it.

 

[survive]

Hi willhorner,

Side question for you......

Are you scrubbing your pools regularly?

I see one reports that it was scanned back in Nov '12 and the other 2 don't report any scrubs at all.

-Will

 

[willhorner]

no, and i have no idea how to! :) I've tried to set them up but it didn't ever seem to run. i figured at some point id read up and figure it out, but i never did.

 

[jgreco]

But isn't it a foregone conclusion that if the sector has no data then it won't be part of the resilvering, so you wouldn't even know if it was bad(or maybe even care at that particular moment)? Personally, I wouldn't care at that moment as I wouldn't want to see my one good drive drop off the system because it couldn't read an empty sector. I'd much prefer that it not read the empty sectors so I don't have the opportunity to see bad sectors.

If you happened to be unfortunate enough to have every other sector allocated, would it be more efficient to issue a single read of a large stretch of them and then just ignore every other block, or issue thousands of individual read requests to make sure you don't inadvertently read a free block?

I think that the best way(which I don't consider to be particularly good) to check all of the sectors on the disk is a long SMART test. But since it's not a write/read/verify I don't put much faith in it.

Yeah, it gets kinda dicey. Many modern RAID controllers (can) do something called a "Patrol Read" which is basically walking all the blocks on the disk while the disk is idle. Back in the late '90's, this was tried in FreeBSD with diskcheckd (part of the base system for awhile, then made a port), I think phk wrote it after a discussion some of us had. It was basically a simple program that would read a small (64KB) block of data every N seconds, with the intent of reading a disk entirely every few days. We were putting system images on a 1GB disk at the time, and those could be read every two days or thereabouts. However, as disk sizes exploded up towards 73GB, there was a multifaceted problem: you could keep reading 64KB a second but it would take two months to make one pass. Or you could read more frequently but you were cutting significantly into IOPS capacity of the drive. Or you could read a larger block, but it'd have to be like one megabyte, every second or two, which was cutting significantly into the bandwidth capacity of the drive. And diskcheckd was basically a userland program, it employed no particular intelligence about system loading and made no efforts to only scan when the system wasn't busy.

So ZFS scrubs serve a somewhat similar purpose, but they don't read the empty space on a disk. And of course none of this does a read/write/verify cycle.

 

[cyberjock]

If you happened to be unfortunate enough to have every other sector allocated, would it be more efficient to issue a single read of a large stretch of them and then just ignore every other block, or issue thousands of individual read requests to make sure you don't inadvertently read a free block?

BUt that's not how a ZFS scrub works. When a scrub is initiated it traverses the directories and files in the order that they are on the zpool directory listing. Basically older directories first and newer directories last. It's far from the most efficient way of doing business, but it does have the virtue of not requiring a free space bitmap or other method of tracking ultra large systems and you can always tell where you are and where you were. Any file added "new" to a directory that was previously scrubbed through in a scrub that is in progress has the virtue of being "trusted" because it was just written and won't be scrubbed, but I'd have to find my reference again. If I find the place where I read about scrubs in exquisite detail I'll post it. There was talk back in the Sun days of allowing you to give priority to data in a scrub or replication task to allow you to restore the most commonly used data first, then the task continuing with the "lower priority data".

So ZFS scrubs serve a somewhat similar purpose, but they don't read the empty space on a disk. And of course none of this does a read/write/verify cycle.

But, if you think about it in the big picture, I consider a ZFS scrub to be "as good" as a write/read/verify cycle since you are reading the data, comparing it to the checksums and redundancy of your zpool, and then correcting any errors.

 

[jgreco]

BUt that's not how a ZFS scrub works. When a scrub is initiated it traverses the directories and files in the order that they are on the zpool directory listing.

That wasn't exactly the point; ZFS does some speculative I/O. So there's this thing called the "software track buffer" (this document calls it a "soft track buffer" and now most people refer to it as the vdev cache ... Sun obtuseness for the win!) and so ZFS tends not to read single blocks, but instead 64KB chunks. So in the situation I suggested above, where there is data in every other sector, ZFS will read 64KB and will scoop up 32KB worth of "used" sectors and 32KB of "free" sectors. Free sectors adjacent to used sectors might be subject to being read as part of a scrub.

To get a handle on at least some of this, if an I/O request comes in that is vfs.zfs.vdev.cache.max (default: 16KB) or less, then the I/O request will be speculatively increased to 2^vfs.zfs.vdev.cache.bshift (default: 16). There's a comment in the code that suggests this is limited to metadata, but I don't see how that restriction is implemented - I know there was some concern about the bandwidth available in I/O subsystems, so it might be limited. Stats are gathered and reported under kstat.zfs.misc.vdev_cache_stats.*

 

[cyberjock]

That wasn't exactly the point; ZFS does some speculative I/O. So there's this thing called the "software track buffer" (this document calls it a "soft track buffer" and now most people refer to it as the vdev cache ... Sun obtuseness for the win!) and so ZFS tends not to read single blocks, but instead 64KB chunks. So in the situation I suggested above, where there is data in every other sector, ZFS will read 64KB and will scoop up 32KB worth of "used" sectors and 32KB of "free" sectors. Free sectors adjacent to used sectors might be subject to being read as part of a scrub.

To get a handle on at least some of this, if an I/O request comes in that is vfs.zfs.vdev.cache.max (default: 16KB) or less, then the I/O request will be speculatively increased to 2^vfs.zfs.vdev.cache.bshift (default: 16). There's a comment in the code that suggests this is limited to metadata, but I don't see how that restriction is implemented - I know there was some concern about the bandwidth available in I/O subsystems, so it might be limited. Stats are gathered and reported under kstat.zfs.misc.vdev_cache_stats.*

Ok, I see what you are saying. I was only saying that if there is even one bad sector for simplification. For the average user the point I was trying to make I think was made effectively. If you have no more redundancy and can't read your data you will have problems to deal with. :P

 

[jgreco]

In the case of a bad sector, it is likely that the affected data will be unrecoverable, but the remainder of the data in the pool should be okay. So it seems like the significant question is, for someone in the uncomfortable position of having less-than-enough redundancy, how much data loss will there be? ZFS appears to implement aggressive strategies to cope with this, but how thoroughly this has actually been tested, I don't really know.

Fun could be had if you have a spare FreeNAS VM by writing random zero sectors onto a disk that is part of a ZFS pool and seeing what happens. One would expect that zeroed blocks would function similarly to unreadable blocks, given the ZFS block checksum feature.

 

[cyberjock]

Fun could be had if you have a spare FreeNAS VM by writing random zero sectors onto a disk that is part of a ZFS pool and seeing what happens. One would expect that zeroed blocks would function similarly to unreadable blocks, given the ZFS block checksum feature.

They pretty much do(but don't). I tested this in a VM. I made a 3 disk RAIDZ1. Then wrote something like 500MB of zeros. If you do a scrub you rack up a boatload of errors(no surprises there). Any new scrub seems to be fine(no surprises there either). But, if you have bad sectors then the hard drive sometimes drops from the zpool(which sucks because it definitely won't verify any further). If the disk controller supports hotswap and the drive comes back online later and the scrub finishes and you don't look closely you might not even realize anything is wrong. There will be read errors on 1 disk if you do a zpool status but if you reboot they'll all be zeroed out and you'll have no evidence of anything being wrong.

Personally, I don't like the idea of hotswap drives. If a drive drops out I want it to stay dropped out so it gets my attention.

 

[titan_rw]

no, and i have no idea how to! :) I've tried to set them up but it didn't ever seem to run. i figured at some point id read up and figure it out, but i never did.

You might want to scrub your pools.

This should be automated through the 'zfs scrubs' section of the gui, but for the first one, I'd run it manually.

It'll also give you some idea as to how the pool(s) perform.

Run "zpool scrub media1", and monitor scrub with "zpool status media1". Wait for it to finish, and check for any errors. Then go on to the next pool.

I'd be curious to know if the scrubs report any errors. Media2 and media3 have never been scrubbed. How old are the pools? Media1 was last scrubbed quite a long time ago. Bitrot is real, and with only a single drive of parity, you don't have much redundancy to counter it.

 

[cyberjock]

Whoa. I missed the post about the scrubs.

Scrubs should be performed anywhere from weekly to monthly depending on various factors. The fact that you didn't do scrubs is fairly risky for your data. It's not the end of the world if nothing breaks(but things do break, which is why scrubs are needed).

 

[willhorner]

ok, all 3 scrubbed. no errors.

 

[willhorner]

Any suggestions at all are welcome!