Reslivering 1.8m/s - GUI odd behavior

[Miniwehats]

OK where to start.... About 2 weeks ago I started to have odd performance issues, such as random cifs timeouts while browsing and slow transfers to the NAS and even moving files around internally also I torrent straight to the NAS and my torrent would throw errors saying the drive went offline. I started with my network and swapped out switches and cables and so forth and still same issue, I also noticed I was on 8.2 so i jumped up to 8.3 which didnt go to smooth, I did it through the GUI and it just threw an error each time and my next reboot I was magically on 8.3. I had a HDD fail today and put a new one in, I did not have any open slots so I offline the drive and shutdown the NAS and put the new one in and booted it up, since then it has been re slivering at about 1.8m/s and has something like 600h left!


SPECS:
AMD A4-3300 APU
Memory 15847MB
8 drives 2 Raidz pools 4 drives each.
Freenas installed on a 4gb flash Drive
Mobo - http://www.newegg.com/Product/Product.aspx?Item=N82E16813130641
Using onboard NIC. Recommend buying a separate intel nic?

ADA3 Failed today from SMART errors

So questions...

1. How to speed up resilvering?
2. was unable to detach the old drive (I think) from the GUI, GUI wont respond when trying to look at disks ATM.
3. All my reporting graphs such as CPU and memory dont show anything and have not for about a week, maybe my upgrade to 8.3 went weird?
4. should I wipe out my image of freenas and load a fresh one? What happens to my raid? backup my config and import it?
5. Also away from everything else I have a SSD laying around, can anyone point me to a guide on how to implament this to help speed up the NAS? I remember seeing somewhere that they are good for something and I cannot remember where.

Here is some outputs that might help.

 

[cyberjock]

If you choose to reload a fresh FreeNAS install all you have to do is reimport your config file and maybe import your zpool. Your zpool data is as safe as it is in its current form(assuming its safe.. I couldn't see your pictures very well). FreeNAS was designed to allow for replacement of failed/lost/stolen(yikes) boot drives as long as you have your config file.

Your pictures were significantly downsized by the forum :(.

If you shutdown and physically remove the bad drive you are trying to retire your resilvering should perform MUCH faster. I try not to resilver with a bad drive if I don't have to. That's why I use RAIDZ2 so I can remove a drive and still have redundancy from a second failure.

SSDs are good for L2ARC and ZIL. Depending on your server's use you may even see a performance loss with using it. I'd read up on both of those before you decide if you want to use it. Generally speaking, unless you are doing things that have tons of random reads and writes(often called sync writes) such as a database there isn't much to be gained from using them. If you are only streaming movies and viewing home pictures you shouldn't use either of those.

I often use the thumbrule "if you can't explain what a sync write is you probably don't want a ZIL". I'd say its probably true most of the time.

 

[Miniwehats]

Ah it was your guides I ran into before so thank you for responding! The failed drive is removed already, its reslivering the new drive. but going incredibly slow, I will post the outputs from putty but they dont show up very well. Looking at it it seems it still sees the old drive and the new drive even though only 1 is in that slot 0.o, i thought i was supposed to detach the drive in the GUI after but the GUI is being very weird. Can I reboot the nas during the reslivering process and will it resume?

Code:

[root@freenas] ~# systat -vm
    1 users    Load  0.20  0.31  0.28                  Jan 17 21:39

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
        Tot   Share      Tot    Share    Free           in   out     in   out
Act  205588   17328   828852    21356  12129k  count
All 3282700   18456 1074692k    30632          pages
Proc:                                                            Interrupts
  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        cow    5500 total
             47       15k    6  267 1504 1615             zfod        ehci0 ehci
                                                          ozfod     1 ohci0 ohci
11.2%Sys   1.5%Intr  0.0%User  0.0%Nice 87.3%Idle        %ozfod  1998 cpu0: time
|    |    |    |    |    |    |    |    |    |    |       daefr    10 re0 irq256
======                                                    prcfr  1159 ahci0 257
                                           dtbuf    42309 totfr   334 siis0 258
Namei     Name-cache   Dir-cache    325517 desvn          react  1998 cpu1: time
   Calls    hits   %    hits   %      4262 numvn          pdwak
       2       2 100                   983 frevn          pdpgs
                                                          intrn
Disks   md0   md1   md2  ada0  ada1  ada2  ada3   3374472 wire
KB/t   0.00  0.00  0.00  1.56  1.75 66.87 65.82    149540 act
tps       0     0     0     5     5   342   342     65488 inact
MB/s   0.00  0.00  0.00  0.01  0.01 22.34 21.99      1104 cache
%busy     0     0     0     0     0    51    74  12128844 free
                                                   173072 buf

[root@freenas] ~# zpool status -v
  pool: Vol0
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Nov 10 23:39:04 2012
        2.90T scanned out of 7.44T at 531K/s, (scan is slow, no estimated time)
        372G resilvered, 39.03% done
config:

        NAME                                              STATE     READ WRITE CKSUM
        Vol0                                              DEGRADED     0     0     0
          raidz1-0                                        ONLINE       0     0     0
            gptid/338243b8-e7fc-11e1-8afc-00045fb309e8    ONLINE       0     0     0
            gptid/33c3355c-e7fc-11e1-8afc-00045fb309e8    ONLINE       0     0     0
            gptid/342046fb-e7fc-11e1-8afc-00045fb309e8    ONLINE       0     0     0
            gptid/34a6ad54-e7fc-11e1-8afc-00045fb309e8    ONLINE       0     0     0
          raidz1-1                                        DEGRADED     0     0     0
            gptid/518cbece-e7fc-11e1-8afc-00045fb309e8    ONLINE       0     0     0
            replacing-1                                   OFFLINE      0     0     0
              2406282357351593836                         OFFLINE      0     0     0  was /dev/gptid/52120a83-e7fc-11e1-8afc-00045fb309e8
              gptid/7b92c455-2bca-11e2-ad71-00045fb309e8  ONLINE       0     0     0  (resilvering)
            gptid/5270dda3-e7fc-11e1-8afc-00045fb309e8    ONLINE       0     0     0
            gptid/52e8cd24-e7fc-11e1-8afc-00045fb309e8    ONLINE       0     0     0

errors: No known data errors

 

[cyberjock]

If you use the code tags then it'll be all pretty again.

I didn't see anything amazingly wrong. Not sure about the CPU performance(not an AMD person) but I'd still expect it to do much much more than what you are seeing performance-wise.

If you reboot FreeNAS the resilvering will start from zero. I'm not 100% sure it will start resilvering again automatically so I'd check it if you reboot and manually trigger a scrub if it doesn't.

I always recommend Intel NICs, but your motherboard doesn't exactly have alot of slots. If you want better network performance Intel is the leader in that category. But with your poor scrub performance that's certainly your limiting factor at the moment.

Few random ideas since I'm about to go to sleep:

1. Are you sure you are using the 64 bit version of FreeNAS? If not then you are using your zpool with 4GB of RAM. That would be a BIG performance hit.
2. Have you had any problems with any other drives? A failing disk can cause very poor performance.
3. I saw 6 SATA ports on that motherboard, so how are you installing 8 drives?
4. If you are running any kind of bittorrent expect that to kill scrub performance. Random reads and writes all over the place from a torrent is bad when a zpool when scrubbing. If you are doing anything that loads the server you may want to try disabling it temporarily. A very easy way is to simply disable FTP, CIFS, AFP as necessary to not share out your zpool.
5. Considering your problems with upgrading to 8.3 I'd probably reinstall FreeNAS and import your backed up config file. It may not fix anything, but it won't hurt. It could be your odd errors during the upgrade was from your USB drive failing.

 

[Miniwehats]

Man you rock, people like you make these forums awesome. I will add a intel nic soon they are cheap anyways. And I will also try to reboot the NAS and see what it does.

To answer your questions....
1. FreeNAS-8.3.0-RELEASE-x64 (r12701M) is the version that is currently running. Can i check it in shell?
2. I have not had any issues yet with failing drives, this drive seemed to start to have issues after the ugrade. Im going to check it on my other PC to see if its actually bad. I had a spare so thought I might as well toss it in.
3. Yes 6 sata onboard I have a 4 port Sata card installed as well the other drives run from. Is this a problem? I dont think I really set anything raid settings on either the card or the MOBO, just default i assume JBOD. Will check.....
4. Torrenting.... Well I would love to torrent to the NAS, would torrenting to a ISCSI Extent be any different? I do have a ISCSI mount to my server as well for a separate user account If not I guess I can Torrent to the internal drive in the server and dump it to the NAS after. - This very well could be my main issue as its a decently high traffic seed box. I will read up on adding my SSD maybe that can help if I continue with torrenting to the NAS.
5. I will check the USB drive and see if it has any issues, and probably format it and start fresh with freenas again.

NAS is mainly used as a media server / Seed box. Media server streams out to multiple devices at once such as ipads and phones + home streaming, so I guess that would put the reads up on the NAS.

Thanks again for your quick responses!

 

[cyberjock]

Man you rock, people like you make these forums awesome. I will add a intel nic soon they are cheap anyways. And I will also try to reboot the NAS and see what it does.

To answer your questions....
1. FreeNAS-8.3.0-RELEASE-x64 (r12701M) is the version that is currently running. Can i check it in shell?
2. I have not had any issues yet with failing drives, this drive seemed to start to have issues after the ugrade. Im going to check it on my other PC to see if its actually bad. I had a spare so thought I might as well toss it in.
3. Yes 6 sata onboard I have a 4 port Sata card installed as well the other drives run from. Is this a problem? I dont think I really set anything raid settings on either the card or the MOBO, just default i assume JBOD. Will check.....
4. Torrenting.... Well I would love to torrent to the NAS, would torrenting to a ISCSI Extent be any different? I do have a ISCSI mount to my server as well for a separate user account If not I guess I can Torrent to the internal drive in the server and dump it to the NAS after. - This very well could be my main issue as its a decently high traffic seed box. I will read up on adding my SSD maybe that can help if I continue with torrenting to the NAS.
5. I will check the USB drive and see if it has any issues, and probably format it and start fresh with freenas again.

Thanks again for your quick responses!

The GUI for version check is fine, you're good with that version.

It sounds like your SMART testing and reporting stuff is already setup in FreeNAS, but if it isn't you should set it up.

If the 4 port card is PCI, thats a horrible bottleneck. You're really limited by the PCI bus(133MB/sec) divided among all 4 drives, or roughly 33MB/sec per drive best case. That's pretty crappy.

Prior history in the forums with zfs and iscsi says just don't do it. One of the forum guys has found that UFS is far better for iscsi because of ZFS's copy-on-write.

Personally I never do torrents directly to ZFS. I think the copy-on-write would cause high fragmentation that can't be corrected since there is no defrag option. But that's just my opinion. For seeding its fine, but not when actually downloading.

If you have a spare USB drive you could also just install to that and then you'll have the old USB if you really need it.

 

[Miniwehats]

Lol man I guess I built this NAS a bit wrong, well I will get a NIC and a different SATA card since it is PCI and I did not even think about the bus issue. Its just about as much to get a new MOBO with 8 slots than get a PCI express Sata card so I will see which way I go. The reslivering is down to 1m/s now after a reboot so not sure what to do with it. I kinda want to just blow out the USB and start fresh and load in my old config, doing that should i put the old (apparently failed) drive back in when I reload the config since when I exported it had that drive? I dont really want to export the current running config because the GUI is acting super strange. I trust the older one i exported a few weeks ago right after before the upgrade. I have a spare USB as well so ill do it on that first and see how things run and fix my terrible build issues. Thanks again man you rock, now go to bed.

 

[cyberjock]

Well, if you've rebooted already I'd try starting fresh with a new USB stick and load the old config. Considering you are using PCI I'd think you should get 10-15MB/sec when resilvering, but that's only a guess.

I have a USB stick that has passed 3 write/read/verify tests but it doesn't work right for FreeNAS. The system has strange performance issues. As soon as I switched to Corsair the problem was gone. Not sure if its just chance but I've had great results with Corsair and poor results with Adata.

If your configuration is easy to setup if you didn't have a config file I'd probably just set it up again to rule out some screwed up setting in the config file as the problem. Probably isn't the cause, but it will definitely rule out that possibility.

But splurge for a new controller that's PCIexpress. You'll be glad you did!

 

[Miniwehats]

Yea I ended up ordering a new Mobo, Nic and Corsair USB drive. Hopefully this should all work better, I think that PCI card has been the culprit for a while lol. The slivering finally finished and I was able to get the old drive finally detached, GUI still wont show up correct so im going to wipe it all out and use the new flash drive when it all comes. So not importing my config.... can I somehow import the volumes I have already made? Will it just see them in volume manager? Im guessing I can find that in the docs for 8.3. But anyways thank you again man you rock, I always see your posts on these forums and I know the community appreciates it. Oh also moving my torrents to a internal drive on the torrent server then ill just have it dump over to the NAS when its done. Thanks again!

 

[cyberjock]

All you have to do to reimport your old zpool is use the import options in FreeNAS. It will find all the drives and create your zpool again with all of your data. That's one of the great things about ZFS.