Noob needs help on slow CIFS

[benderunit1]

I have setup a basic NAS server via CIFS share - starting out with 3x4TB hard drives. I put them in RaidZ1 to maximize some space. After some network troubleshooting, I got my transfer speed up towards a gigabit maximum of ~125MB/sec. That's better than I need, and things were running great. I quickly filled it all up, and moved forwards with my NAS project.

So I went into the GUI, expanded it with 3x4TB more (also in RaidZ1). The expansion was successful, all my old data was fine.
I was running two of the drives through a PCI sata card that was known to be good, and the last one in the MB.
The problem was - now my transfer speeds to the server are 350KB/sec or worse!
Hard drive total is less than 50% full.

I figured, the PCI card had to go. I got a 5gbps raid pcie raid card, put all three drives into it - and it really didn't help. Best case is now I see bursts at 500KB/sec.

I did a scrub and reboot - took just under 13 hours - no change in speed/behavior.

I can still watch movies (upwards of 14GB files) just fine. Large file transfer from server is a paltry 30MB/sec. But, still leaps ahead of my transfer to the server.

Common sense tells me a hard drive is bad. But, I used two of those hard drives to load onto the first array, and the other was at least at USB2.0 maximum ~20MB/sec (ripped it out of the external case - my likely culprit). But, maybe I'm missing something? Can I prove it using the shell fairly easily? Is expanding a known issue and I'm an idiot for trying to avoid two network drives?

Here are some system specs:
Supermicro X7DA3 motherboard (has Intel 82563EB Dual Port Gigabit)
16GB DDR2 ECC RAM
2x Intel Xeon X5355 @ 2.66GHz
Boot is 16GB SanDisk Cruzer
6x4TB (Share size 14TB)

I would not like to spend a fortune on this one, and preferably not lose the data transferred - I spent years collecting some stuff. I understand if I shot myself in the foot, but I am an engineer and like to debug before I just scrap it.

All suggestions are welcome, I will help as much as I can.

 

[gpsguy]

You can query the SMART information from the disks, using this command: smartctl -a /dev/adaX, where X is a number. Sometimes the device names might be /dev/daX. If you can SSH into the server with PuTTy, you could run the command on one or more of your drives and capture the output to a file.

If your aren't familiar with PuTTY/SSH, you could go into the shell and pipe the output to a file and then grab the file. For example: smartctl -a /dev/adaX >> filename I used the ">>", so if you do run the command against several drives, the output will end up in the same file.

If you post the results online in code tags (formatting is important), we can interpret the result for you.

 

[benderunit1]

Thanks for your time.
Attached.
Windows notepad makes this look like quite the blob of text - hard to decipher.
I'll try some programs that may be better viewers for my own learning.

Edit: notepad++ was definitely helpful

 

[jde]

I was running two of the drives through a PCI sata card that was known to be good

Your MB appears to have 6 chipset SATA ports as well as an integrated Adaptec sas controller with 8 ports. Is there a reason you aren't using these ports?, I would recommend connecting all 6 drives to the chipset SATA ports and see how it affects speed.

I got a 5gbps raid pcie raid card, put all three drives into it

What type of card did you get. Freenas doesn't generally play well with raid cards. Host Bus Adapters ("HBA") such as the IBM M1015 (flashed to IT mode) is a community favorite and is known to work with freenas. Again, given what's on your MB, it's unclear why you need an additional HBA as you have enough chipset SATA ports for 6 disks. I don't know if the integrated Adaptec sas controller is supported on freenas. Perhaps some of the more experienced forum members can answer that for you.

I quickly filled it all up, and moved forwards with my NAS project.

How full did your three drive pool get before you expanded? My understanding is that if it gets to the mid to high 90% range, it can really affect performance. Also, freenas does not automatically rebalance your vdevs upon expansion. You have to copy and paste your data to a new location on your pool and then delete the original to rebalance the data. Not sure if this is causing your problem, but it's something to think about.

X7DA3

Cyberjock will likely tell you your gear is too old and performance is going to be poor on a system with a front side bus. It may be that the memory is taxed more as pool size increases and that your system's limitations only became apparent as your pool size increased.

 

[SweetAndLow]

You need to run smart tests on your drives. Read the freenas manual to learn how.

 

[benderunit1]

It seemed like a great SATA port fit to me too until I discovered this motherboard bios only shows up 4 sata ports. I have read the other 2 are accessed in hardware raid modes only, and the manual states 6 through RAID specifically. I avoided that since I read zfs doesn't work well with hardware raids.
Maybe I can poke around the BIOS and there's a way around it that I missed earlier.

I don't have SAS on hand, that's pretty much the only reason for that one. Although if I'm replacing drives, it seems like an option now.

The expansion card is a Syba SI-PEX40064 PCI-Express 2.0 Low Profile Ready SATA III (6.0 Gb/s) Controller Card. It detects the hard drives through it, so it seems like it's working fine? I used this due to the above issue, and the reviews looked promising for the price.

My pool was 84% full prior to expansion, so that's definitely a factor. It seems kind of silly FreeNAS cannot shift the data upon expansion. I'm happy to move the data, but I don't really see a way to access them separately anymore. Maybe I can just delete data until it's equivalent?

While I certainly will agree that it's aging - performance was just fine before the expansion. With the expansion, I'm barely taxing the system when I look at the CPU/RAM resources.

I'll look into more in depth SMART tests.

 

[jde]

I don't have SAS on hand, that's pretty much the only reason for that one. Although if I'm replacing drives, it seems like an option now.

I believe that you can use a sas controller for SATA drives, just not the other way around. I think all you need is the SAS to SATA breakout cable. Although, like I said, I don't know if the the Adaptec chip is supported in freenas.

I'm happy to move the data, but I don't really see a way to access them separately anymore.

I'm not sure what you mean by access them separately. Do you have two pools or did you stripe you two vdevs together? Please post the output of spool status in code tags.

 

[gpsguy]

As I suspected, the OP has never run SMART tests on the drives.

Please run both short and long tests on each of the drives. Someone else can give you the syntax, I have to run.

 

[jde]

Someone else can give you the syntax, I have to run.

smartctl -t long /dev/adaX
Smartctl -t short /dev/adaX

 

[Robert Trevellyan]

It seems kind of silly FreeNAS cannot shift the data upon expansion.

ZFS will rebalance data that's added or changed. What it won't do is rebalance data at rest.