Looking to improve 70MB/s read speed over SMB/CIFS

[palmboy5]

I have a gigabit network but have only been able to get up to 70MB/s sequential read from the server using CrystalDiskMark on my desktop. Copying a large file to the desktop SSD is worse, at less than 60MB/s. However, write speed with the CrystalDiskMark program gets up to 110MB/s, so I think the CPU, NIC, and other hardware are adequate.

Server specs:
AMD Phenom II X2 560 3.3GHz
Gigabyte GA-78LMT-S2P Motherboard
2x4GB DDR3
IBM M1015 IT-mode
Intel EXPI9301CTBLK NIC
RAIDZ2 8x 2TB Green
FreeNAS 8.2.0 RELEASE x64


dd benchmark on server:

Code:

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
261117219 bytes/sec
dd of=/dev/zero if=tmp.dat bs=2048k count=50k
494663520 bytes/sec

Desktop specs:
Intel Core i7 2600K @ 4.5GHz OC
ASUS P8Z68-V Pro Motherboard
4x4GB DDR3
Intel EXPI9301CTBLK NIC
Crucial M4 128GB, WD 2TB Green
Windows 7 Professional x64

I have tried enabling autotune, but the only noticeable change is that it somehow decides that over 6GB of RAM should remain unused at all times... (versus about 1GB) and with this setting I can't even write multiple GB's of data (at no more than 20MB/s from a SD card) and stream a video from the server at the same time. Eventually it just becomes a lag-fest.

I have also noticed that (from the desktop) opening a folder with a few thousand pictures will take 10 seconds or more for it to list all the files. This didn't happen when I ran with FreeBSD itself instead of FreeNAS and has always happened since I started using FreeNAS.

Anyway, I'd like to improve the performance and any help will be appreciated!

 

[gianpy69]

dd benchmark on server:

Code:

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
261117219 bytes/sec
dd of=/dev/zero if=tmp.dat bs=2048k count=50k
494663520 bytes/sec

The above datas are very high. You read 261MBytes/s and write 494 MB/s. Do you have SSD on your freenas? Magnetic hard disk cannot do that speed. I think that you did dd not the test an hard disk. You have to change directory on your right mount point and redo the test (may be cd /mnt/"somethings").

write multiple GB's of data (at no more than 20MB/s from a SD card)

SD card is bandwidth limited from usb (480Mbit) so 20MB/s is the right value. You have test from hard disk or better from a ssd to/from freenas.

 

[palmboy5]

The above datas are very high. You read 261MBytes/s and write 494 MB/s. Do you have SSD on your freenas? Magnetic hard disk cannot do that speed. I think that you did dd not the test an hard disk. You have to change directory on your right mount point and redo the test (may be cd /mnt/"somethings").

It is a RAIDZ2 (RAID6) array of 8 hard disks.

SD card is bandwidth limited from usb (480Mbit) so 20MB/s is the right value. You have test from hard disk or better from a ssd to/from freenas.

My example with the SD card is showing that I am not overusing the server at all, but the server is still unable to stream a video at the same time.

 

[yaneurabeya]

Those dds aren't blowing out the drive cache. You'll need to up the bs argument to something more reasonable.

 

[palmboy5]

Those dds aren't blowing out the drive cache. You'll need to up the bs argument to something more reasonable.

You have to consider the count as well, it is reading/writing 100GiB of data. I took the command from here: http://forums.freenas.org/showthread.php?981-Notes-on-Performance-Benchmarks-and-Cache

 

[yaneurabeya]

You have to consider the count as well, it is reading/writing 100GiB of data. I took the command from here: http://forums.freenas.org/showthread.php?981-Notes-on-Performance-Benchmarks-and-Cache

Ok. You guys are looking for not looking for worst case numbers. Nm I guess..

 

[palmboy5]

Ok. You guys are looking for not looking for worst case numbers. Nm I guess..

Say I was looking for worst case numbers, what was your point?

I have a server that I know is capable of doing 471MiB/s read and it is serving me an 8GB file at 55MiB/s through CIFS, to an SSD that can write at 200MiB/s. This is a basic file copy with nothing else happening, a sequential process, the "best case".

If I copy three different files simultaneously from the server to the SSD, I can reliably reach over 110MiB/s. So, my hardware is fully capable of maxing out gigabit ethernet. I want it to do that same speed with just a single file.

 

[William Grzybowski]

CIFS uses samba which is a single-threaded implementation. Thats is why you can saturate line with multiple files.

You might get better results "tuning" CIFS, there are some socket options and AIO buffer size that _might_ help. Do a search on the forum for those subjects.

 

[palmboy5]

CIFS uses samba which is a single-threaded implementation. Thats is why you can saturate line with multiple files.

You might get better results "tuning" CIFS, there are some socket options and AIO buffer size that _might_ help. Do a search on the forum for those subjects.

I had acted on the suspicion of the CPU simply not being fast enough in single-threaded performance and got a Core i3 2125 that is 41% and 30% faster in singlethreaded Cinebench R10 and R11.5, respectively. Even so, my throughput with that CPU was virtually identical. But maybe Cinebench isn't the most relevant to the task at hand, so I am open to suggestions for a different benchmarking program.

 

[mattlach]

The above datas are very high. You read 261MBytes/s and write 494 MB/s.

Firstly, thats backwards. Secondly, you forgot your 1024 vs 1000's.

He's writing at 249MB/s and reading at 472MB/s

Do you have SSD on your freenas? Magnetic hard disk cannot do that speed. I think that you did dd not the test an hard disk. You have to change directory on your right mount point and redo the test (may be cd /mnt/"somethings").

Incorrect again. With many drives in an array, sequential read and write speeds can be very high.

More spinning disks usually doesn't help seek times as much though, so for transfers of many small files the results are typically significantly lower (and SSD L2ARC and ZIL's may help more)

So this is definitely inline with what is possible with modern drives and large arrays.

I have 6 WD greens in mine running RAIDz2, and I'm reading at 436MB/s and writing at 295MB/s

SD card is bandwidth limited from usb (480Mbit) so 20MB/s is the right value. You have test from hard disk or better from a ssd to/from freenas.

Here you are correct (provided its USB 2.0 and not USB 3.0). USB 2.0 does support 480Mbit/s, which if translated into byts is 60MB/s, BUT there is significant overhead when it comes to storage over USB, so typical transfer speeds are still ~21MB/s max. With USB 3.0 (both computer and USB drive must support it) speeds can be higher.

 

[palmboy5]

Firstly, thats backwards. Secondly, you forgot your 1024 vs 1000's.

He's writing at 249MB/s and reading at 472MB/s

His values are backwards but technically, 261MB/s and 494MB/s are correct. Tera/giga/mega/kilo are all units that go by 1000's. If you wanted to go by 1024's, it would be Tebi/gibi/mebi/kibi. Take a look at this: http://en.wikipedia.org/wiki/Mebibyte

I fully approved of FreeNAS actually giving units in the correct "bi" unit on the web GUI (check it out). I wish Windows would use the correct units as well. Technically, hard drive manufacturers are the ones using the correct unit on their drives and Windows is misrepresenting all of them by saying giga instead of gibi.

 

[mattlach]

His values are backwards but technically, 261MB/s and 494MB/s are correct. Tera/giga/mega/kilo are all units that go by 1000's. If you wanted to go by 1024's, it would be Tebi/gibi/mebi/kibi. Take a look at this: http://en.wikipedia.org/wiki/Mebibyte

I fully approved of FreeNAS actually giving units in the correct "bi" unit on the web GUI (check it out). I wish Windows would use the correct units as well. Technically, hard drive manufacturers are the ones using the correct unit on their drives and Windows is misrepresenting all of them by saying giga instead of gibi.

I know the ISO standard (or was it IEC?) came out in 1998, and I still couldn't disagree more with it.

The multipliers in the computer world have always been binary (2^10, or 1024) and all of a sudden overturning decades of precedent by moving to a 10 base system is absolutely dumbfoundingly stupid.

The only people who embraced this new naming convention were the drive manufacturers, who love to make something sound bigger than it is for marketing/sales purposes. I have always suspected that the individuals involved in the authoring and approval of the ISO standard either were paid off by, or had conflict of interest or some sort of involvement with the drive manufacturers.

Luckily at least memory manufacturers have not adopted this nomenclature.

Eitherway, the way it has evolved in common use today is that when it comes to drives and storage hardware, 1000 is kilo, whereas when it comes to RAM and software representation of storage 1024 is kilo, except for Apple, which has decided to adopt 1000 as kilo for storage in software as well as hardware.

To me, when it comes to computing kilo is and will always be 1024, and I only reluctantly agree to call 1000 kilo when referring to bare uninstalled hard drives and other storage hardware labeled as such. This is the way it has always been and I utterly refuse to call it anything else. Over my dead body.

 

[palmboy5]

Why is it that kilo, which has meant 1000 since the beginning of the metric system, suddenly has to mean 1024 just for computers? 1024 is not 1000, therefore use a different unit nomenclature.

 

[dfsooner]

1024 is a power of 2 and therefore fits the binary paradigm much better.

 

[cyberjock]

Computers don't count in base-10 like humans do. Since they use base-2 you can't easily get to 1000 without going slightly over to 1024. In computer terms, a kilo-whatever has always meant 1024 until recently.

 

[palmboy5]

Yes binary computers don't use base-10, so don't use base-10 nomenclature. Solution: kibi instead of kilo.

EDIT: Do any of you have a better argument for kilo = 1024 in computers than that it "has always meant 1024"? All I'm seeing is a stubbornness to stick with an incorrect unit usage.

 

[cyberjock]

EDIT: Do any of you have a better argument for kilo = 1024 in computers than that it "has always meant 1024"? All I'm seeing is a stubbornness to stick with an incorrect unit usage.

It's 2^10 bytes. Its equivalent to 1111111111 in binary. What more would there be. The reason for 1024 is strictly technical based on binary. Nothing more. When computers had 640kbytes of RAM, that extra 24 bytes was so small as to be insignificant. Now that we are in TB, that's a HUGE amount of space.

RAM will almost certainly always be in 1024 because it HAS to be. You can't start counting RAM up and then part-way through binary counting skip from 1111101000 to 10000000000. That'll confuse the heck out of ALOT of things. But hard drives, while they do count in base 10, it's easy to count in ANY base as long as you stop counting at the end. You could also use base-8 for hard drives if you wanted. RAM doesn't work the same way however.

I understand your feeling about kilo not being 1000 with computers, but it's just the nature of the beast. If someday a new type of computer comes out that doesn't work in base-2 and is base-10 or something that multiplies/divides into base-10 then we may see kilo=1000 in computers. I'm not sure I expect that in my lifetime however.

Edit: You want to know why I think we've invented this "kibibyte" and "kilobyte"? I think it's because some people are just too dumb to realize that a kilobyte in computer terms isn't 1000, and instead of going with the industry they want the industry to change for them. This dumbing down is through society, and is quite sad IMO.

 

[palmboy5]

My issue is that kilo should never have been adapted to mean 1024 just for binary computers. It is plain and simply wrong.

When computers were in their infancy, there was no "-bi" standard to use, so they went with the closest preexisting standard. I understand that. Not that it mattered back then, as you said: "When computers had 640kbytes of RAM, that extra 24 bytes was so small as to be insignificant." This was a bad choice, just like choosing to represent the year as a two-digit number. ASCII only supporting the English language also comes to mind.

Bad decisions like that were completely short-sighted, but fortunately there are fixes. Of the three I listed, a new standard for base-2 ("-bi") was created, the transition for Y2K, and Unicode. Why for the case of kilo vs kibi, must the old and wrong way still be used?

EDIT: I think I need to make something clear because it keeps getting repeated to me. I understand what binary is and I understand why binary computer storage doesn't go by base-10. That isn't the POINT. I am NOT saying computers should start counting in base-10, I AM saying to stop representing base-2 with base-10 terminology. Base-2 terminology exists as tebi/gibi/mebi/kibi, etc. USE THAT. Why anyone would reach a different conclusion baffles me.

 

[cyberjock]

Hehe. A lot of people, like myself, use the old terminology because that's what we've always used. And although the ISO standard(or whatever it is) recognizes 'kigi, mebi, etc' alot of OSes still don't. Windows uses Kilobyte and does the true 1024 calculation. Apple uses Kilobyte and uses 1000 calculation. Hard drives are sold in gigabytes and terabytes, but they are neither of those. So who is more screwed up? Windows for not using 1000 so that 1TB drive looks like a 1TB partition, Apple for using Kilobyte but then using 1000, or the hard drive manufacturers for calling it TB when it isn't?

I really see this as an uphill battle for the "bi's" because sectors, RAM, and other places are still in binary no matter what we want to do. A sector is now 4KB, that's 4096 bytes for all of us, and while in a lot of situations byte versus bibytes doesn't matter a whole lot, there are some places where it can make a huge different. If ZFS stripes were 128x10^3 bytes we'd have a whole series of performance killing problems for ZFS. Our ZFS thumbrules for RAIDZ1 and RAIDZ2 depend on 1 KB being exactly 1024 bytes. Do the math wrong and all sorts of nasty stuff comes into play.

This will be a VERY long fight for years to come because consumers will be even more confused. They're going to freak if Windows says they have 1.2Tebibytes free and a game says it needs 500Gigabytes. ZOMG what is someone to do! To you and me, we'll say it's no big deal because we'll still have :a lot" of free space. But to a lot of people(like my mom for instance) she'll question whether it'll be "compatible" and if she "has enough free space" after you convert Gigabytes to Tebibytes. Then she'll ask why we can't ALL use 1 metric. RAM, will always be in MB, GB, etc. Hard drives, at least for the forseeable future, are sold in Mibibytes, Gibibytes. But they are marketed in terabytes and gigabytes on the box unless you look at the fine print. Neither of those will change, so there will always be a conflict.

The crappy thing for me, as an engineer, is that I really like K being 1000 and M being 1x10^6. If someone says they need 1MW of power I know that is 1x10^6, NOT some convoluted 1024 derivative. I'd love to see computers go to calculations where you just move the decimal. But like I said before, it's not likely in my lifetime.

If I had to blame this whole fiasco on something, I'd blame the hard drive manufacturers for short changing us and trying to remold the industry to fit their definitions. I can't sell you tin and call it gold. Then when I get caught scamming everyone on the planet I try to get the definition of gold changed to include my special "tin" and call it "gold II", but then box it as "gold" with a small print note that it isn't real "gold". Frankly, as Mattach said, someone had to have paid someone off or had a reason to profit from calling hard drives TB but then using Tebibyte. Everyone seems to be inconsistent in their use of the proper terms.

And until the hard drive manufacturers start calling their hard drive capacity by their own standard that they paid for they can eat shit and die. I can't believe the hard drive manufacturers bought, lobbied, and paid for their own standard and then didn't follow it because they didn't want to inconvenience the little folk that would be more confused. I'll still call everything TB and GB and leave it up to the person I'm listening to or talking to to realize if those extra 24 bytes really matter.

Good discussion palmboy5, but I have the opposite argument you do. I think that computer terms should stay in KB and MB until the industry really does adopt the "bibytes" unit of measurement. If only because that is what we have always used. Changing an industry standard is hard and will only cause more confusion later on down the road. There are so many engineering problems created from standards coming and going over the last 30+ years in various electronic fields that it really makes a mess. Sometimes you don't know what standard you are in! Sometimes the company that built the stuff for you custom 30 years ago doesn't even know, but we paid millions of dollars for it!

Think of it like this. If tomorrow everyone on the planet started using bibytes for binary bytes and kilobytes for base-10, how long until someone screws up a conversion and forgets that at one time bibytes was equal to kilobytes. And when someone asks the question a bunch of people in a room making lots of money will be asking themselves if a kilobyte was 1024 bytes or was 1000? It'll be as awesome as NASA's conversion from metric to imperial standard that desimated a satellite. But will lives be lost due to the mistake?

I'd love to hear you defend your viewpoint just for curiosity. Not to start an argument, but it's always interesting to listen to the other side of the coin. You never know, you make make me a convert.


-----

This isn't much different from the IPv4 to IPv6 fiasco. IPv4 was limited to roughly 4.3 billion IPs because the guy that was designing it said "that's really big.. I doubt this will really matter later on anyway". There's a video on Youtube of the guy pretty much saying that was why he made the decision that he did. VERY few people ever think big enough to really plan for the whole planet the first time around.

 

[palmboy5]

An ideal in this messy world, but my viewpoint is simple and absolute:
Kilo = 1000, Kibi = 1024. Use the correct notation everywhere.

Hehe. A lot of people, like myself, use the old terminology because that's what we've always used. And although the ISO standard(or whatever it is) recognizes 'kigi, mebi, etc' alot of OSes still don't. Windows uses Kilobyte and does the true 1024 calculation. Apple uses Kilobyte and uses 1000 calculation. Hard drives are sold in gigabytes and terabytes, but they are neither of those. So who is more screwed up? Windows for not using 1000 so that 1TB drive looks like a 1TB partition, Apple for using Kilobyte but then using 1000, or the hard drive manufacturers for calling it TB when it isn't?

I'm ok with hard drives not being exactly what they advertise to be since they always end up a bit more than advertised. Almost 400MB over exactly 2TB on my server's drives? Fine by me, give me less than advertised then I will have a problem. Windows is wrong but should keep the 1024 calculation and use "bi" units. Apple is right for using 1000 correctly with kilo. Ideally, they should all use 1024 and use "bi" units.

Your ZFS paragraph seems to be arguing that mistaking the notation could lead to massive performance problems, which is totally correct but still won't happen if correct notation is used everywhere.

If I had to blame this whole fiasco on something, I'd blame the hard drive manufacturers for short changing us and trying to remold the industry to fit their definitions. I can't sell you tin and call it gold. Then when I get caught scamming everyone on the planet I try to get the definition of gold changed to include my special "tin" and call it "gold II", but then box it as "gold" with a small print note that it isn't real "gold". Frankly, as Mattach said, someone HAD to have paid someone off or had a reason to profit from calling hard drives TB but then using Tebibyte. Everyone seems to be inconsistent in their use of the proper terms.

I don't know the history of how the 1998 standard came to be, but I'll take your word for it that the hard drive manufacturers pushed for the standard. The fact is, Kilo has always meant 1000 since well before computers came into existance. The definitions they pushed for aren't "their" definitions. Those definitions are simply the correct ones; 1024 is not 1000 and should never be called Kilo. Period. If they want to label their products using 1000's, that is their choice. If they ever release a 1000GB drive but call it a 1000GiB drive, then I will have a problem, obviously. 1024 had no place being called a Kilo, so being negative towards manufacturers for pushing to finally give 1024's a standardized naming system is a gross misplacement of emotions.

And until the hard drive manufacturers start calling their hard drive capacity by their own standard that they paid for they can eat shit and die. I can't believe the hard drive manufacturers bought, lobbied, and paid for their OWN standard and then didn't follow it because they didn't want to inconvenience the little folk that would be more confused. I'll still call everything TB and GB and leave it up to the person I'm listening to or talking to to realize if those extra 24 bytes REALLY matter. If you aren't smart enough to know when those 24 bytes matter, you should leave it up to me as the geek you are paying to fix your computer or getting service from to handle the "heavy lifting" :)

Again, what they pushed for is not theirs at all, as is evidenced by the fact that they never used that standard. You should be seeing their push as a service to the industry. 1024 needed a home and they gave it one. Sure they aren't using GiB to represent their drives, but they don't need to.

Overly-frequently changing standards isn't good, but neither is an incorrect or incomplete standard. You can't possibly argue that having Kilo flip flop between 1000 and 1024, purely by context and guesswork, is a good standard. The creation of the Kibi was a necessity no matter who pushed for it.

Think of it like this. If tomorrow everyone on the planet started using bibytes for binary bytes and kilobytes for base-10, how long until someone screws up a conversion and forgets that at one time bibytes was equal to kilobytes. And when someone asks the question a bunch of people in a room making lots of money will be asking themselves if a kilobyte was 1024 bytes or was 1000? It'll be as awesome as NASA's conversion from metric to imperial standard that desimated a satellite. But will lives be lost due to the mistake?

Virtually no one is using Kibi but yet your feared scenario of conversion confusion is already rampant just by looking at Windows vs Mac. They both use Kilo but one uses 1024 and the other uses 1000. Horray for the flip flopping Kilo? Is this really better than if Kibi was widely used? Say the 1998 standard doesn't exist, aren't those bunch of people in a room still going to be asking themselves if a kilobyte means 1024 or 1000? Say the 1998 standard was regularly used. Kibi has no mixed history, it means 1024 without a doubt. By simple process of elimination, anyone can deduce that Kilo will mean 1000 instead, just like it does for meters, liters, and everything else in the world.

The first step is for people to actually start using the standard which was provided to them. Mattlach's outspoken stubbornness to not change his ways is disheartening to say the least.