-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
BUILD 6 drives with ECC on a spousal budget
- Thread starter malcolmputer
- Start date
- Status
jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
Well, really, the whole dd /dev/zero thing ... it really is the best test readily available from the average UNIX system's CLI, and substitution of /dev/*random can fail for the reasons cj pointed out. It also fails in a most obvious manner if you happen to do something dumb like have compression on.
You know, the thing is, people come to this forum full of preconceived notions, based on their personal experiences and prejudices, and oftentimes it is hard to make the adjustment to ZFS, different ways of thinking about things, whatever.
cyberjock was once new here and was all full of the non-ZFS knowledge he had obviously fought hard for and gained through much learning, error, and perhaps even tears. I seem to recall that he was doubtful of many of the things we told him, but he was hearing them from multiple sources. He obviously had the ability to take the ball and run with it, not being dependent entirely on the forum for his info, but often coming back with experimental results, or clearly having researched things elsewhere. My recollection was that he was often tough to influence, but that he had a most agreeable quality - he could be shown (or discover on his own!) the evidence that his view was wrong and he would ... sometimes grudgingly, sometimes slowly, adjust his worldview until it reflected the evidence.
If he is wrong and you are right, CONVINCE him. It works. Been there, done it.
cyberjock can come across a little harsh, especially if you intend to be intransigent about your positions. But do try to listen. He's one of the best fountains of knowledge that we've got on this forum.
It is perfectly fine for you to prefer alternative tools for testing, but realistically due to the way ZFS works, the dd test is kinda useful. Cyberjock has pointed out the issues with /dev/*random and they're valid; we kind of prefer tools that work on slow AND fast platforms. Compression is easy to temporarily disable. Making a slow CPU generate a rapid stream of urandom is hard.
@malcolmputer: That is a hilarious pool name! Your pool is fine. I thought the 600MB/sec was your pool. With the 10 disks you mentioned in the first post I thought you might have a configuration that isn't what you want. It happens to alot of people. I can't tell you how many people have a disk fail and realize after that disk fails that they had no redundancy for their data for years when they thought they did.
@jyavenard: You are totally missing the point with the dd test. It gives throughput for reading or writing since dd does reading or writing depending on the command. Why would you even try to compare it to a cp since you've just created more complexity in your test. Now, if the pool you are reading from isn't fast enough to keep the writing pool filled you'll get an artificially low number. If it's latency isn't several orders of magnitude smaller than the pool you are trying to test you'll get lower values. It's the exact same reason you don't ever use /dev/random or /dev/urandom.
If you are trying to test the writing speed of a pool you gotta be 100% sure that whatever source you are getting your data from must be low enough latency and have a sufficiently high enough throughput to make the test value. The same is true if you want to do a read test. I'd be a moron to copy data cross my Gb LAN, get 120MB/sec, and then claim my pool can only do 120MB/sec. I'd be totally wrong because my bottleneck is the LAN. Even if you had a pool that was twice the speed of the pool you want to test, you are still having to deal with the ZFS load from doing a cp from one pool to the other. You want to get rid of as much loading as you can in all ways possible while simultaneously making sure you aren't artificially bottlenecking your system during the test. This isn't any different than the guys that want to run some CPU benchmark and go into their OS and shutdown every extra unnecessary service and program they can to minimize system loading during the benchmark.
What you need to keep in mind is that at the end of the day it generally doesn't matter if I get 400MB/sec or 500MB/sec on my pool dd tests. I'll still be bottlenecked at my Gb LAN(120MB/sec or so). Additionally, most people won't try to copy data from a pool to the pool because you'll end up with even slower speeds than Gb LAN(if you're lucky you'll get 70MB/sec). Even if they did, and even if the Gb LAN wasn't the bottleneck you'd STILL see slower speeds than the cp test you got because of the extra latency of the data being sent through the network to a remote system, cached in the remote system's RAM, then written back in regular intervals. They will typically be reading or writing large amounts of data to the pool, which makes the dd test far more realistic but will most certainly always be limited to your Gb LAN.
It's important to understand all of the subsystems and where your data moves about in various scenarios and how to make your benchmark tests(or even if you are just doing it for diagnostic purposes) work for you.
All of this stuff is precisely why when random people have a problem with "slow LAN" one thing we tell them to run is iperf. We don't ask them to do CIFS sharing(could be CPU bound, could be pool speed limited, could be a configuration problem with CIFS, could be a failing hard drive, could be a power management setting, the list goes on) and we need to be 100% sure that its none of those. iperf pulls data out of thin air and is light on CPU usage so you can rule out the LAN once and forever. Again, it's all about removing possible bottlenecks during your testing.
And again, just like I said yesterday, the fact that I'm even having to explain this tells me you do not have the necessary skills and knowledge to be doing any kind of benchmark and publishing it or even trying to interpret it as you don't have the skills to ensure that you haven't bottlenecked yourself in some fashion you aren't aware of.
Don't take offense to this statement, but you are precisely the kind of person I hate to see on the internet because you throw some benchmark results on a blog and provide some cool chart or graph with numbers and you shouldn't be. Then some other guy read's those numbers, thinks you are an informed and education person, and come here and fight us for days on end on why their system has better specs and can't do the crazy high I/O or throughput that you've turned around and validated as completely accurate and very trustworthy by virtue of cool graphs and charts. That's why so many people say don't believe 1/2 the crap you read on the internet. You have no way of knowing if the guy that wrote it is a genius or can't even wipe the drool off his chin without momma to help him. I choose to take the assumption that any random person I read online is talking out their butt(it's pretty obvious how many people do it here regularly if you just sit back and read every thread and see how often they are corrected). It also seems to be far more often than not. Most people in IT seem to have no detailed official training in computer engineering but they think their 15 years of experience counts for something. It does, but almost nothing compared to the guy that went to college and understands how all this technology talks to each other.
It's also precisely why we admonish those Youtube videos that say "repurpose your old hardware into a fast reliable FreeNAS server in 60 minutes!" then they don't show you how to do file permissions(which you'll need to understand), they don't know you how to do SMART testing(which is absolutely necessary for a trustworthy server), they don't show you how to setup the emails(which is absolutely necessary for a trustworthy server), and they don't show you how to do a proper zpool configuration for your intended purpose(which can make the server far too slow for your intended function). Then those individuals that blindly followed those youtube videos lose data and are just appalled that they believed the jerk-off that wrote the youtube video because they deemed him an expert. After all, experts wouldn't post videos to youtube and miss out on absolutely essential functions in FreeNAS, right? Wrong. They're only interested in the subscriptions and video views so they can make more money. I've tried to email them to and ask them to remove their videos or at least post that SMART and email should be added. They usually don't respond or they try to argue with me they aren't important. Who the heck do they think they are and what do they think they know to try to tell me that emailing and SMART aren't important when those are the main ways that your server determines a disk is failing AND informs you!
Anyways, I'm done here. Good luck to all. We've totally off-topic so I'll bow out here.
@jgreco: /wave
@jyavenard: You are totally missing the point with the dd test. It gives throughput for reading or writing since dd does reading or writing depending on the command. Why would you even try to compare it to a cp since you've just created more complexity in your test. Now, if the pool you are reading from isn't fast enough to keep the writing pool filled you'll get an artificially low number. If it's latency isn't several orders of magnitude smaller than the pool you are trying to test you'll get lower values. It's the exact same reason you don't ever use /dev/random or /dev/urandom.
If you are trying to test the writing speed of a pool you gotta be 100% sure that whatever source you are getting your data from must be low enough latency and have a sufficiently high enough throughput to make the test value. The same is true if you want to do a read test. I'd be a moron to copy data cross my Gb LAN, get 120MB/sec, and then claim my pool can only do 120MB/sec. I'd be totally wrong because my bottleneck is the LAN. Even if you had a pool that was twice the speed of the pool you want to test, you are still having to deal with the ZFS load from doing a cp from one pool to the other. You want to get rid of as much loading as you can in all ways possible while simultaneously making sure you aren't artificially bottlenecking your system during the test. This isn't any different than the guys that want to run some CPU benchmark and go into their OS and shutdown every extra unnecessary service and program they can to minimize system loading during the benchmark.
What you need to keep in mind is that at the end of the day it generally doesn't matter if I get 400MB/sec or 500MB/sec on my pool dd tests. I'll still be bottlenecked at my Gb LAN(120MB/sec or so). Additionally, most people won't try to copy data from a pool to the pool because you'll end up with even slower speeds than Gb LAN(if you're lucky you'll get 70MB/sec). Even if they did, and even if the Gb LAN wasn't the bottleneck you'd STILL see slower speeds than the cp test you got because of the extra latency of the data being sent through the network to a remote system, cached in the remote system's RAM, then written back in regular intervals. They will typically be reading or writing large amounts of data to the pool, which makes the dd test far more realistic but will most certainly always be limited to your Gb LAN.
It's important to understand all of the subsystems and where your data moves about in various scenarios and how to make your benchmark tests(or even if you are just doing it for diagnostic purposes) work for you.
All of this stuff is precisely why when random people have a problem with "slow LAN" one thing we tell them to run is iperf. We don't ask them to do CIFS sharing(could be CPU bound, could be pool speed limited, could be a configuration problem with CIFS, could be a failing hard drive, could be a power management setting, the list goes on) and we need to be 100% sure that its none of those. iperf pulls data out of thin air and is light on CPU usage so you can rule out the LAN once and forever. Again, it's all about removing possible bottlenecks during your testing.
And again, just like I said yesterday, the fact that I'm even having to explain this tells me you do not have the necessary skills and knowledge to be doing any kind of benchmark and publishing it or even trying to interpret it as you don't have the skills to ensure that you haven't bottlenecked yourself in some fashion you aren't aware of.
Don't take offense to this statement, but you are precisely the kind of person I hate to see on the internet because you throw some benchmark results on a blog and provide some cool chart or graph with numbers and you shouldn't be. Then some other guy read's those numbers, thinks you are an informed and education person, and come here and fight us for days on end on why their system has better specs and can't do the crazy high I/O or throughput that you've turned around and validated as completely accurate and very trustworthy by virtue of cool graphs and charts. That's why so many people say don't believe 1/2 the crap you read on the internet. You have no way of knowing if the guy that wrote it is a genius or can't even wipe the drool off his chin without momma to help him. I choose to take the assumption that any random person I read online is talking out their butt(it's pretty obvious how many people do it here regularly if you just sit back and read every thread and see how often they are corrected). It also seems to be far more often than not. Most people in IT seem to have no detailed official training in computer engineering but they think their 15 years of experience counts for something. It does, but almost nothing compared to the guy that went to college and understands how all this technology talks to each other.
It's also precisely why we admonish those Youtube videos that say "repurpose your old hardware into a fast reliable FreeNAS server in 60 minutes!" then they don't show you how to do file permissions(which you'll need to understand), they don't know you how to do SMART testing(which is absolutely necessary for a trustworthy server), they don't show you how to setup the emails(which is absolutely necessary for a trustworthy server), and they don't show you how to do a proper zpool configuration for your intended purpose(which can make the server far too slow for your intended function). Then those individuals that blindly followed those youtube videos lose data and are just appalled that they believed the jerk-off that wrote the youtube video because they deemed him an expert. After all, experts wouldn't post videos to youtube and miss out on absolutely essential functions in FreeNAS, right? Wrong. They're only interested in the subscriptions and video views so they can make more money. I've tried to email them to and ask them to remove their videos or at least post that SMART and email should be added. They usually don't respond or they try to argue with me they aren't important. Who the heck do they think they are and what do they think they know to try to tell me that emailing and SMART aren't important when those are the main ways that your server determines a disk is failing AND informs you!
Anyways, I'm done here. Good luck to all. We've totally off-topic so I'll bow out here.
@jgreco: /wave
- Status
