Odd ZFS Alert after upgrade to 9.2.0-RC

[joeschmuck]

I received this Warning when I logged in this morning.

Warning: The volume test (ZFS) status is ONLINE: One or more devices are configured to use a non-native block size. Expect reduced performance.Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool.

So it looks like there is a check to verify the block sizes. It's funny because I created the pool using 9.2.0-Beta default values and it didn't give me this warning message. I suspect the 512 byte block sizes are due to my drives being rather old and not Advanced Format 4K drives. So this makes be wonder about drives that have built in 512 byte compatibility so the host never sees the 4K sectors, how does this impact them?

I think the warning could have more meaningful data in it such as specifically telling the user which devices are the issue and what the issue really is.

Code:

[root@test_freenas] ~# zpool status
  pool: test
state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
  scan: none requested
config:
 
        NAME                                          STATE    READ WRITE CKSUM
        test                                          ONLINE      0    0    0
          gptid/1429c4a5-54ac-11e3-9f43-50e549b78964  ONLINE      0    0    0  block size: 512B configured, 4096B native
          gptid/149df17a-54ac-11e3-9f43-50e549b78964  ONLINE      0    0    0  block size: 512B configured, 4096B native
 
errors: No known data errors

 

[cyberjock]

I *think* this changed a few days ago and 4k is now the default. Try creating the pool with the RC. I finally got my way with ticket https://bugs.freenas.org/issues/2404 for what its worth. :P

 

[Yatti420]

Did you even have the option to specify 4k from the 9.2.0 beta zfs volume manager?

 

[cyberjock]

The really shitty part is I'm betting we're gonna get a bunch of people that got boned with that improper block size. ;)

 

[Dusan]

It seems it's being already looked at: https://bugs.freenas.org/issues/3626

 

[cyberjock]

Did you even have the option to specify 4k from the 9.2.0 beta zfs volume manager?

You had the option to force 4k sector size (ashift=12) from 8.2.0 on. But, starting with 9.x it was removed and replaced with an intelligent(but not so intelligent) system that was supposed to pick smartly for you. It wasn't 100% accurate(as we just witnessed) AND you couldn't plan ahead by forcing 4k sector size. So if you have really old(or misdetected) drives that you later replace(d) with newer drives you're about to see the "Wrath of the Ashift".

 

[cyberjock]

It seems it's being already looked at: https://bugs.freenas.org/issues/3626

That one is because the ashift=12 is now forced but that user has 512-byte sector drives.

This thread is the opposite. His drive has 4k sectors, but configured as 512-byte sector drives.

I *think* you get this warning if your drive sectors don't match the configured size.

They need to ignore warnings for 512-byte sector drives with ashift=12 but warn for 4k sector drives that are 512-byte. :/

 

[joeschmuck]

I don't believe I had the 4K option in 9.2.0-Beta when I created the pool. So for anyone who made the mistake of creating a pool using the Beta and transferring a lot of files to it, well yes, I would say they got boned.

Tomorrow I will destroy the pool and recreate it using the RC and see what that looks like.

 

[garfunkel]

Ah yes, this one bit me too. I spent about a week, backing up, destroying, recreating and resilvering my pool because I wanted to sort this out. It also turned out that for some reason my pool (created with 9.2 beta didn't work with 9.2 RC. It turned out it was because my disks in the pool did not have the gptid/blabla name, but instead used /dev/adaX names, and for some reason the RC didn't like that, so I had to resilver the disks, which renamed them and then upgrade to the RC...

 

[cyberjock]

So your pool that had the dev/adaX was created from the CLI? The GUI uses the gptids for pool creation. That could explain some of your problem.

As a general rule we try not to tell people to use the CLI because FreeNAS expects things to be a certain way. When you do it your own way it doesn't jive with FreeNAS and you can potentially have problems later that you didn't even know you had.

 

[ZFS Noob]

OK, help a n00b out here.

I've got 2 servers that I'm installing from scratch. One uses 2TB Seagate Constellation ES.3 SAS drives, and the other will use 2TB WD Red drives.

Should I be installing these with 4k sectors rather than the default?

 

[cyberjock]

Well, there's lots of grey area, but the "proper" way is this:

1. Each vdev in your pool(s) should be all 4k or all 512-byte sector drives. No mixing.
2. Each vdev in your pool(s) should have an ashift value that properly aligns with the sector size if your disks.

Now, there's an exception in my book, but it's technically improper.

If you have 512-byte/sector disks you could still go with a 4k/sector ashift. But you will get the warning until you fix it. The reason why you may want to go this route is because an ashift cannot be changed once the vdev has been created. So if you plan to go with bigger disks in the future versus creating a new pool you may want to set the ashift to 4k as you will be guaranteed to have 4k/sector disks when you upgrade someday. So some planning is involved.

Here's where the confusion begins. Back in 8.0.4 or 8.2.0(I forget which) you had the option to force a 4k sector size. This let you plan ahead(woohoo). But starting with 9.x that option was removed. The decision was made to let FreeNAS make the proper choice for your current hardware. This had 2 issues.

1. Sometimes it would improperly determine your disks sector size. (Many hard drives lied about their sector size, and the exception list FreeBSD had was not 100%)
2. If you used old drives(or newer 4k/sector disks that lied and FreeBSD didn't catch this error with its exceptions list) then it would give you a wrong ashift value. It seems that many people fell into this category with 9.x(ug).

I kind of figured many people would fall into #2, so I tried to push for 4k to just be forced for everyone. But my ticket didn't get closed until last week.

So if you created the pool with 9.x you might luck out and have the proper ashift for 4k sector disks, or you might not. But, if your disks were 512 bytes/sector disks you can't upgrade a pool by resilving 1 disk at a time and autoexpanding the pool without getting the warning. So you lost the futureproof-ness. Not cool in my opinion.

Now, there is a new problem. 9.2.0-RC and newer seems to force the ashift to match 4k sector drives(I requested this when 9.1.0 was in beta). So starting with 9.2.0 any pool you make WILL be 4k sector compatible . But since 9.2.0 also appears to properly determine sector size, it will yell at you if your disks and ashift don't match. So now we're future-proof with the 4k sector problem. But if you have old disks, you fell through the cracks with 9.x, or you used 8.x and didn't properly choose your ashift value you are about to get a warning....forever.....or until you fix the problem. Many home users will not be able to fix the problem, but that's their problem.

Long story short, I personally feel that the warning should be there. I think you should also have the option of disabling it in the GUI so it doesn't put you in a state of perpetual warning. We're going to have a large number of people that are going to be in a perpetual state of warning as soon as they upgrade to 9.2.0 with things the way they currently stand. So clearly this is something we need to deal with appropriately.

Hopefully it's all cleared up now. ;)

 

[Xaro]

That one is because the ashift=12 is now forced but that user has 512-byte sector drives.

This thread is the opposite. His drive has 4k sectors, but configured as 512-byte sector drives.

It was me who commented on this issue.

To clarify, all my HDDs have 4K physical and 512B logical sectors (partitions correctly aligned). All OK.

My SSDs have 512B physical and logical sectors (if we believe that camcontrol said). When using these SSDs for L2ARC, ZFS report warning, when using same SSDs for ZIL or in the pool, no warnings at all.

Regardless of use, in each case, partitions seems to be always 4K aligned.

I don't know if it really or not a bug in FreeBSD 9.2, a partition generating problem in FN9.2-RC or any else, but this warning is a bit confusing and it should be correctly catched/interpreted before any report to user (FreeNAS GUI)

It was in my opinion, in the sense that Jordan has opened this issue and I commented.

 

[cyberjock]

But notice how your drives report 4k physical but 512B logical? That's the hard drive lying to the OS because many software programs freak out if you use anything except 512 bytes/sector. That's what lead to this whole problem with hard drives trying to emulate 512 bytes/sector even though its not.

And I *think* that all SSDs are 4k sectors. It's hard to keep up with all this stuff, and hopefully the following paragraph is accurate.

Your ticket I believe is related to the fact that l2arcs always use 4k blocks. This has to do with how the l2arc functions and can't be changed as far as I know. But your SSD has 2 things working against it. One: It's an SSD, so its already 4kbytes/sector, but reporting itself as 512-bytes/sector for compatibility reasons. Two: FreeBSD doesn't have the brains to realize its actually an SSD nor does it have any info to make it think that the drive is 4k/sector. To make matters worse if you try reading up on this topic many SSDs are referred to as "pages" of 128kb up to 1MB or so. That's not the same as the sector size and some companies just won't say what their sector size actually is. Then, some companies report a sector size that is simply "the smallest amount of data that the SSD controller can read from the memory chips". So its similar to old fashioned platter disks, but there's no write component specified for whatever reason.

Anyway, your ticket leads me to believe that the problem is you are using an SSD(so by definition its 512 bytes/sector) but the l2arc wants to use 4k/sector. The reality is that SSDs aren't appreciably hurt in the performance arena despite using different cluster sizes in windows. Most companies also contend the 512-bytes per sector so they can claim higher IOPS than claiming 64kbytes as your transfer rate could be limiting over SATA.

The reality of this whole situation(if I'm understanding it all and don't have any gross conceptual errors) is that if you are using an SSD as an L2ARC, all this crap doesn't really mean anything in the performance arena. As long as your writes are aligned to the proper boundaries(it is if you setup the L2ARC with FreeNAS' GUI) then you can safely ignore the warning for the L2ARC. And the solution for many people that are going to get screwed with this error despite doing things correctly is to ignore any warnings related to ZILs and L2ARCs and just make the assumption that you are smart enough to use an SSD and not platter based disks.

Clear as mud? Join the club!

 

[Xaro]

Overall, I agree with you.

Yes, FreeBSD has no brain probably but it could detect SSD or HDD, How ? With some flags and values reported when querying device. For example: media RPM, TRIM support (that only available on SSD).

Anyway, I did not noticed any performance degradation in my comparative tests between FreeBSD 9.1 vs. 9.2, FN9.1.1-RELEASE and FN9.2-RC.

Again, as in my previous post, this warning is confusing for user and makes debate ;-) So be it...

 

[cyberjock]

My Intel SSD claims to spin at 5400RPM... /sigh

As for TRIM support I guess you could use it(assuming your SSD supports TRIM). Many SLC drives don't because TRIM doesn't matter with SLC memory. SLC happens to be the most preferred memory type for ZILs and L2ARCs too. You could definitely identify drives that do support TRIM as SSDs, but so many don't support TRIM you'd be in a situation like we are currently discussing with the 4k sector support. It's better to come up with a solid solution for everything. I just don't know what that solution is. ;)

I personally feel the best solution is to ignore all l2arc and zils that dispute their sector size. If you are dumb enough to use platter media for those 2 situations then you deserve the performance penalty if its wrong. You clearly aren't doing your server any favors if you are using them, so why worry about those users?

You shouldn't notice any performance degradation between 9.1 or 9.2 You've done everything correctly(woohoo!). It's just as you said... the warning is confusing for users.

 

[joeschmuck]

My Intel SSD claims to spin at 5400RPM...

That is awesome, how did Intel get those chips to spin in the case so fast?

 

[cyberjock]

I don't know, but I should have bought the 7200RPM version for lower latency and higher transfer rates!

 

[Spanner]

Hi cyberjock

I am a business user of FreeNas and use it at face value, i.e. what is presented to me on the GUI. I do not have CLI experience. I have just upgraded to 9.2RC. After the upgrade I have also received the flashing alert on GUI :

One or more devices are configured to use a non-native block size. Expect reduced performance.Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool.

However as a GUI user of Freenas, I do not understand what "ashift" is and all the technical details mentioned in your discussion with joeschmuck.

My key questions are :-

1. I am worried that my data is at risk due to the flashing icon in the GUI. What are the long term implications?
2.How does one resolve this issue?
3.Why did this occur in the first place as I have religiously been upgrading from each version of FreeNas?

You assistance would be appreciated.

 

[joeschmuck]

1. Your data is not at risk right now but if you have a drive failure and need to install a replacement drive that doesn't have 512 byte sector emulation, I don't know what could happen. Maybe Cyberjock could explain a little better.

2. To clear the flashing warning just click on the icon, now it tells you why it's flashing. Uncheck the box and the flashing will go away. When you do click on the alert icon you will still see the warning message so you do not loose that info.

3. You probably originally created your pool without the 4k sector option or your hard drives emulate 512 byte sectors (for backwards compatibility) even though you have an advanced format drive.

Hope that helps some.