Intel 750 PCIe SSD Drive

[kspare]

I thought I was start a thread for others to follow.

I picked up two of these pci ssd drives. I'm having problems with it hanging on the volume manager when I try to add the drive as a L2Arc.

I'm running the latest version of Freenas as the date this posted.

 

[kspare]

This is the message log, it just gets stuck. I tried creating a mirror with the intel 750 as L2.

May 19 14:46:17 kspsan02 notifier: 1+0 records in
May 19 14:46:17 kspsan02 notifier: 1+0 records out
May 19 14:46:17 kspsan02 notifier: 1048576 bytes transferred in 0.006331 secs (165620279 bytes/sec)
May 19 14:46:17 kspsan02 notifier: dd: /dev/nvd0: short write on character device
May 19 14:46:17 kspsan02 notifier: dd: /dev/nvd0: end of device
May 19 14:46:17 kspsan02 notifier: 5+0 records in
May 19 14:46:17 kspsan02 notifier: 4+1 records out
May 19 14:46:17 kspsan02 notifier: 4284416 bytes transferred in 0.004406 secs (972410345 bytes/sec)
May 19 14:46:17 kspsan02 notifier: 1+0 records in
May 19 14:46:17 kspsan02 notifier: 1+0 records out
May 19 14:46:17 kspsan02 notifier: 1048576 bytes transferred in 0.063128 secs (16610317 bytes/sec)
May 19 14:46:17 kspsan02 notifier: dd: /dev/da0: short write on character device
May 19 14:46:17 kspsan02 notifier: dd: /dev/da0: end of device
May 19 14:46:17 kspsan02 notifier: 5+0 records in
May 19 14:46:17 kspsan02 notifier: 4+1 records out
May 19 14:46:17 kspsan02 notifier: 4284416 bytes transferred in 0.055333 secs (77429812 bytes/sec)
May 19 14:46:18 kspsan02 notifier: 1+0 records in
May 19 14:46:18 kspsan02 notifier: 1+0 records out
May 19 14:46:18 kspsan02 notifier: 1048576 bytes transferred in 0.059917 secs (17500484 bytes/sec)
May 19 14:46:18 kspsan02 notifier: dd: /dev/da1: short write on character device
May 19 14:46:18 kspsan02 notifier: dd: /dev/da1: end of device
May 19 14:46:18 kspsan02 notifier: 5+0 records in
May 19 14:46:18 kspsan02 notifier: 4+1 records out
May 19 14:46:18 kspsan02 notifier: 4284416 bytes transferred in 0.051237 secs (83619398 bytes/sec)
May 19 14:46:19 kspsan02 notifier: vfs.zfs.vdev.larger_ashift_minimal: 0 -> 1

and from the debug.log

May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:172] Executing: swapoff -a
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:186] Executed: swapoff -a -> 0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:172] Executing: dd if=/dev/zero of=/dev/nvd0 bs=1m count=1
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:186] Executed: dd if=/dev/zero of=/dev/nvd0 bs=1m count=1 -> 0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: diskinfo nvd0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:172] Executing: dd if=/dev/zero of=/dev/nvd0 bs=1m oseek=381550
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:186] Executed: dd if=/dev/zero of=/dev/nvd0 bs=1m oseek=381550 -> 0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart create -s gpt /dev/nvd0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart add -a 4k -b 128 -t freebsd-swap -s 4194304 nvd0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart add -a 4k -t freebsd-zfs nvd0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart bootcode -b /boot/pmbr-datadisk /dev/nvd0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: zpool status freenas-boot
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: camcontrol devlist -v
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:172] Executing: dd if=/dev/zero of=/dev/da0 bs=1m count=1
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:186] Executed: dd if=/dev/zero of=/dev/da0 bs=1m count=1 -> 0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: diskinfo da0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:172] Executing: dd if=/dev/zero of=/dev/da0 bs=1m oseek=1907725
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:186] Executed: dd if=/dev/zero of=/dev/da0 bs=1m oseek=1907725 -> 0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart create -s gpt /dev/da0
May 19 14:46:17 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart add -a 4k -b 128 -t freebsd-swap -s 4194304 da0
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart add -a 4k -t freebsd-zfs da0
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart bootcode -b /boot/pmbr-datadisk /dev/da0
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: zpool status freenas-boot
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: camcontrol devlist -v
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:172] Executing: dd if=/dev/zero of=/dev/da1 bs=1m count=1
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:186] Executed: dd if=/dev/zero of=/dev/da1 bs=1m count=1 -> 0
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: diskinfo da1
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:172] Executing: dd if=/dev/zero of=/dev/da1 bs=1m oseek=1907725
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:186] Executed: dd if=/dev/zero of=/dev/da1 bs=1m oseek=1907725 -> 0
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart create -s gpt /dev/da1
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart add -a 4k -b 128 -t freebsd-swap -s 4194304 da1
May 19 14:46:18 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart add -a 4k -t freebsd-zfs da1
May 19 14:46:19 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: gpart bootcode -b /boot/pmbr-datadisk /dev/da1
May 19 14:46:19 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: zpool status freenas-boot
May 19 14:46:19 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: camcontrol devlist -v
May 19 14:46:19 kspsan02 manage.py: [middleware.notifier:172] Executing: /sbin/sysctl vfs.zfs.vdev.larger_ashift_minimal=1
May 19 14:46:19 kspsan02 manage.py: [middleware.notifier:186] Executed: /sbin/sysctl vfs.zfs.vdev.larger_ashift_minimal=1 -> 0
May 19 14:46:19 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: zpool create -o cachefile=/data/zfs/zpool.cache -o failmode=continue -o autoexpand=on -O compression=lz4 -O aclmode=passthrough -O aclinherit=passthrough -f -m /ESXi_2 -o altroot=/mnt ESXi_2 /dev/gptid/18cb7189-fe68-11e4-8870-0025902f96f4 mirror /dev/gptid/193e9bbe-fe68-11e4-8870-0025902f96f4 /dev/gptid/19bfb35a-fe68-11e4-8870-0025902f96f4
May 19 14:46:30 kspsan02 alert.py: [middleware.notifier:208] Popen()ing: /sbin/zpool status -x freenas-boot
May 19 14:46:43 kspsan02 manage.py: [middleware.notifier:208] Popen()ing: /sbin/zpool status -x freenas-boot


any ideas?

 

[kspare]

This command keeps running.

[middleware.notifier:208] Popen()ing: /sbin/zpool status -x freenas-boot but the gui never completes the addition of the L2Cache.

 

[mjws00]

Watching this one as I want to see what is up with the 750's.

Not sure why you are trying to mirror l2Arc. It should be striped automatically by the GUI when added. Also wondering why da1 and freenas-boot are showing up in the logs if you are only adding cache. Probably something I've never noticed to do with swap. We'll see.

Upload the debug file so we can get a better lay of the land. System | Advanced | Save Debug.

Also was the drive previously played with? Did it already have partitions or anything?

I'd also add the cache from the CLI just to see what's up and if everything is functioning. This is just burn-in and testing right. Not live data?

 

[kspare]

I'm not mirroring the l2arc.

I found this https://bugs.freenas.org/issues/5390

It has to do with trim. i'm going get the trim on init turned off and try it in couple of minutes.

 

[mjws00]

Cool if disabling TRIM gets it. This will tag lots of people if that is necessary.

 

[kspare]

it looks like it could take 2 hours or so to trim the whole drive. I wonder if thats a good thing or not.

 

[mjws00]

Everything I see says these do TRIM and garbage collection properly. But there aren't many in the wild yet.

 

[kspare]

yep added no problem with trim disabled.once I get the volume enabled i'll likeyl enable trim.

 

[kspare]

I wasn't able to get trim on init disabled. though. So I guess the process would be to setup the volume how you want it and then re-enable trim? It should logically be faster with trim enabled should it not?

 

[kspare]

here is another interesting read.

http://fs.freebsd.lists.muc.groups.com.ru/18177-Re_Creating_zpool_on_NVMe_Disks_takes_forever

 

[cyberjock]

I know someone that had a problem like this. If you wait long enough, it *does* eventually finish. He didn't have an exact time as he started it and went to bed. In the morning the drive was added to the zpool as expected.

I'd be curious to see what happens if you remove the disk from the zpool, turn TRIM on, then try to add the disk again.

 

[kspare]

i'm just waiting for some vms to move off my other san and then i'll be able to play around a bit more. I'm sure it wouldn eventually finish now that I know. But I did some other reading and it sounds like most of the trim commands wouldn't be support right now anyway, so probably not a big deal.

 

[kspare]

You must set vfs.zfs.vdev.trim_on_init as a Sysctl with the value of 0 in order to make this work. This will disable trim on initialization. It apparently will eventually work but it will take quite some time.

 

[kspare]

So far so good with the new cards, our terminal servers have no lag anymore and our deduplication drive is seeing a 94% hit rate on the L2 cache. we're pretty happy over here.

 

[joeschmuck]

What are the specs of your system, I'm very curious? Apparently it's fairly robust.

It should logically be faster with trim enabled should it not?

As for TRIM, it only speeds up writing data to the SSD. Even if you have TRIM turned off it will not stop your SSD from working, it will just slow down write operations. Keeping things simplified, your data is written in 4K blocks of memory on the SSD. When you change some data that 4k chunk gets copied to another 4k chunk (with the changed data) and then is marked as unused. During the write operation the data location must first be erased and this operation takes time (albeit a small amount of time but it adds up) and then the data can be written. To mitigate waiting for the unused 4k chunk to be erased the TRIM command comes in and will erase at a device level all the blocks marked as unused and not yet erased. Keep in mind that this is a very simplistic explanation. Real details can be found by searching the internet and you can also read up about Wear Leveling.

On the positive side, when looking into the Intel 750 (what an eye opener and could you ship me one for an evaluation? I want to evaluate if I could handle using it on my main computer system for a few years) it appears it has it's own internal garbage collection (GC) as well as TRIM support but even if you leave TRIM off, it should still work well using it's internal GC. If you find out TRIM will work for this device on FreeNAS, I'd turn it on of course.

 

[kspare]

I'm running 24 2TB WD Black Drive, 64GB Ram, Dual Xeon E5-2620, LSI 9207 HBA, and 1 Intel 400GB PCI Card. I use a Intel 10GB 520 Nic with iscsi out to my vms. It honestly works so good.

We started using symantec dedup on our other san and it's working incredibly well with the ssd. I have a 90% hit ratio on L2 and 87% on L1 for that san. pretty happy!

 

[joeschmuck]

It's a bit much for a home system but I'd take it ;)

 

[kspare]

It makes duke nuke'm run at blazing speeds!

 

[joeschmuck]

It makes duke nuke'm run at blazing speeds!

I remember when that came out, I played the hell out of it. Simpler times.