Rackable C1000 and Enclosure with m1015

[some_guy_named_ed]

So I picked up this on ebay in hopes for a badass freenas setup: http://www.ebay.com/itm/171192192174

Then I read this post (Yes - my confusion led me to it): http://forums.freenas.org/threads/confused-about-that-lsi-card-join-the-crowd.11901/

That's when I realized my new system didn't exactly come with an optimal raid controller. If I bought an IBM m1015 controller and flashed it, would I still be able to connect it to the 16 bay enclosure? Are the connectors the same? Would I still get to use all 16 bays? If I can then I'm sure it'd be ugly looking to have to connect the cables internally then run them out of the box somehow but I wouldn't mind that so much if it means a more stable zfs setup overall.

I consider myself an IT pro but I'm generally lacking in my knowledge in this particular area (I'm not too proud to admit that) so any help and knowledge to be shared is appreciated.

 

[jgreco]

I don't recall the details of that drive chassis. Go dig up the manuals and we can figure it out together.

The chassis will either be based on an SAS expander (likely, good) using SFF8088 external 4lane SAS cables or it'll have some sort of RAID controller - possibly bad if it cannot be convinced to go eff off and let you deal directly with the raw disks.

The M1015 will have SFF8087 internal connectors. You will need an external to internal backplate and cable. Anything labeled SGI is probably old enough to be 3Gbps SAS so there may be limits as to the drive sizes (2TB if so).

That's all I have for you offhand.

 

[some_guy_named_ed]

That pointed me in the right direction. It looks like they are SFF8088 ports after all. The c1000 I got came with a Rackable se3016 SAS expander to be more specific.

It lead me to a post on another forum where somebody just arbitrarily posted they have it working with an IBM m1015 crossflashed to IT mode.. Looks like I may be shopping for cables as well as a new card.

My current raid card is an LSI SAS8888. In a final sanity check before I commit to buying a different raid card: There's no way to flash one of those to IT mode is there? I'm guessing no since I see no mention of it on these forums, and it's cheaper then the m1015 on ebay. I figure if it could be done, the demand for them would have hiked the price up a bit more as it did with the m1015.

Thanks much.

 

[jgreco]

No idea what that card is (and at a party on my phone, so not gonna look further right now). Suggest: identify card chipset then try to match it against chipsets known to be capable of IT (i.e. lsi2308 is, lsi2208 isn't, etc)

 

[Richman]

Make sure you put jumper wires across your electric meter so it doesn't spin like a Firestone tire at Daytona after you hook that thing up. ;) :DLOL

 

[jgreco]

It'd actually be nice to know how the thing does on power consumption.

 

[Richman]

Make sure you put jumper wires across your electric meter so it doesn't spin like a Firestone tire at Daytona after you hook that thing up. ;) :DLOL

I accidently reposted in stead of edit. How do you delete these posts ........DELETE, DELETE, DELETE, DELETE

 

[some_guy_named_ed]

Just a quick update. I'm now waiting for an IBM m1015 to arrive in the mail as well as an sff8087 to sff8088 adapter. The card I currently have apparently is just a raid controller that can't be converted into a plain HBA. Oh well. The system I got was still a pretty good deal. This is mostly a learning exercise for me.

I'm sure this is going to be a power whore, but I had to keep my up front costs low. My wife tolerates my learning methods as long as I stay cheap. My evil plan is to get it built, working, reliable, then get my wife to depend on it by encouraging her to store all her crap somewhere on it. Her pics, videos, files.. whatever. Then when she sees and complains about the power bill, I show her how spending more in a better more efficient system will reduce the power bill enough to magically cover the upgrade cost over the first year or so. mwahahahahahaha!!! Achievement get: Enterprise NAS/SAN in the home! :P

Seriously though - now I'm curious about power consumption too. Where did my kill-o-watt go? I'll post some findings when I get to it.

 

[Jayhawk734]

Any update on this? I was looking at buying the same setup, especially since the ebay seller is a few minutes from my house. I couldn't tell from the listing, did it come with the LSI raid card or did you have to buy it? What besides disks was necessary to get it up and running? I'd also love to know any power reading from the kill-o-watt if you've had the time.

 

[Richman]

Any update on this? I was looking at buying the same setup, especially since the ebay seller is a few minutes from my house. I couldn't tell from the listing, did it come with the LSI raid card or did you have to buy it? What besides disks was necessary to get it up and running? I'd also love to know any power reading from the kill-o-watt if you've had the time.

His kill-o-watt probably made a loud 'POP' sound followed by whistling wine noise accompanied with sparks and ending in smoke and flames. Its hard to read the digits with all of that going on. ;)

 

[some_guy_named_ed]

Lost my kill-o-watt unfortunately. I probably left it on a client site and forgot about it some time ago. If/when I find it or buy a new one I'll follow up.

@Jayhawk734: The c1000 came with an LSI raid card (MR SAS 8888 I think) however I couldn't find a way to bypass the raid firmware and just run it in any sort of IT/IR mode. I think that's the gamble on ebay with rackables though. You may get lucky enough to get one with a better suited card for zfs, but unless you find a listing specifically stating the raid card that's in the box, assume the worst. Anyway, if you end up having to get something like an m1015, you'll also need a flexible riser cable so you can re-position the card to make room for the sff8087 to sff8088 adapter you'll also need to buy in order to get the ports needed to attach the sas expander. That's what I ended up doing for my rig.

Small disclaimer though. After struggling with the 'zfs breathing' issue (google it) with freenas and my setup, I ended up switching to nas4free. I honestly don't like it as much but the performance is consistent and large writes don't just stop for no reason.

 

[Richman]

Small disclaimer though. After struggling with the 'zfs breathing' issue (google it) with freenas and my setup, I ended up switching to nas4free. I honestly don't like it as much but the performance is consistent and large writes don't just stop for no reason.

Are you using NAS4Free with ZFS or are you using another FS? Looks as though the 'zfs breathing' issue means the NAS running ZFS pauses for a couple seconds once in a while. Trying to think how that would be problematic for me as a home file server.

 

[some_guy_named_ed]

@Richman: ZFS all the way. Same exact setup on nas4free and freenas.

Hardware specs are intel nics, 16gb ram, ibm m1015 in IT mode, 4x 15k rpm 70gb sas drives in raidz2 with no compression or dedup. I normally would use SMB/CIFS to transfer about 20gb of isos as a benchmark. Typically, my first hang would happen when about 4gb of data was written (xfer speeds drop from 115megs per sec to well... 0). The hang would be between 10-20 seconds at a time. When it was too long, my copy would time out altogether and I'd be prompted to retry. When somebody suggested it may be the cifs protocol, I tried a local test using the dd command (I forget the command line format, but it was 1000x 8mb blocks I wrote using /dev/zero). Unfortunately, It still hung up in the middle of the file write. I've even tried many suggested tunables but some of the most relevant ones are deprecated in freenas 9.2 with no viable replacement. I wasn't about to put my box at risk using a deprecated tunable so I never got very far with those.

If I have to accept that the breathing issue is normal behavior then I have to accept that it's too disruptive of a behavior to be production ready and can't use it. Random copy time-outs with large amounts of data with hangs that block everybody out of the system until they've cleared kinda puts a damper on things if you ask me. Unfortunately, to add insult to injury, the fact that this behavior with zfs doesn't happen in nas4free means that either the assumptions that zfs writes are working as they're supposed to are flawed, or nas4free has a magic zfs genie bottled inside laughing at all those 'normal' freenas installs with 'normal' breathing issues. Ok.. maybe I'm being a bit of a sarcastic turd at this point but it's nothing personal towards anybody.. just venting frustration.

I haven't brought this up before because there's already a lot of complaints about it on the net and my hardware specs aren't really much different from others faced with the issue. The norm seems to have been to try to convince people to accept the behavior and move on.. So there you go.

Since I put this out here now, is this something I should dump in it's own topic in hopes it'll be seen as a problem and addressed by people curious about a solution? Or can I expect to be told it's the norm still? Since the software is free and everybody that helps out here volunteers I wouldn't exactly hold it against anybody if that were the case. I'd still prefer to use freenas, but for me, reliability comes before preference and while I'm having problems with AD and ACLs in nas4free that I didn't have in freenas, I'm a lot closer to sorting those out then I ever was with the zfs hangs in freenas.

I didn't expect this to become such a long post but there you have it. Thanks for reading!

 

[cyberjock]

Actually, to be honest, I can't remember the last time there was a user that complained about ZFS "breathing". If you setup your server properly it shouldn't be a problem at all. ZFS "breathing" shouldn't last more than a second or two, so you have other problems.

So something is wrong IMO. Are you using compression? Dedup? Are your hard drives in proper working order per SMART tests?

IMO you have either a failure to properly administrate your server or failing hardware. So you'll need to figure out which one it is.

Post the output of some dd tests. There's stickies around that explain how to do it if you don't know how.

 

[jgreco]

The supposed "breathing" thing appears to be mostly an interaction in older ZFS versions where poor pool design (fundamentally too slow among other things) combined poorly with aggressive defaults in FreeBSD (see bug 1531) and the large FreeNAS footprint which enhanced that problem.

I've been busy with a client for several months now and haven't really played much with 9.2, so I haven't looked into what has been changed. I know that several of my "fixit" tunables from 1531 have been deprecated.

cyberjock is essentially correct about dd tests. Do these things in particular:

1) Individually test the access speed of each disk (read speed test). "dd if=/dev/daX of=/dev/null bs=65536" or whatever. Speeds for all disks should be within about 10% of each other. Report to us the average speed. Should be well north of 100MB/sec per drive.

2) Create a default RAIDZ2 pool. Go on to it, making sure compression is disabled, and do "dd if=/dev/zero of=testfile bs=1048576 count=50000". If the average speed is less than 125MB/sec, your pool cannot keep up with the speeds that data can be thrown at it over the network.

In normal systems, a storage subsystem that is slow simply slows everything down a little at a time. If you're writing to a 5400RPM drive, you write block ... wait ... write block ... wait ... etc. So the delays are spread out evenly. With ZFS, that works differently. ZFS is gathering up writes to batch them out as a transaction group, so it eagerly soaks up writes as fast as the system demands, and then blats it out as a big blob while the next transaction group starts accumulating. But if the block that is being written can't finish before the next one has to start writing out, then you have a crisis. ZFS resolves this by simply waiting for the previous transaction group to finish flushing out to disk. This is primarily indicative of a bad pool design and/or bad tuning decisions.

 

[ZFS Noob]

It might be worth destroying that pool and creating a mirrored pool instead. Writes should be ~ twice what they are with a RAIDZ2 pool, shouldn't they? This might be enough to solve the "you're filling the sink faster than the water can drain" problem.

 

[some_guy_named_ed]

IMO you have either a failure to properly administrate your server or failing hardware. So you'll need to figure out which one it is.

@cyberjock: I'd love to prove you wrong on both, but even if I don't I'll be happy just knowing this is figured out.

@jgreco:

I'm tunneled into my system from work right now and the box is set to nas4free at the moment. When I get home I can switch up the usb stick for the one with freenas and repost results if you still need them.

Drive specs from when I was having the problem were four 15k RPM sas drives in raid z2 with no dedup or compression. The drives are only 70gb each.

SMART results are at: http://pastebin.com/DL9hv4Vh

da0 to null: 73407820800 bytes transferred in 662.276371 secs (110841673 bytes/sec)
da1 to null: 73407820800 bytes transferred in 663.047911 secs (110712694 bytes/sec)
da2 to null: 73407820800 bytes transferred in 662.856467 secs (110744670 bytes/sec)
da3 to null: 73407820800 bytes transferred in 662.014703 secs (110885484 bytes/sec)

zpool test result: 52428800000 bytes transferred in 316.950403 secs (165416417 bytes/sec)
...

The results don't seem to be stellar, but they do seem to be at or above the thresholds you quoted. Is there something in the SMART test I'm failing to interpret that could have caused the problem on freenas?

 

[jgreco]

Those speeds are marginal, while they aren't a blinking red "PROBLEM" light, you probably want to retry what you were doing that stalled and monitor the filer disks with gstat. I'm guessing they all go red shortly before your stall, and if so, then you're basically looking at remediation steps for a busy pool.

 

[some_guy_named_ed]

Took a break from this for a while - didn't think this was going anywhere then it dawned on me. I think I'm a step closer to figuring this out. By any chance, does freenas write any logs to the embedded media or anything of the sort? I've found that when I would attempt to adjust the tunables, the system would hang up for as much as 5 minutes trying to write it out (but would eventually successfully write). That makes me think I may have a usb problem.

The reason I'm asking is I just ran the same tests again with sata 3gbps drives in a 6 disk raidz2 pool that gives me roughly 300megs per second and it still died in the middle of a write. And once again, this is something nas4free had no problems with. I'm thinking n4f doesn't bother with some sort of logging freenas does? My last clue to think along this path is on startup, n4f has sometimes 'lost' my config file. I'm thinking maybe it silently timed out on writing it out. My usb also only seems to be running at 1.0 speeds and the boot sequence says only one of my ports can run at 2.0 speeds (I still haven't figured out which one though).

Any input on this idea? Is this something possibly causing my problem or am I looking at an 'unlikely'?

 

[Michael Wulff Nielsen]

Seriously, just trash the usb stick and buy a new one. If you suspect that a $10 usb stick could be at fault, then just get a new good one.

But I would suspect that it's going bad.