zfs send error "Out of Memory"

[SCSPhil]

Ok, so in a couple of different scenarios lately I've been getting this error when trying to do a resumable send. It's happened with both an incremental, and a whole dataset send. The first time it happened was a local send, so I just assumed that you shouldn't do -s on the receive command when going from volume to volume on the local machine. Weird, but only a minor annoyance as why would I really need resumable sends when done locally? Maybe in a freak accident, but really no biggie.

However, I just got this error when doing it over the wire. This is far more concerning, and something I need solved.

Anybody else get this? Any way to determine why it's happening or what can be done to mitigate it? And no, neither pool is full, or anywhere near full. What's weird is I did a resumable send earlier today that was a much larger incremental gap than the one I just tried, and it went through normally.

 

[dlavigne]

FreeNAS build version (System -> Information)? Hardware specs?

 

[SCSPhil]

This has happened on three dissimilar hardware setups, well I guess two of them are pretty similar. The similar ones were Xeon E5620, dual CPU based systems with 90 Gigs of ECC memory, and uhh, I forget which SAS controller exactly; 9211-8i in IT mode I think. The third system is just a real basic small device, pentium processor based.

Versions I've seen this on:
9.10.2-U6
9.11.0-U4
and whatever the latest build is (I updated last night in an attempt to fix this). I don't know what build it was before this.

 

[dlavigne]

Please create a report at bugs.freenas.org. Include a debug from the most updated one (System -> Advanced Save Debug) and post the issue number here. Note that the bug will be hidden from others until a dev has a chance to review the debug.

 

[SCSPhil]

Should I include the debug for the sending or the receiving machine? I looked through the logs in that debug under ZFS and it doesn't seem to mention anything. Though it does mention when I successfully send things, without the -s. Seems like every time the recv command is made with -s on any of these three machines it errors out.

Honestly, I'm not convinced this is a "bug" or something I'm doing wrong, but I really can't think of anything. I've been syncing using this method for almost a year and a half at this point on one of these pools and never seen this error. But recently had to do a full resync, which involves sneakernetting a drive up to the main host, which is when I first saw the error.

Edit= I've just had an epiphany. This is happening on volumes created by freeNAS recently when they are set to resume. Is there a property on the pool itself that would cause this kind of error that freeNAS has recently decided to change? The volume I was successfully sending to previously became desynced and I deleted it and remade it, but it was made last year some time.

 

[dlavigne]

Start with the debug on the sending machine. Include your steps to reproduce. Thanks!