Replication "up to date" but I see no files on target

[fullspeed]

If you knew where your data was, it's possible to replicate it locally and then change the freenas job. but since it's not clear and it seems like there have been other issues. my suggestion would be to destroy the target datasets and start over.

But, if you want to keep trying, provide the full output of "zfs list" and "zfs mount" (in code tags). You can replace any sensitive names with placeholder text, so we know what to call it.

Yeah I really don't want to re-replicate 145TB of data x 3 servers :/

Here is output of zfs list/zfs mount etc,. looks like the file system is there I just can't see it?

 

[depasseg]

I'm not at my computer but try to manually mount with something like "zfs mount storage01/dbbackups/blah-db180"

Alternatively, have you tried rebooting?

 

[fullspeed]

I'm not at my computer but try to manually mount with something like "zfs mount storage01/dbbackups/blah-db180"

Alternatively, have you tried rebooting?

First thing I tried is rebooting, mount does nothing. Replication has always been a nightmare in Freenas.

Hopefully its better in 10

 

[depasseg]

Storage layout is just 3 vdevs in a raidz3 all healthy no errors.

Here are my repl settings, I have another server with the same settings replicating to a target that I can see the data (on the same OS version as well)


View attachment 13391

This job doesn't match the datasets that you are showing in zfs list. There shouldn't be any other datasets below Storage01 on fs09 since db180 is being replicated to Storage01/ and db180 doesn't have any child datasets. Is fs09 in production or is it only used as a replication target?

What version of FreeNAS are these boxes?

 

[fullspeed]

This job doesn't match the datasets that you are showing in zfs list. There shouldn't be any other datasets below Storage01 on fs09 since db180 is being replicated to Storage01/ and db180 doesn't have any child datasets. Is fs09 in production or is it only used as a replication target?

What version of FreeNAS are these boxes?

The very latest version of 9.3, FS09 is a replicated target of FS10 and not written to directly. I wouldn't consider it production (unless FS10 failed)

DB180 is a child dataset of dbbackups on FS10 and recursively replicated it set to yes.

When I login to FS09 and click on "volumes" the GUI shows it identical to FS10 (aside from pending snapshot replication)

 

[depasseg]

Do you have any other replication jobs going to FS09? Something is wrong with the FS09 dataset structure. If you started with a blank storage01 pool and dataset and replicated db180 to it, then that is the only thing you will see. You won't see the dbbackups dataset.

 

[fullspeed]

Nope, these are all single purpose.. so 1 dataset replicated to one server, both identical.

I have 2 other servers with the exact same issue, setup the same way.

I have 1 server where I can see the data, the difference there is that I am replicating storage01/dbbackups on FS12 to storage01 on FS11. Everything else is identical.

Logic tells me that Freenas doesn't like the something-db180 extra sub/dataset, but why would it fully replicate it and keep it up to date but just not mount it? no errors no complaints.

 

[depasseg]

Can you post a screenshot of all your replication tasks on fs10 (the summary screen), and please use a color other than white to redact, it makes it really tough to figure out?

Freenas is fine with subdatasets. I have 2 replication jobs that handle a main dataset + ~45 child (and sub-child) datasets going to 2 different destinations. What's very strange is that you have datasets that shouldn't be there, so either you attempted something previously which caused them to be created and then fixed it, or you are still replicating incorrectly. Honestly, the simplest and safest thing at this point will be to wipe out the FS09 dataset and start from scratch. Create a new subdataset called "From-FS10" and replicate "storage01" and children to "From-FS10". And once that works, then fix the other 2 systems.

 

[depasseg]

As an alternative, detach storage01 from fs09 and then look at the contents of /mnt. It should be empty. Then import storage01 and see what it looks like.

 

[fullspeed]

I tried detaching and re-importing the volume, accepting a complete data loss scenario..

It failed importing so I rebooted and tried again and this is what I saw in the logs:

It was initially mounted now it won't even do that, I think this might be a bug.

 

[depasseg]

Feel free to open a bug.

 

[fullspeed]

I'm probably going to look at other solutions completely, I've had to resync so many times due to bugs. It isn't a big deal for most people but because of this particular issue I'll have to re-replicate 450TB of data. I originally had 2 servers now I have 8 so the work is split out but it's still a major problem.

I still haven't been able to find a solution to CIFS maxing out at 45MB/s either so overall i'm pretty frustrated. I love Freenas but it has been a real struggle in this environment at this scale.

 

[depasseg]

If you decide to file a bug, please post the number here.

As for CIFS performance, what speed are your CPU's? and do you have atime disabled?

 

[fullspeed]

If you decide to file a bug, please post the number here.

As for CIFS performance, what speed are your CPU's? and do you have atime disabled?

I think I tried that, I've tried basically everything.
CPU ~ http://ark.intel.com/products/37111/Intel-Xeon-Processor-X5570-8M-Cache-2_93-GHz-6_40-GTs-Intel-QPI

Here is my thread on it ~ https://forums.freenas.org/index.ph...-40-45mb-s-per-host-copying-to-freenas.28296/

Everything is piss poor slow but FTP and specifically FTP (not SCP/SFTP) and my servers are way overspec'd. I've all but closed the book on that one and given up.