FreeNAS capturing private information

[freenasisbest]

It hadn't been used in ages, see https://redmine.ixsystems.com/issues/41172.

Why are you looking?

I'm just getting 11.2 setup for the first time and when I logged into the new UI (looks freaking amazing, btw!!), my logs show that it is sending JSON event data to sentry.ixsystems.com.

It's easy enough to block out, I'd much rather have there be an option to disable it outright. =)

 

[jgreco]

I'm just getting 11.2 setup for the first time and when I logged into the new UI (looks freaking amazing, btw!!), my logs show that it is sending JSON event data to sentry.ixsystems.com.

It's easy enough to block out, I'd much rather have there be an option to disable it outright. =)

That's correct. It's more noticeable when your administrative workstation can't *reach* the Internet. Also then the new UI simply doesn't work. Which seems like a bit of a feature to me.

 

[Chris Moore]

That's correct. It's more noticeable when your administrative workstation can't *reach* the Internet. Also then the new UI simply doesn't work. Which seems like a bit of a feature to me.

I had not tried that. Are you saying that the new UI is unusable if the system can't reach the internet? That is a significant problem, not a feature, because none of the systems I manage at work have internet access.

 

[jgreco]

I had not tried that. Are you saying that the new UI is unusable if the system can't reach the internet? That is a significant problem, not a feature, because none of the systems I manage at work have internet access.

I haven't bothered to debug what exactly is going on.

NAS units here do not have direct internet access because they are only on storage and management networks, which do not have any sort of NAT support. They can reach our DNS, mail, netmon, paging, web cache, etc., resources which is sufficient. That's been that way for MANY years, and if I desperately want to do something like have them auto-update I can configure them for the cache proxy, which I sometimes do.

I've also moved a lot of our other hosts off the public Internet as well. It's a similar situation. This machine that I'm typing on can't ping anything past our network border. For web browsing, like typing this message, it has an RDP session to a VM that *does* have Internet visibility through a UTM, which is fairly safe.

So what I can tell you is this... if I connect up to ${nas}, enter credentials and click LOG IN, I get

https://${nas}/ui/sessions/signin

a blank white screen and it telling me that it's trying to connect to "sentry.ixsystems.com" for about ten seconds and then it craps out. If I log in with "LEGACY WEB INTERFACE" it's totally happy.

And, playing with it a little more, it looks like it's just FIrefox that's broken. I don't always try every possible combination of stuff when I'm busy.

 

[fracai]

So, what's the story on "telemetry" now? As far as I can tell the timeline has been:

- FreeNAS silently started uploading user information to iX servers.
- Users notice and complain
- iX points to documentation that mentions "daily telemetry", but doesn't define what that is.
- Several bugs are opened related to disabling, complying with GDPR, and disabling again the telemetry reporting.
- The only bugs that have been implemented are clarifying the tooltip, removing the opt-out option, and removing the opt-out field from the database.

So it looks like FreeNAS is back to silently uploading "telemetry" data. And I guess I'm back to questioning how much trust I can put in my own hardware.
It's bad enough that iX thinks it's OK to start reporting this data without warning to the user, but to put in a way to disable that reporting and then take it back out is a breach of trust.

BRB: blocking sentry.ixsystems.com at my DNS

 

[freenasisbest]

And I guess I'm back to questioning how much trust I can put in my own hardware.
It's bad enough that iX thinks it's OK to start reporting this data without warning to the user, but to put in a way to disable that reporting and then take it back out is a breach of trust.

I did take a few minutes to look into my initial concern about outbound JSON posts.

It appears iX setup a Sentry server to collect Javascript runtime errors in the new Angular user interface. Basically anything that shows in the "Developer Console" of your browser will be posted to their server. The subdomain suggests it's a server controlled by iX, not a third party. What is currently posted is:

- the browser/javascript exception string
- middleware software errors
- version/build of FreeNAS you are running
- your browser user agent
- your browser's public ip
- the address you use to access your FreeNAS server (192.168.1.200, or freenas.local, or not-so-well-known.my-company.com)
- the timestamp of all user interface buttons you clicked on leading up to the error, etc.
- the time since you reloading the browser window

The Sentry API makes it very very easy for iX to send much more than that. But that's not what they are doing.

I'm very new to FreeNAS so I haven't done a full-blown data audit concerning the backend BSD software, but these calls in the new (Angular) UI seems pretty innocent to me. They are basically trying to squash bugs. The new UI uses code found here in the Sentry API documentation. That all being said, they should probably provide an opt-out mechanism.

And yes, a certain amount of paranoia when dealing with SaaS/IaaS software is definitely considered healthy, from my experience.

[Edit 20190121: found that they pass middleware software errors into the web stack disguised as Javascript exceptions.]

 

[freenasisbest]

I had not tried that. Are you saying that the new UI is unusable if the system can't reach the internet? That is a significant problem, not a feature, because none of the systems I manage at work have internet access.

I can confirm that blocking the (potential) posts to sentry.ixsystems.com does not render the new UI unusable by itself. It will just timeout.

(On Firefox anyway) exceptions seem to happen frequently, especially when it comes to potential authentication issues, but I haven't experienced any actual breaks in user experience. My experience only, though.

Here is the custom rule in uBlock/adblock I used for testing:

Code:

# block freenas javascript error reporting
||sentry.ixsystems.com^

 

[fracai]

The Sentry API makes it very very easy for iX to send much more than that. But that's not what they are doing.

I appreciate you looking in to this and I'm sympathetic to what iX is trying to do here. That doesn't change at all my opinion that this has been handled incorrectly.
- It should be opt-in, not opt-out.
- If they insist on opt-out, it should at least be declared to the user when the feature first becomes active.
- Removing the feature after the first time this came up is a breach of trust. I was under the impression that telemetry could already be disabled. Then it turns out that that feature hasn't been active for some time. And now it turns out that they're sending data again. And I'm amazed that the GDPR ticket is still open.

 

[JoshDW19]

Hey everyone. I don't have an answer to this but let me talk with a few people and see if I can get you an answer.

 

[JoshDW19]

Okay, I talked to our development team about this and here's the explanation that I was given.

"That's used to help us find a lot of errors that come up in MW and UI when users may not file them. It's been pretty handy and AFAIK it's only sentry. If a user doesn't want this, they can touch the file /data/.crashreporting_disabled and it will be disabled on mw restart." I was told this option works for middleware only and that there's no option to disable it in the GUI yet. On that subject, they went on to say that "We do have a ticket open to make it possible to explicitly opt-out, that will probably be in 11.3".

If you have any other questions please let me know and I'll be glad to ask the devs!

 

[fracai]

Is there a command to restart the middleware? Or would that require a reboot?
Thanks for looking in to this.

 

[danb35]

"We do have a ticket open to make it possible to explicitly opt-out, that will probably be in 11.3".

So you're collecting user data without their knowledge or consent, and with no way to opt out (leaving aside that it should be opt-in, not opt-out). Stipulating that you have perfectly good reasons for wanting the data, that you aren't doing anything remotely untoward with it, that none of what you're collecting could compromise anything user-side, and that you'll never have a data breach (although the necessity for all those stipulations should itself give you pause), do you have any idea how bad the optics of this are? Especially today?

GDPR is rightly decried as massively overbroad. But then we see software developers doing stuff like this, and the reasoning for it becomes somewhat more evident.

 

[jgreco]

GDPR is rightly decried as massively overbroad. But then we see software developers doing stuff like this, and the reasoning for it becomes somewhat more evident.

Aaaaaaand for everyone who puzzles at policies such as "administrative workstations shall not have Internet access" ... a classic example of why.

 

[anodos]

Is there a command to restart the middleware? Or would that require a reboot?
Thanks for looking in to this.

service middlewared restart

 

[seanm]

JoshDW19, can u give us the link to the ticket please?

 

[JoshDW19]

I can understand your concerns, and I think they're completely valid in today's day and age. My understanding though is that this is for error reporting in the software only and does not include any personally identifiable data which is what the GDPR concerns itself with.

Either way, our goal is to be transparent when we collect data and we're going to make sure that we address this promptly. I just spoke with Kris, and he's going to make sure this stays at the front of the dev team's radar. The dev team also said that work is already in progress to create a welcome wizard that will prominently show the user the options to disable telemetry (in the GUI) without issuing a middleware command. I'll also check with our legal team and make sure that our disclosures properly indicate how we collect data and use it to improve our software.

 

[freenasisbest]

Aaaaaaand for everyone who puzzles at policies such as "administrative workstations shall not have Internet access" ... a classic example of why.

So true!

This should also be a great example why you should always disable automatic updates from third party software. Even though it's against convention some developers still think it's acceptable to overhaul everything in a MINOR release.....

Today I am pleased to announce that FreeNAS 11.2-RELEASE is now available! While this is a point-release from 11.1 to 11.2, it’s actually much, much more than that -- it is indeed a MAJOR release. FreeNAS 11.2 marks the culmination of a nearly 18-month development cycle, completely overhauling the existing legacy FreeNAS API and user interface subsystems.

I mean, credit where credit is due, 11.2 is super sexy and major props to the dev team, but my trust in iX to "do the right thing" went waaaaaay down. In my ongoing search this weekend I notice the method in which they funneled their 'middleware' error reporting through to the user's browser. Kinda elegant. Quite sneaky. =)

 

[jgreco]

I mean, credit where credit is due, 11.2 is super sexy and major props to the dev team, but my trust in iX to "do the right thing" went waaaaaay down. In my ongoing search this weekend I notice the method in which they funneled their 'middleware' error reporting through to the user's browser. Kinda elegant. Quite sneaky. =)

Well, the problem is that there's a whole new generation of developers who have grown up in an era that's completely different than what I did.

When I write C code, I'm thinking in terms of the assembly code it generates, and while I don't normally go poking around in the low level assembler or actual machine code these days, I understand it. Likewise, I generate my own HTML (not with someone else's authoring tool) and I understand what it does.

We have a generation of developers (not necessarily the ones at iX, but also not necessarily NOT) who have instead grown up with building blocks of Perl or PHP or Node.JS modules and who have little understanding of what goes in inside these things, sometimes, much less how that eventually translates into the low level stuff. Likewise, the norms for software of 25 years ago are very different than modern norms.

I don't buy into a lot of the modern crap. I don't do cloud. I generally don't rely on services by others. This is not a common mentality in this era, so while I may not approve, and think that some education is warranted here, I at least understand how this kind of thing comes to be.

 

[rvassar]

I don't buy into a lot of the modern crap. I don't do cloud.

I've been doing Cloud for the last couple years. My specialty used to be performance / stress testing & tuning. Cloud play hired me to do performance & stress testing of their cloud management offering... That lasted about two months. The AWS bill for a stress test arrived, and well... I don't get to do that anymore. ;)


I'm also quite proud of my personal data collection manipulation... I had Facebook convinced I'm in the logging or mining industries.


The problem iX faces of course is at some point you have to face the kernel crash dump problem. Anything... Anything at all stored on that NAS could be in that dump. Maybe I'm reading too much into it, but.. How do you comply with GDPR without exchanging licensing contracts for every crash dump?

 

[danb35]

does not include any personally identifiable data which is what the GDPR concerns itself with.

I understand that GDPR is very broad in what it considers personal information, but as an American with no business contacts in .eu, it's mainly a matter of curiosity for me--I'd expect that you actually need to comply with it though. But my point was less about strict compliance with that legislation, than about the situations/concerns that lead to such legislation (poor as it is) being enacted. Again, even stipulating everything I previously mentioned, you're collecting data from your end users without their knowledge or consent, and with no way to opt out. That your devs found the time to code in the data collection apparatus, but not time to code in a checkbox to turn it off (or, really, to turn it on, which should be the case) doesn't speak well of your "do the right thing" bona fides.