Impending Google Photos policy changes

[rvassar]

Can you elaborate please? I'd like to know more. I've been looking at various open source mail servers such as Mailcow as alternatives to Gmail.

Nextcloud addresses most of my privacy concerns. It has a built-in calendar and contacts server; access to in-house cloud-based office suites such as OnlyOffice and Collabora; photo/video and file storage, and a host of plugins that extend its functionality. It has a mail client. It just needs to interface with an in-house mail server.

This just leaves Google Maps and Waze as the only Google dependencies that still remain for me. I'm not sure there are viable open source alternatives to these?

Well... No, I'd have to write a book. Which I may do someday. I have 16+ years professional experience in telco scale mail servers, and 9 more years in storage R&D and DevOps, and I'm still learning things. Running my own mail server is just kind of a hobby, but it's getting increasingly difficult to do. The Internet is not the friendly academic network I started out on. Yes, you can rent a virtual Linux server and set it up using open source software for $10 a month. But the know-how is a mountain of knowledge to climb, and the major players are actively hostile to your doing this.

 

[Jailer]

But the know-how is a mountain of knowledge to climb, and the major players are actively hostile to your doing this.

This is exactly what has kept me from trying to set up my own mail server. There's plenty of tutorials and open source options available but if you get it wrong the consequences are pretty steep.

 

[jgreco]

I've been running e-mail since the '80's, so my opinions may be slanted somewhat, but I kinda-agree and kinda-disagree with @rvassar

This is exactly what has kept me from trying to set up my own mail server. There's plenty of tutorials and open source options available but if you get it wrong the consequences are pretty steep.

Well, there's a lot of crap that's been piled on top of plain old SMTP that makes it more complex than it ought to be, but the biggest issue is deliverability - the term for whether or not other mail sites will accept mail from you. Some will "accept" mail and dump it, some will "accept" it and junkfolder it, and some will outright refuse your mail.

If you are sending from a residential broadband connection, or from a scuzzy cloud vendor (Amazon, OVH, Digital Ocean, etc) where lots of spammers have spun up VM's and spammed and then vanished, you will find poor-to-nil deliverability.

You really need to find some way to send your mail out, and this could be as easy as forwarding the mail to your ISP's outbound mail servers for delivery. Not all ISP's have great deliverability either, but it is going to be 10,000% better than trying to originate e-mail to other sites from a resi broadband. You can also get an account with certain companies that specialize in sending non-spammy mail on your behalf. Mailchannels comes to mind, but their pricing is not designed for individual use.

The rest of the puzzle is really in your hands. It isn't rocket science to set up Postfix and rspamd (or SpamAssassin) and Dovecot, maybe with a Roundcube webmail frontend if you want to do that. Don't think that it is EASY, either, and there are a lot of sharp edges and jaggy bits to bleed upon, but at the end of the day, you can create a secure e-mail system that speaks SSL to the rest of the world, stores your mail securely, and communicates with all your devices over SSL. You can even use your own local CA for your IMAP/POP/MSA sessions to create a mail system that cannot be spoofed by someone with a rogue certificate, making for a highly secured mail environment within your domain. Some of us throw things like AV scanning and other fun stuff in there too.

So, please, "consequences are pretty steep" is perhaps overly dramatic. If you enjoy a technical challenge, it is one of the more difficult things to do on the Internet these days, simply because there are a lot of moving pieces and a lot of them were heaped on after the original design of SMTP. You do need to make sure that you are properly limiting and securing your setup, but once you've done that, any consequences of screwing it up tend to be limited to damaging your own mail.

 

[rvassar]

I've been running e-mail since the '80's, so my opinions may be slanted somewhat, but I kinda-agree and kinda-disagree with @rvassar

Well, there's a lot of crap that's been piled on top of plain old SMTP that makes it more complex than it ought to be, but the biggest issue is deliverability - the term for whether or not other mail sites will accept mail from you. Some will "accept" mail and dump it, some will "accept" it and junkfolder it, and some will outright refuse your mail.

This is one of the key things. Deliverability & site reputation management. It sad that there are people that have taken the incompetent route in blocking spam by implementing accept and drop. It's stretching the RFC's a bit too thin in my opinion...

If you are sending from a residential broadband connection, or from a scuzzy cloud vendor (Amazon, OVH, Digital Ocean, etc) where lots of spammers have spun up VM's and spammed and then vanished, you will find poor-to-nil deliverability.

When I first started you could still run on a residential DSL line. You had to apply for a port 25 exemption and get a static IP, but provided you behaved yourself, it worked. But you had no control over your IP address PTR record. Once the spammers started compromising systems and running "zombies", anything with a "pppoe" or "dsl" in the PTR record got dropped. Most ISP's no longer offer port 25 exemptions. Amusingly, even today, my single most effective anti-spam rule is:

Code:

/^unknown$/   HOLD

It denies delivery from naked IP addresses, and in compliance with RFC, informs the sender that the message will be held. I then have automation that parses the hold queue and eliminates the easy stuff.

But having a reputable IP address isn't even enough these days. You need to have your DNS records completely correct. No shortcuts, no illegal CNAME pointers, "acceptable" reverse-PTR, proper SOA glue, etc... Then you need to publish an SPF record and set up DKIM. Email and DNS are inextricably intertwined, so you really need to manage both. Most people in IT these days never touch DNS. It's a service provided to them, and they may have some understanding about how it works, but there's a bunch of subtle details.

So, please, "consequences are pretty steep" is perhaps overly dramatic. If you enjoy a technical challenge, it is one of the more difficult things to do on the Internet these days, simply because there are a lot of moving pieces and a lot of them were heaped on after the original design of SMTP. You do need to make sure that you are properly limiting and securing your setup, but once you've done that, any consequences of screwing it up tend to be limited to damaging your own mail.

Agreed. You can set it up over a weekend. But don't expect to be successful for several months.

 

[jgreco]

But having a reputable IP address isn't even enough these days. You need to have your DNS records completely correct. No shortcuts, no illegal CNAME pointers, "acceptable" reverse-PTR, proper SOA glue, etc... Then you need to publish an SPF record and set up DKIM. Email and DNS are inextricably intertwined, so you really need to manage both.

It's even worse than that. Some of the big sites will judge you harshly simply for having low mail volume to them. I run a local Public Access UNIX system and have the region's oldest continuously operating mail system. I had always taken a certain amount of pride in being a good net neighbour, and we rarely if ever had spam problems. One day, maybe half a decade ago, I started getting delivery problems with (${BIG-SITE}), couldn't see any reason for it, used some backdoor industry contacts, and was blithely told by some jerkarse in their mail operations that sending too *few* emails scored badly with their system, and that I had to live with it.

Not finding that acceptable, we created some accounts on their service and started pumping a low volume of mail at them. Problem resolved, plus, bonus, we get closed loop confirmation that emails are being delivered. It feels dirty to me, but perhaps that's because I'm an old-timer.

Most people in IT these days never touch DNS. It's a service provided to them, and they may have some understanding about how it works, but there's a bunch of subtle details.

DNS is so much fun though. I think we made a mess of that too but in different ways.

Paul Vixie's gig and mine are cagemates out in Ashburn. I don't think I've ever actually had to ask him a question about DNS though.

Agreed. You can set it up over a weekend. But don't expect to be successful for several months.

I feel that's a bit pessimistic. The biggest issue is still deliverability, and you can get around that by using someone else's sending infrastructure.

 

[rvassar]

The site dropped as I was posting this...

One day, maybe half a decade ago, I started getting delivery problems with (${BIG-SITE}), couldn't see any reason for it, used some backdoor industry contacts, and was blithely told by some jerkarse in their mail operations that sending too *few* emails scored badly with their system, and that I had to live with it.

I can't say I've had that problem, but I ended up 1st-degree from a bunch of (${BIG-SITE}) staff. Maybe all those extra monitors and pizza box upgrades paid off. Given the way some of them are run now, I don't doubt it though, and I would consider this an example of "major players are actively hostile to your doing this". Some of them like reading your email. It helps them target ad's at you.

I feel that's a bit pessimistic. The biggest issue is still deliverability, and you can get around that by using someone else's sending infrastructure.

Well... No, don't get me wrong, it's not that I'm being derogatory, consider it more of a planning item. I'll set up a VPS and park it for a month just to rehabilitate the IP address, and get the security dial'ed in. Drop a web server on it and see if there's any zombie DNS records still pointing at it, etc... That's more what I was suggesting. You can script up a mail server install in Ansible or Chef, and have it up and running as fast as your DNS TTL will permit. But if you're renting IP's that have been rode hard and put away wet, you'll be disappointed, possibly for weeks...

 

[Patrick M. Hausen]

You need to have your DNS records completely correct. No shortcuts, no illegal CNAME pointers, "acceptable" reverse-PTR, proper SOA glue, etc...

I wonder if anybody still groks classless reverse delegation these days. RFC 2317 ...

 

[jgreco]

I wonder if anybody still groks classless reverse delegation these days. RFC 2317 ...

Of course. I raise you an RFC4183 and DeGroot...

 

[Spearfoot]

I've been running e-mail since the '80's, so my opinions may be slanted somewhat, but I kinda-agree and kinda-disagree with @rvassar



Well, there's a lot of crap that's been piled on top of plain old SMTP that makes it more complex than it ought to be, but the biggest issue is deliverability - the term for whether or not other mail sites will accept mail from you. Some will "accept" mail and dump it, some will "accept" it and junkfolder it, and some will outright refuse your mail.

If you are sending from a residential broadband connection, or from a scuzzy cloud vendor (Amazon, OVH, Digital Ocean, etc) where lots of spammers have spun up VM's and spammed and then vanished, you will find poor-to-nil deliverability.

You really need to find some way to send your mail out, and this could be as easy as forwarding the mail to your ISP's outbound mail servers for delivery. Not all ISP's have great deliverability either, but it is going to be 10,000% better than trying to originate e-mail to other sites from a resi broadband. You can also get an account with certain companies that specialize in sending non-spammy mail on your behalf. Mailchannels comes to mind, but their pricing is not designed for individual use.

The rest of the puzzle is really in your hands. It isn't rocket science to set up Postfix and rspamd (or SpamAssassin) and Dovecot, maybe with a Roundcube webmail frontend if you want to do that. Don't think that it is EASY, either, and there are a lot of sharp edges and jaggy bits to bleed upon, but at the end of the day, you can create a secure e-mail system that speaks SSL to the rest of the world, stores your mail securely, and communicates with all your devices over SSL. You can even use your own local CA for your IMAP/POP/MSA sessions to create a mail system that cannot be spoofed by someone with a rogue certificate, making for a highly secured mail environment within your domain. Some of us throw things like AV scanning and other fun stuff in there too.

So, please, "consequences are pretty steep" is perhaps overly dramatic. If you enjoy a technical challenge, it is one of the more difficult things to do on the Internet these days, simply because there are a lot of moving pieces and a lot of them were heaped on after the original design of SMTP. You do need to make sure that you are properly limiting and securing your setup, but once you've done that, any consequences of screwing it up tend to be limited to damaging your own mail.

I run a small Linux-based mail/web server at DigitalOcean for cheap -- roughly $12 a month. You're absolutely right that setting up all the required bells and whistles -- Apache, Postfix, certificates, DKIM, DMARC, fail2ban, firewall, etc. -- isn't a trivial undertaking, though it's really "No big step for a big stepper".

Given that DigitalOcean is, as you pointed out, well-known for allowing unsavory users, who would you suggest as an alternative provider? AWS is too pricey for my use-case.

 

[rvassar]

Given that DigitalOcean is, as you pointed out, well-known for allowing unsavory users, who would you suggest as an alternative provider? AWS is too pricey for my use-case.

I've been running at Linode for more than a decade. But similar problems and prices. They do seem slightly better than DO on the popup spammers, but this could be an illusion simply based on my tiny user base. I do have automation harvesting/blocking roughly 60 DO IP addresses a day. Linode IP's, few enough I haven't taken note, but not zero.

(Employment disclaimer) I am currently playing around with Oracle Cloud free-tier, but not sure I want to make that jump. Though it sure was nice to have a 129.146.x.x. IP address again.

 

[rvassar]

I might add... If you haven't looked lately, AWS lightsail instances now allow static IP's, which helps on the cost.

 

[jgreco]

I run a small Linux-based mail/web server at DigitalOcean for cheap -- roughly $12 a month. You're absolutely right that setting up all the required bells and whistles -- Apache, Postfix, certificates, DKIM, DMARC, fail2ban, firewall, etc. -- isn't a trivial undertaking, though it's really "No big step for a big stepper".

Given that DigitalOcean is, as you pointed out, well-known for allowing unsavory users, who would you suggest as an alternative provider? AWS is too pricey for my use-case.

That last sentence is the kicker. The same thing that draws you is drawing undesirables, and the lack of any substantial dollar amount and the "joyous nature of the ephemeral cloud" makes long-term contracts not-a-thing. The community no longer punishes bad behaviour in any substantial way, and large providers not caring and emitting crap from both their IP ranges and mail servers means that you cannot safely block them without significant collateral damage.

I run several ASN's, both for my companies and on behalf of clients, with pre-ARIN-delegated IP space. Some of it is still virgin, meaning never actually used on the Internet, and most of it is lightly used, never by spammers. Several CAUCE members have hosted here and SpamCop used to get image hosting through us, until they were bought out and funded better. I still operate a network that operates on the "if you are seeing badness from us, we are happy to make it stop" principle that was common on the Internet before Eternal September and the commercial "it's okay as long as they pay" Internet came into being.

The solution that works best for *me* is for my businesses to run our own private cloud. With about half a terahertz of CPU and more than 2.5TB of memory, and we've been making the switchover to SATA and NVMe flash (much of it consumer grade, like the dozen 1TB Evo 960's we just bought) over the last few years, the big trick is simply to keep finding the sweet spot for compute gear with a long remaining tail coming out of data centers, finding inexpensive rack space, inexpensive bandwidth, and using our own ASN's.

Just like you do not pay NetApp or EqualLogic ridiculous amounts for your NAS storage, and roll your own instead, I've rolled an inexpensive cloud that only costs several thousand dollars a year in opex to run. By transforming what most people pay as cloud opex into one-time capex costs, I get a lot more bang for my buck. If I go to AWS and ask how much for 100 small VM's and 100TB of storage, I get a ridiculous answer like $2K/month.

Now it used to be that it was possible to sell services to clients, which I've been happy to do as one of my businesses is a service provider, but the cloud pricing model of selling $5/month VM's means that the only way to do that sensibly is through 100% automation and mostly ignoring the potential abuse issues, and that just isn't worth it. This is the long way around to showing you that your "too pricey for my use-case" is the thing that's really the problem.

 

[Spearfoot]

That last sentence is the kicker. The same thing that draws you is drawing undesirables, and the lack of any substantial dollar amount and the "joyous nature of the ephemeral cloud" makes long-term contracts not-a-thing. The community no longer punishes bad behaviour in any substantial way, and large providers not caring and emitting crap from both their IP ranges and mail servers means that you cannot safely block them without significant collateral damage.

I run several ASN's, both for my companies and on behalf of clients, with pre-ARIN-delegated IP space. Some of it is still virgin, meaning never actually used on the Internet, and most of it is lightly used, never by spammers. Several CAUCE members have hosted here and SpamCop used to get image hosting through us, until they were bought out and funded better. I still operate a network that operates on the "if you are seeing badness from us, we are happy to make it stop" principle that was common on the Internet before Eternal September and the commercial "it's okay as long as they pay" Internet came into being.

The solution that works best for *me* is for my businesses to run our own private cloud. With about half a terahertz of CPU and more than 2.5TB of memory, and we've been making the switchover to SATA and NVMe flash (much of it consumer grade, like the dozen 1TB Evo 960's we just bought) over the last few years, the big trick is simply to keep finding the sweet spot for compute gear with a long remaining tail coming out of data centers, finding inexpensive rack space, inexpensive bandwidth, and using our own ASN's.

Just like you do not pay NetApp or EqualLogic ridiculous amounts for your NAS storage, and roll your own instead, I've rolled an inexpensive cloud that only costs several thousand dollars a year in opex to run. By transforming what most people pay as cloud opex into one-time capex costs, I get a lot more bang for my buck. If I go to AWS and ask how much for 100 small VM's and 100TB of storage, I get a ridiculous answer like $2K/month.

Now it used to be that it was possible to sell services to clients, which I've been happy to do as one of my businesses is a service provider, but the cloud pricing model of selling $5/month VM's means that the only way to do that sensibly is through 100% automation and mostly ignoring the potential abuse issues, and that just isn't worth it. This is the long way around to showing you that your "too pricey for my use-case" is the thing that's really the problem.

You swing a big bat in this arena; I'm just a software engineer nearing retirement age, and running a 'hobby' mail/web server for the fun of it.

In the days when I was a partner in a startup, we co-located our servers at Navisite's San Jose data center. The expense was well worth it. But now? No way! I have plenty of hardware capacity for running the few server instances I want; and I once ran these from my home, before moving to DigitalOcean. What I lack is the infrastructure, particularly networking, but also backup generators and all the other stuff you mentioned that goes into making a truly robust system.

 

[Heracles]

Here, I built my own cloud for privacy. I interconnected a few residential cable modems with pfSense firewalls. I distributed my FreeNAS on 2 sites and HAProxy is doing the load balancing.

I also created a free-tier VM in AWS. To avoid paying for their VPN and a static IP, I installed OpenVPN on that box and it calls back home on 2 of my firewalls. I run my HA Pi-Hole instance from that one. That way, I have 3 ad-blocking DNS servers in normal time and no single incident will drop me below 2. Even 2 major and unrelated incidents will leave me with still 1 PI-Hole available from anywhere in my MAN.

Should I wish to do HA on my cloud, I just need a TCP-level load balancing HAProxy in the cloud. That one will monitor both of my entry points and will connect to the secondary should the primary fails. Being TCP level, TLS will not be decrypted there and the cloud provider will remain blind to my own activity.

Such a light weight load can be run for free in AWS. Dynamic DNS will manage the dynamic IP address I will receive from AWS. Should I wish to start selling cloud services and hosting more than my own stuff, then only that free frontend in AWS would need to be switched to a more powerful profile that would cost me $$$. But again, should I commercialize my services, up to me to factor in these charges and include them in the price.

Another free option would be to use Cloudflare as a web frontend for my cloud. The minus would be that they would do it at HTTP level and for that, would decrypt my cloud trafic. Still, it would be another free and commercial-grade option. Also, I say free because I already host my DNS there but consider what they charge for that, it is almost free as well...