It used to be possible to run your own SMTP server, inbound and outbound, from home. This was so badly abused by spam that port 25 is blocked almost everywhere.
Domestic systems tend to be in configurations that make it hard to accept inbound TCP connections. You could serve SSL on a random port and open a port using UPNP, and it will work most of the time.
It's a difficult circle to square. The most trustworthy system is one you administer yourself and manually inspect all updates, but in practice the amount of work required makes that almost impossible. If you allow the OEM to do updates they can compromise you. If you don't do updates you end up vulnerable to exploits.
Even if your home provider allows you to accept SMTP on port 25, you're going to have a hell of a hard time with outbound delivery. The big mail providers give very low reputation ranking to e-mails originating from consumer connections. Even commercial connections w/ hosting providers need to be "warmed up" for a period of time to get their sending reputation to an appropriate level where they can expect reliable delivery without automatic spam-boxing.
What I'd rather see is a company go into partnership with the big providers to certify individual senders at home. The idea is that you'd put up some kind of bond (a few hundred USD?) to help ensure that you won't be sending spam. In exchange for your bond, the company vouches for your IP to Google/Yahoo/Hotmail/etc and you're allowed to send your own email. After some period of use w/o any spam complaints, you get your bond money back.
One option is to pay for mail hosting from a "big mail provider" or hosting company and simply use it only for relaying/sending outbound mail but not for incoming. Assuming 25/TCP inbound is not blocked by your ISP, you can point the MX RR at yourself and handle incoming mail on your own.
I work for an ISP and run my own personal mail system. This (25/TCP outbound being blocked/restricted) is, as you mention, one of the biggest hurdles for an average person to be able to run their own mail system (especially on their home connection). I've debated offering up a cheap "outbound relay only" service for fellow geeks but I would have to (initially, at least) institute some pretty low rate restrictions in order to help prevent any spam issues. That's the biggest thing stopping me from doing it so far.
one thing you'd lose that way is the direct secure delivery. not that it'd matter if you send it to someone on google, but you can't trust the relay server not to tamper with the mail
unlikely, but if you are the type to run your own mailbox, probably that matters to you
Good point, although if my options were either a) handle inbound myself, rely on a third-party for outbound or b) rely on a third-party for inbound and outbound, I'd pick the latter (although not everyone would, of course).
Obviously, you'd have to trust whomever you're using for relaying not to mess with your mail.
Another option could be a VPN connection to bypass ISP-level packets filters.
It would be a nice feature add-on if Own-Mailbox users could add a special TXT record to the DNS that identifies them as Own-Mailbox users. The presence of this record would tell Own-Mailbox that it could do a direct delivery to that user and get the increased end-to-end security. For all other outbound mail, it could use the provider's SMTP relay.
> you're going to have a hell of a hard time with outbound delivery.
Been there done that. Takes a good while to build up reputation, but it's possible, even without dkim signing or anything like it. Also since it's my personal email, my policy has always been "if you can't receive it because of your mailserver, sucks to be you -- you're using that service so it's on your end."
Really? So if my email sends get stuck in a client's Gmail spam queue, sucks for me or them? Sucks for me a lot more and I'll move to a hosted service instead that can guarantee email delivery.
I mentioned it's my personal email, not really meant for work (though I did use it for that also).
But technically yes, if they choose to trust Google with delivering the email and Google doesn't deliver, it's a problem on their end. Setting up your email in a way that a particular service (in this case Google) accepts it, doesn't scale.
How do you contact family and friends who aren't savvy enough to setup their own email server? What about replying to job interviews? It would really suck if you didn't get the job because the hiring manager never received your email.
In my few years of using this, and I'm by no means an expert on email servers, I have rarely had issues beyond ending up in Hotmail's and Gmail's spambox. Never outright rejections I think. Google's spamfilter is the most annoying one, followed by Hotmail; everyone else usually works pretty much fine.
Ah, blacklists. One of the companies I work for is on the RFC-Clueless blacklist. Gmail handles the business mail, but something was screwed with the abuse@ and postmaster@ mailboxes, so RFC-Clueless listed the domain years ago. Eventually we started running into trouble with mail admins who used RFC-Clueless. I got gmail involved, escalated to third-level support, and got those two RFC addresses fixed.
I've submitted this fix to RFC-Clueless several times over the past few months, but it's abandoned. Every request times out after a few weeks. Great, we're on a blacklist whose raison d'etre is "blacklist those not following the rules", and the bastards don't even follow their own declared workflow.
man, no-one is going to jump on this bandwagon if it requires a several-hundred dollar outlay. Considering the massive volume required for spamning to be a worthwhile venture, wouldn't 50 bucks suffice?
Maybe the big mail providers need to update their strategy regarding automatic spam-boxing?
Most personal mail servers are not going to be sending mass emails. An algorithm that auto-spam-boxes mail from domains it's never seen before (or only receives a small trickle from) is ignorant.
Most personal mail servers won't, but most sources of e-mail sending un-authenticated e-mail direct to the target mail servers from consumer internet connections are spam these days because almost no consumers does that - instead most e-mail sent that way come from malware.
An algorithm that auto-spam-boxes mail from address blocks used for consumer connections or from "new" servers is not ignorant - it is taking into account years worth of evidence that it is an effective counter-measure against spam.
I'm saying that as someone who would like to run my own home mail server, and someone who is constantly battling to make sure our (100% opt in) bulk mail servers at work don't get blocked by overzealous spam filters - I wish it wasn't this way, but it is.
Do you have any data on the percentage of spam email sources (of all email sources) that send 1-10 emails per day?
I'd expect that for a spam source to be effective, it'd need to send many more messages. Some spammers may try to run many low-rate sources, but you could likely do statistical analysis on the content of the messages to differentiate the spam sources from the non-spam sources.
Further, allowing the occasional doesn't-look-like-spam-but-is-actually-spam message (if such a thing exists), isn't the end of the world. The user can mark it as spam and have the source blocked forever.
The real spam problem that needed to be addressed was mailboxes being completely flooded with unsolicited messages. So flooded that it becomes impossible to read or even find the legitimate messages.
> Do you have any data on the percentage of spam email sources (of all email sources) that send 1-10 emails per day?
Do you have any method by which all the different mailservers worldwide can successfully collaborate to determine precisely how many emails a particular mailserver sends per day?
You're proposing that mailservers with a low emails-per-day metric be trusted. How is that metric established? How does everyone agree on it? Do we have to implement a new protocol in addition to SMTP in order for this to be viable?
The most obvious response to this is "well each service provider can just count the stats internally and use that" but it's a non-starter. That means that Google with their, what, 1000 email servers (or more) now has to figure this out. And every other service provider too.
Finally, given all the service providers out there, and the total number of "consumer" internet connected computers, giving all the computers a free 1-10 email pass means that spam is back to the worst of 2000-2010 levels. Here's some math:
Let's suppose that there are 20 big players in the mail provider category, Google, Yahoo, Microsoft (hotmail, office365, etc), AOL, comcast, at&t, time warner, etc. 20 seems like a good number.
Since these systems can't be made to communicate with one another easily about the sent-count of every IP address in the world, they just do it locally. That means that the virus that infected your computer and turned it into a spam-bot gets:
10 free messages * 20 major providers * 93mm computers = 18 billion spams per day
And that's just from the computers in the US. Once you take this global, you're probably talking at least 100 major providers and 300-500 million computers. Then you're talking something like a trillion spams a day.
Seems like a reasonable idea until you look at how it would actually be implemented. Then it doesn't seem so great.
Modern spam filtering is not purely based on white/black-lists. Statistical analysis of the content is performed.
You don't need to auto-spam-box sources that are sending low volume emails to the same set of addresses, especially when the messages don't get categorized as spam by the content analysis.
You don't need the kind of global coordination you are talking about. Further, whitelists don't solve the problem. Under your scenarios, you can just as easily receive low volume spam from millions of fake accounts at the big providers. You're just relying on the provider to solve your spam problem.
Sorry, but these claims just demonstrate that you have not operated a large volume mail servers.
Yes, most of us do need to automatically block such sources because content analysis does not do a good job, and experience demonstrates that the vast majority of such messages are still spam.
> You're just relying on the provider to solve your spam problem.
Yes, we are. And you should be extremely happy they put in the effort they do, or your e-mail would be completely useless.
> Modern spam filtering is not purely based on white/black-lists. Statistical analysis of the content is performed.
Not purely, no, but content analysis is typically performed after whitelist/blacklist checks. If a host connecting to my mail servers is on a blacklist, there is no content analysis because the mail will be rejected before it gets that far.
It's assuming that they're all infected by some malware. Of course not all of them are, but realistically there are also thousands of providers, not 20.
How would my mail server know that the sender is a low volume source?
It's trivial to randomly cycle through the domains you're sending to in order to deliver at a low rate to any single domain.
> Further, allowing the occasional doesn't-look-like-spam-but-is-actually-spam message (if such a thing exists), isn't the end of the world.
It is when you allow the occasional such message from each of tens of millions of compromised machines worldwide.
Recently we had to block inbound mail from most of Ukraine because hundreds of thousands of IP addresses belonging to Ukrainian ISPs were used to spam us. I started manually blocking /24's, but couldn't keep up, so we moved to blocking /16's, and even that was going too slowly to avoid affecting our systems.
Another time recently we had to block a particularly notorious Asian ISP that basically don't care that the millions of addresses in their net bocks all gets hijacked for spamming purposes (or quite possibly they do care as a result of being paid to look the other way).
This is business as usual when you operate anything but the smallest mail servers where you might get lucky and not get detected by spammers.
We still receive spam sent from malware addressed at domains of clients we haven't hosted for half a decade, that's been picked up from peoples mail clients....
This is why a lot of people decide it's easier to just blanket block all dynamic IPs. It costs a lot of money to stay on top of spam, and a lot more if you decide to try to avoid collateral damage.
> The real spam problem that needed to be addressed was mailboxes being completely flooded with unsolicited messages. So flooded that it becomes impossible to read or even find the legitimate messages.
The practices you complain about are the reason why the spam problem is now contained sufficiently for e-mail to still be usable.
Sounds like you can map out the compromised addresses and selectively block them.
Avoiding auto-spam-boxing new domains (as opposed to IP addresses), would be helpful. Any strategy that attempts to leverage multiple machines would still need to register and maintain many domain names, making such a spam strategy costly and more technically difficult. Further, an easy check for malicious users is to examine the websites at the domains suspected of spamming. Or even just WHOIS info.
If spam detection requires human intervention, it is too slow. Domain reputation needs to converge within 3-5 minutes---even 30 minutes means you have lost the spam race.
Many ISPs add their own consumer/residential netblocks to some blacklists (such as the Spamhaus PBL [0]) in order to help prevent spam from originating from the PCs of compromised users. Obviously, it is a receiver's choice whether to use such lists or not (although many/most? large mail providers do) but rejecting mail from these netblocks does result in a very noticeable decrease in spam.
>Most personal mail servers are not going to be sending mass emails.
There's no way to tell whether a random host on some residential ISP or VPS provider is doing personal email or spam.
>An algorithm that auto-spam-boxes mail from domains it's never seen before (or only receives a small trickle from) is ignorant.
Since blacklisting is so prevalent, spammers who aren't idiots will be sending most of their spam from different IPs. They go out harvesting unpatched VPSes and Windows botnets. The latter is patched by residential ISPs blocking port 25, the former is still a problem.
>Maybe the big mail providers need to update their strategy regarding automatic spam-boxing?
Why? Almost all legitimate email is coming from one of the big mail providers or a big-corporate Exchange server. What reason do they have to let more spam into their users' inboxes?
Despite my grievance elsewhere in this thread that mail servers do not come pre-configured with sensible defaults, I would guess that no modern mail servers are pre-configured as an open-relay when installed. The concern about port 25 relaying being abused by spammers strikes me as an anachronism.
In any event, I found that mainstream ISPs are willing to unblock port 25 if you tell them you are running a personal mail server and have it properly configured to not relay for unauthenticated users.
Aside from that, you'll want one static IP to make your life easy so you don't need to use dynamic DNS.
I think matters such as availability of your server and configuration are surmountable challenges.
But I strongly agree with you about security. Although I really like the idea of this device, the first thing I noticed in the product web page is the use of PHP Roundcube for webmail. I have a fear of running PHP anywhere on my network. The track record [1] is way better than, say, Wordpress, but seeing things like Exec Code, even back in 2008, gives me pause.
Nevertheless, in my experience, ISPs will unblock port 25 for customers on request. I've only worked with a few in my years, but I've not yet encountered any resistance.
Most ISPs that I'm familiar with won't unblock common server ports (25, 80, 443) unless you are on a business account (though that's not much more these days).
Though it isn't always just inbound 25 traffic... often it's outbound as well. Also, the larger issue is hacked/pwned machines and bots, not just open relays. Which reminds me, I need to update Tomato on my router.
My ISP only blocks common ports on dynamic ip ranges. If you order a static IP, you do whatever you want with it. They even set reverse dns record for me when I asked, because some blocklists/mail servers want it that way.
Issue with running your own mail server nowadays is spam. Gmail does a decent job filtering all the crap.
Of course! And I am certain that she and colleagues always used GPG or similar when e-mailing between her personal mail server and the State Department's official mail server. Because it goes without saying they would do so, otherwise those e-mails carrying untold sensitive information would have been routed in the clear. And that obviously never happened.
What I've gathered from news reports is that her mail server was accepting SSL connections, with a certificate that was probably strong enough at the time it was installed, but that was insufficiently strong for the brute-force attacks available today. Aside from that, it wasn't too bad - especially given the physical security of her residence (i.e., Secret Service).
Note that the State Department's score on the FISMA report was significantly worse than that of the infamously hacked Office of Personnel Management, so maybe Ms. Clinton was on to something. https://www.whitehouse.gov/sites/default/files/omb/assets/eg...
I remember reading some early RFC's that suggested pre-SMTP, remote "email" was "sent" using FTP.
Perhaps it was something like a PUT command to append to a user's "mail" file on the recipient computer?
This seems to suggest email was always envisioned as something to be "received" rather that voluntarily "retrieved".
In retrospect, knowing what happened to email, the later idea (e.g., djb's proposal) makes more sense.
The funny part is that even though the "receive" idea is still the core part of the email process, that is not what email users do in practice.
Users "retrieve" their mail from some third party who receives it for them.
First there was "POP" then there was "webmail".
Back in the Stone Age when email was "invented" computers were too expensive for the users to own.
Each user shared the computer with other users.
Each had an email account, usually administered by the organization that owned the expensive computer.
Today computers are inexpensive and seemingly all users own one but they still do not have control of their email accounts.
Some organization is still receiving all their mail before they do. And keeping a copy of course. :)
Then, in more recent history, the domain name "business" (a monopoly run by an organization that derives its "authority" from nowhere), fueled by the popularity of the "www", became the gatekeeper to email.
When was the last time you sent an email to user@[ipaddr]? Originally if my memory of the old RFC's is correct that was how it worked. No domain name needed.
Today, if there is no purchased, monopoly-blessed "domain name", then according to the people who control users' email there is no "valid" email address. No pay, no play. Even if port 25 is open.
Assuming a user who has paid for access to the "internet" wants control of their own email (seems reasonable), then "email" is still an addtional fee.
Back in the 90s, I used to run my own. It was extremely frustrating when big ISPs started blocking mail from dynamic addresses. I understand why they had to do it but it was still inconvenient for people like me.
port 25 blocking is just the beginning. In most cases mail sent from a consumer facing ISP (Verizon, AT&T, RCN, etc) will not be accepted if the destination server is implementing SPF mail authorization, which most are these days.
Domestic systems tend to be in configurations that make it hard to accept inbound TCP connections. You could serve SSL on a random port and open a port using UPNP, and it will work most of the time.
It's a difficult circle to square. The most trustworthy system is one you administer yourself and manually inspect all updates, but in practice the amount of work required makes that almost impossible. If you allow the OEM to do updates they can compromise you. If you don't do updates you end up vulnerable to exploits.
The "send a reference to the message not the message" technique was part of DJB's "internet mail 2000" proposal: https://en.wikipedia.org/wiki/Internet_Mail_2000