See the next post in this series: Greenhouse development update 1
When I started this blog in 2015, I was just playing around. I published a few small technical guides, cross-posted a couple articles I was paid to write for my employer's blog, and even the occasional opinion piece commenting on tech trends of the day.
However, I believe the wind changed in 2016 when I posted what is still probably one of the best articles on this site, titled Pragmatic Path Towards Non-technical Users Owning Their Own Data. In between the bombastic remarks, silly interactive elements, and gag formatting, I believe I struck at the crux of the internet's problems over the past ~15 years:
These days, people don't bother to self-host anything because it's a nightmare to manage, not because it's [materially] expensive.
The internet of 2020, amazon, google, facebook, twitter, snapchat, etc, the nightmarish digital surveillance panopticon we all know and love, was enabled by laziness just as much as it was engineered by greed & malice. Not just the laziness of users who'd rather click "I Accept" on the terms for a Software as A Service Account. It was also enabled by the laziness of developers who allowed themselves to be placated by lavish (yet, in an economic sense, insultingly-stingy) stacks of cash, and failed to consider whether or not what they were doing was right. They were only concerned with their good standing in the workplace, to protect the sanctity of their 30-year mortgage. They called it an "honest living", and left it at that. I may have been that person, minus the mortgage, but I'm not willing to be done there. I'm not done just yet.
Ever since I published the "Pragmatic Path" article, I believe SequentialRead has had a higher purpose. To try to make a difference in the world, no matter how long it takes.
server garden, the dream I pivoted away from
Those close to me will know I've been cooking up a project for world domination full-time ever since I quit my job. I've been reluctant to post about it, I think because I'm afraid of failure & I wanted to have things "just right" with a nice video demo and download-able alpha version people could try out. I made the mistake of feeling like I had "once chance", feeling like I'd better not mess it up!!
I'm trying to correct that now. It's a small step, but I can at least share an unpublished draft of an introduction to my Automated Server Garden project and a super scuffed 25-minute demo video to go along with it.
The extremely short version of the story goes: Automated Server Garden leverages/wraps commercial cloud services like Backblaze and DigitalOcean to try to make sophisticated self-hosting techniques easy enough that average computer users could dabble in it. It provides a layer of automation between the user and the cloud service provider to ensure that at the end of the day, the system works every single time & the user doesn't have to fight their way through an unfamiliar hell to make that happen. But that's not what this article is about.
As the product began to materialize and I could put my hands on it, feel what it felt like to use it, I started realizing that even though I had done all I could to smooth out the experience, asking users to sign up for multiple cloud service accounts was probably too much.
Almost all commercial cloud service providers are primarily designed around / marketed towards technical people. And NONE of them are offering the exact kind of service that's needed to make self-hosting easy enough, so I ended up having to require each user to sign up for a veritable smorgasbord of services, some of which cost at least $5/month minimum! Depending on your income, that might not sound like much, but it really bothered me because it adds up, and it's a lot more expensive than it needs to be. I wanted something that would be as close to free as possible.
Cloud Providers Roundup
PageKite gets close to offering what I need, but it's fairly antiquated and really really expensive. It costs 30 to 60 times more per unit (bandwidth) than over-the-counter prices one would get from something like DigitalOcean. Plus, it doesn't offer the dedicated IP addresses needed for hosting email servers. Finally, the way that PageKite's default & recommended option for new users employs a "just trust us" security model kinda rubbed me the wrong way.
ngrok may be easy to use, but it's exclusively aimed at testing and debugging, not operation of a service in "production". Also, it somehow manages to be even MORE overpriced than PageKite.
DuckDNS and FreeDNS are both great, but they lack in automation options and as free services, they're relatively "sketchy".
Gandi.net, Backblaze, and DigitalOcean, are great, with excellent automation options, quality of life features, and decent prices, but they're still just traditional general purpose cloud providers aimed and techy customers. They don't have any services which make self-hosting easier. In fact, they have an interest in getting customers to decide AGAINST self-hosting & lock themselves into a monthly payment!
Non-provider serivices, the distributed web projects like BitTorrent, libp2p/IPFS, Ethereum, Bitcoin, Namecoin, etc, are all great, but as of today they aren't supported by any mainstream web browsers, so anyone who is interested in publishing can't really rely on them. Like I said before,
If your app doesn't have a URL, who's going to use it?
So, you might be able to tell where this is going. I started thinking about making my own cloud provider. But before I talk about that, let me fill in some of the context. In the "Pragmatic Path" article, I talked about using Virtual Private Network software like OpenVPN to connect the home servers to a public-cloud based gateway so that users could operate globally-reachable servers more easily. This worked ok for a small-scale test I did with my own sequentialread.com at the time, but it's not perfect. A VPN makes the cloud server and home server intimate with each-other, and in my opinion, that's an undesirable relationship for security reasons. My solution also required a lot of manual fiddling to bring online, and if I wanted to connect more servers, it would only get worse.
When I was building my Automated Server Garden prototype piece by piece, I created a piece of software called threshold to replace the VPN. Threshold is a slightly esoteric piece of network protocol software called a reverse tunnel.
(If you're curious what that means or where the term comes from, I've created a handy network protocol software terminology guide you can refer to 😉).
Reverse tunnels are the right tool for the job if you want to make self-hosting radically easier:
- They only publish exactly what the user asks for to the internet, following the principle of least privilege when it comes to connectivity.
- In particular, I specifically set up threshold's tunneling protocol so the client is authoritative regarding what the threshold server can connect to, the server can't send malicious connect requests to any host or port.
- Reverse tunnels are dramatically simpler and more secure from the self-hoster's perspective.
- They do not require modifications to the network configuration on the self-hosters's computer, and they can run as any unprivileged user.
- It's very difficult to run a VPN without that VPN software having root access to the host machine at some point.
- Reverse tunnels still mask the server's real IP address and act as a lightning rod for denial of service attacks like a VPN would.
- In my opinion, it's easier and cheaper to make reverse tunnels redundant/scalable, so they are less likely to have outages.
However, right now the only commercial providers selling reverse tunnel services are extremely overpriced, have abusive security models, and/or have pigeonholed themselves into specific use cases.
Enter greenhouse
When I was working through the process of building and testing threshold, I began to realize that since the threshold tunnel server was designed from the ground up to be relatively trustless*, that implies it should be fairly trivial to turn threshold server into a multi-tenant service. That is, one application on one server can host multiple users & those users can benefit from it independently of each-other. Slowly the idea of selling threshold server as a service bubbled into my mind, and greenhouse was born. I had already created the webapp for cyberia's "artisanal" cloud compute provider capsul, which is seeing continuing success. So I had a boost of confidence, feeling like creating a cloud service provider was not nebulous anymore, it was an attainable goal.
As I started to work on greenhouse, I started realizing how unique it would really be. NO ONE ELSE IS DOING THIS. Well, that's not exactly true. PageKite is doing the same thing, but pagekite is so expensive, antiquated, and techy-user-focused that it barely counts. So, I feel like there is a lot of opportunity here. I was apprehensive about investing my life into the Automated Server Garden idea, because even though I trusted my convictions regarding why a new project is necessary, I wasn't sure I wanted to compete with the already-established self-hosting platform projects like YunoHost, Syncloud, and Nextcloud. However, I have no such reservations regarding greenhouse. I believe greenhouse could be a massive boon to YunoHost, Syncloud, and Nextcloud, extending their appeal to reach more users and amplifying their potential benefit rather than competing with them.
I think that from a user experience perspective, its abhorrent that YunoHost, Syncloud, and Nextcloud all throw the domain name / internet networking problems "over the wall" at the user and say
Oh, you wanna self host? Then you must be an expert, you go figure it out yourself 😈
User experience is hit hardest here, but system stability & security are also endangered by this behavior. Setting up a server on your network isn't trivial, especially if you are paranoid about security and want to make sure you are doing everything safely.
But I'd rather solve those problems as directly as possible, instead of going all (ノ°Д°）ノ︵ ┻━┻ and making my own end-to-end solution completely from scratch.
So for now, I think I'm going to keep working on building greenhouse first, and try to build greenhouse integrations into other existing projects before I go further with Automated Server Garden. I believe I can offer a product which easier to use, secure by default, and is straight up thirty times cheaper than my competition (I would charge $0.01/GB of bandwidth), with no credit card required to start out and no minimum payment. I believe this could be a big deal, so I'm going to go for it.
And after that, I might try to implement the WebRTC+ServiceWorker+WebTorrent-powered bandwidth-optimizing Reverse Proxy Server, which could turbo-charge greenhouse and enable bandwidth-intensive applications like video without breaking the bank. I've already picked out a name for it: 🍠 Tuber.
Stay tuned.
— Forest
See the next post in this series: Greenhouse development update 1
Comments