Hi, We're Fly.io!

Fly.io builds computers for agents. It’s the thing we’re excited about right now and the future we’re building toward: agents that need a real computer, fast to start, isolated, disposable, reachable from anywhere, spun up thousands at a time and gone again. That unlocks a category of work that doesn’t happen on a shared sandbox, and demand for it is a big part of why our fleet is growing the way it is.

The good news, and the reason we can move quickly on it, is that we already built the machine that does this and it has thousands of customers. Our platform transmogrifies Docker containers into Firecracker micro-VMs that run on our own hardware around the world, and connects all of them to a global Anycast network that picks up requests from everywhere and routes them to the nearest VM. That’s what runs apps close to users, and teams run their production workloads on it today. Computers for agents isn’t a separate product bolted on the side, they are born out of the same platform. The networking and load balancer infrastructure needs to support everything, be it an app made by an agent, or a computer that an agent runs on, or the millions of apps that have been running on our platform since before the age of agents.

Agent workloads churn machines faster, so the routing state our proxy depends on changes more often. That “more” is where the interesting engineering is. Sessions run long and stateful alongside ordinary request/response traffic. Bursts arrive in shapes human traffic doesn’t produce. Every one of those pressures lands on the same proxy and the same routing database that a customer’s web app depends on, which means the work of making agent workloads fast and reliable is mostly the work of making the platform faster and more reliable for everybody.

Most of the platform is written in two systems languages. Go powers our orchestration (it’s what converts Docker images and provisions VMs). Rust drives the networking: fly-proxy, our Anycast proxy, and Corrosion, the distributed database that keeps the whole fleet pointed in the right direction. Those aren’t the only languages in play. Elixir runs a significant chunk of our control plane and customer-facing services, and you’ll run into other things around the edges. This role, though, lives squarely in the Rust half of the house.

We’re looking for someone to work on that networking team.

​

What this team owns

fly-proxy is an interesting piece of code. A request for some Fly.io app running in Dallas and Sydney arrives at our edge in, say, Toronto. We need to get it to the closest VM (Dallas, here). So the proxy keeps a live picture of where all the VMs are running and makes quick decisions about where to bounce each request.

We don’t simply forward those requests. fly-proxy runs on both our external-facing edge hosts and our internal workers, and builds on-the-fly, multiplexed HTTP2 transports (over our internal WireGuard mesh) to move requests around. It does raw TCP, not just HTTP and WebSockets. It balances across the bunch of instances most apps run, automatically configures HTTPS, and gets certificates issued over the LetsEncrypt ALPN challenge. A single proxy is juggling connectivity for lots and lots of apps, isolating each one’s concurrency budget so a busy app can’t starve its neighbors, and tracking per-app metrics the whole time. It’s built on Tokio, Hyper, and Tower.

Then there’s Corrosion. Corrosion is our distributed database, built mostly to move routing state around the fleet, that constantly-updating global picture of where every VM is, kept eventually-consistent across nodes all over the planet. It lives under the same umbrella as the proxy, and it’s a genuinely fun distributed-systems problem in its own right. It’s also where fleet growth shows up first: the more machines there are and the faster they turn over, the harder the problem gets.

And it goes lower than that, too: there’s BGP at one end of the stack and we own the BGP infrastructure that tells the internet how to route to the nearest “edge”. You’ll move between wildly different layers of abstraction depending on what the day demands.

There’s no shortage of distsys to chew on here. State machines and invariants that span multiple nodes across the globe, load stats that change every second and drive how we balance, requests that fail and get retried somewhere else (that last one is the core of how we run distributed Postgres). It’s gnarly and technical and always-on and constantly being updated, and it also has to stay up. If that sounds fun, great. This might be the gig for you.

​

What we’re looking for

Familiarity with Rust. Obviously.

You’re not scared off by complex codebases or distributed systems. The kind where the state machines and invariants stretch across nodes on different continents.

You can work at wildly different layers of abstraction. This team is responsible for (or shares responsibility for) systems spanning a lot of layers, from BGP networking up to L7 load balancing, with Corrosion (a distributed database mostly meant for routing) under the same umbrella. A successful candidate needs to be able to wear all of those hats.

You think clearly about tradeoffs. We’ll be honest with you: the proxy carries some technical debt that we’re steadily grinding down, and we’re doing it while supporting a user base that’s growing almost explosively, especially this year. A failure here can’t be allowed to take out the entire platform, and we want both our team and the teams we work with to be able to reason about how proxy and Corrosion fail.

You’re operationally minded. “How are we going to operate this reliably?” matters as much as the codebase itself, if not more. The first job of this team is keeping the networking side of Fly.io running smoothly.

​

How we hire

This is a mid-level, remote, full-time position.

The salary for this role is $190k USD, and we offer competitive equity grants with a long exercise window. We provide health care benefits, flexible vacation time (with a minimum), hardware/phone allowances, the standard stuff.

Our hiring process may be a little different from what you’re used to. We respect career experience but we aren’t hypnotized by it, and we’re thrilled at the prospect of discovering new talent. So instead of resumes and interviews, we’re going to show you the kind of work we’re doing and then see if you enjoy actually doing it, with “work-sample challenges”. Unlike a lot of places that assign “take-home problems,” our challenges are the backbone of our whole process; they’re not pre-screeners for an interview gauntlet. (We’re happy to talk, though!)

If you’re interested, mail jobs+networking-engineer@fly.io. You can tell us a bit about yourself, if you like. Please include your location (country / state), a bit about an excellent meal you recently ate, and your Github username so we can give you access to a work sample. We probably won’t respond to emails that don’t include all three items.