About sixteen months ago, I started working for Stratum Security, and was brought on to head the development of a new project called XFIL (like exfiltration). I've written a fair bit already about some of the particularly interesting things we've been doing, but I wanted to write specifically about our choice to use Rust. The short story explaining what XFIL is goes like this. XFIL is a network security testing tool that tests defenses from the perspective of an attacker that has already managed to infiltrate into your network. XFIL's agent software attempts to do all kinds of sneaky things to exfiltrate synthetic "sensitive-looking" data, download "dangerous-looking" files, and plenty of other network and host-based tests. Accompanying this agent software is an entire suite of backend services, but those aren't the focus of this post.
Before I started working on XFIL, there had actually been something of a prototype written in Python. It did a handful of things that we knew the new version would have to be able to do. Considering the functionality of the Python version and some of the loose ideas floating around in peoples' heads for future functionality, we came up with a list of requirements for our next language.
- We have to be able to target Windows first and foremost.
- We need to be able to work with a lot of common network protocols.
- We need to be able to dig into some really low-level networking.
- We would like to do very systems programming-type stuff (that I can't get into here).
There are, as you might expect, a lot of other "requirements" that are implicit when choosing a technology that has to be used by a team of developers, and in particular when deploying software with the kind of behaviour our agent has, but I'll get to that in the following sections.
Before settling on Rust, we considered a handful of alternatives that seemed like natural choices for our kind of application. I'd like to caveat this entire section by acknowledging that, technically, any programming language is capable of doing anything at all, if you're willing to put enough work into it. Of course, the amount of work that has to be done to twist a given technology for a use that it's not designed for is often so expensive that it's better to just use a technology better suited to your requirements, and that's precisely why there are so many programming languages in existence.
Even though the prototype of the XFIL agent had been written in Python, we hadn't considered it a viable option for the new version. We want the agent to be running with a minimal footprint, to be able to take advantage of concurrency, and to be able to do "system-level stuff" with relative ease. Python just doesn't really fit the bill here as a high level, interpreted scripting language with no "real" concurrency story.
The next options we considered were C and C++. We could absolutely do systems programming with these languages, "real" concurrency is certainly feasible, and being as low-level as they are, both languages would allow us to meet some threshold of resource efficiency. However, some new secondary requirements became apparent as we considered these options. First and most important, we wanted the agent to be relatively safe. That is to say that we couldn't accept deploying a piece of software to our customers' machines that might (or more accurately, would almost certainly) contain many bugs caused by improper memory management which have been the source of many of the kinds of security vulnerabilities that have cost companies billions of dollars in aggregate. Seriously, take a look at lists of common software weaknesses such as Use After Free and feel your mind melt for a bit. Writing safe C and C++, even with many of C++'s newer features, is really something of a master craft. It's certainly not something I could imagine being capable of without at least the equivalent of a Ph.D's worth of study.
The last alternative that we considered in our early research was Go. We had already written quite a bit of software in Go, so it seemed like it would be pretty reasonable to use it again. Go has a great concurrency story and it's safe thanks to its runtime and garbage collector. However, despite its early marketing, Go isn't really that much of a systems programming language and has, in recent years, moved more into the network services space, where it's enjoyed a quite some success. We ended up deciding against using Go for the simple reason that it seemed like we might have to twist our arms a bit to get it to do some of the really interesting systems-level work we had in mind. In fairness, Go might have turned out to be a perfectly reasonable choice.
Of the languges that we considered using to build the XFIL agent in, Rust could be considered the youngest, with its first 1.0 release having been published exactly two years and one week ago at the time of this writing. However, Rust has a lot going for it, with an impressive roster of strengths that convinced us it was worth a bit of a gamble. In no particular order:
- Rust's ownership semantics guarantee that safe Rust code that compiles will not contain bugs like use-after-free, dangling pointers or other resources, data races, and various memory corruption issues.
- Rust, like C++, implements zero-cost abstractions, which means users of the language enjoy pretty high levels of productivity without incurring runtime performance costs.
- Cargo, Rust's package manager, is incredibly simple and effective, and make's working with the almost 10,000 (at the time of this writing) crates/libraries published to crates.io very easy.
- The Rust community is incredible, and it is possible to receive kind and careful guidance even from the compiler's developers as well as many of the authors of your favorite crates.
Rust is designed in such a way that any safe code (i.e. all code not marked "unsafe") that compiles is guaranteed by the compiler to be memory safe. This is tremendous for two reasons. First and most obviously, because it means Rust's compiler enforces the complicated safety rules that we would have had to understand and apply (if we were using C/C++) for us, and it's way better than we ever will be at it. Second, it means that we can bring developers who might not be expert systems programmers already and have them work on our code without the fear of doing something catastrophically wrong. We get all of this while still being able to write extremely low level, highly performant software.
In the interest of keeping this post as short as possible, I won't go into the details of the other points I made, but I will provide some links to reading material that should be pretty compelling if the above paragraph wasn't enough to convince you that Rust is the right choice.
It goes without saying that the XFIL team has encountered some difficult challenges as a result of our decision to use Rust. Because the language is fairly new, we can't always find a crate with the functionality we need. For example, we have not yet found an SMTP library that supports sending emails with attachments. Similarly, figuring out how we should handle doing some Windows systems programming in Rust hasn't been very easy with few, if any, public examples to follow.
Certainly, Rust is not the easiest language to pick up, but with loads of excellent resources out there, along with the help of the community and plenty of blog posts and tutorials, I firmly believe Rust to be much more approachable than C or C++ in particular, and it lets us do a lot more in the systems space than languages like Go make it easy to accomplish.
- fireflowers, The Rust Programming Language in the words of its practitioners by Brian Anderson
- Plus each of the posts linked to in the above, including the epilogue.
- Abstraction without overhead: traits in Rust by Aaron Turon
- Would Rust have prevented Heartbleed? Another look by Tony Arcieri
- Fearless Concurrency with Rust by Aaron Turon