Sunday, 2 October 2016

systemd: OpenSSL of the future?

OpenSSL has received a lot of flak over the years. Both cryptographic and implementation flaws have been plentiful, and researchers are all over it. Much has been said about the software engineering process that was used to put it all together.

It appears to me that systemd is following a similar path, with another vulnerability turning up in the last few days. A very quick assessment of this vulnerability would place it as a 'medium' (CVSSv3 between 5.5 and 6.5) severity vulnerability. Ayer's post goes on to use this example as another data point in favour of the "systemd is insecure" camp.

OpenSSL is bad. It is security-critical software that has a constant stream of high and critical severity bugs. However, it's not too much of a problem for me. I update the library and restart the affected services of a test system, then on to the production systems. Usually this is just reloading or restarting nginx and sshd. The impact to end-users is small or non-existant. For some people, an OpenSSL bug is much worse, it all depends on the exact nature of the system you run and the particular bug.

On the other hand, we have systemd. We've not seen as many issues so far. But we can't just update and restart systemd, because it is a tightly integrated with many components. I feel like this is similar to the libc vulnerabilities we saw a little while back. The maintenance impact is much bigger because we're forced to do a full system restart.

This is one of the many reasons why privilege separation is such a good thing. Every new process gets the updated code. So while the minimal core of the application may contain vulnerable code, it might be easy to show that it is never reached. On the other hand, the privilege-separated workers don't need that much attention, so we just restart them and have them pick up the upgrades. Further, each worker process can be individually sandboxed and allowed to fail.

I agree with Ayer's perspective. Software is written by fallible humans who make mistakes with alarming regularity. When we dive into coding without thinking about the long-term implications, we set ourselves up for failure. If we build a monolith, any RCE bug is fatal. If we fail to sandbox our systems, any path traversal bug is fatal. We should avoid designs where common flaws are fatal.

Engineers need to be thinking about a few core questions:

  • How important is this system going to be?
  • How do we make bugs less likely?
  • How do we make bugs less problematic?
These are the central questions that a software development process answers. These are the questions which we frequently ignore. We purchase time-to-market with steep maintenance costs. If the systemd developers persist in ignoring these deeper issues, I think systemd will be the OpenSSL of the future. A constant stream of high and critical severity issues causing a never-ending headache for those that use it.

If you are starting a project, or running a project, please look at the design of vsftpd and OpenSSH. Look over the various secure software development life-cycles. Think about using a language that supports you. If you're going to push people to use your system, consider the security implications of your decisions and remember that hope is not a strategy.


  1. The scope of the two projects is dissimilar.

    Problems in systemd do not affect me since I do not use it.

    Problems in openssl however had DO affect me, largely due to other problems requiring it. For instance, to the best of my knowledge, rubygems depends on openssl, at the least for when you push new gems yourself.

    The more code you have, the higher the chance for bugs and side effects.

    > If the systemd developers persist in ignoring
    > these deeper issues, I think systemd will
    > be the OpenSSL of the future.

    The systemd developers have ignored critical statements ever since and will continue to do so.

    As for systemd itself, one stated goal is to "change Linux". There is not really any middle ground here.

    The much more interesting question is why Red Hat and co pushed this onto the users. You can reason that Red Hat does not care - fair point - but what about debian? That was very surprising to bear witness too. The old debian goes with devuan.

    1. / due to other problems / due to other programs /

      But often also problems!

  2. Why can't you update and restart systemd? systemctl daemon-reexec is a thing.

    In what way does that break other services?