Systemd is the stupidest piece of stupid brain-dead init software ever invented.

A tale, in a log.
systemd: BOOTING !
systemd: Oh noes! /var needs fscking! let’s drop into emergency shell
systemd: Emergency shell! You should check out the log. It’ll totally tell you something pointless that’ll send you on a wild goose chase for half an hour.
systemd: Oh, by the way, I mounted /var cos I want to log there, and I won’t unmount it for you in emergency shell, cos that’s just helpful.
systemd: What, wait, you’re not going to use SysV are you? No, don’t touch tha<URK>
SysV: Booting.
SysV: Oh noes! /var needs fscking! Let’s try fscking it for you …
SysV: /var failed. needs manual intervention. Here’s an emergency shell. Not with /var mounted because I’m not a moron of a boot loader.
SysV: What’s that, you’ve fixed it? Oh, you can reboot with “shutdown -r now”, it’s fine, I won’t force you to use my own internal reboot command, because we’re all adults here.

*sigh*. Can I get those 2 hours of my life back please?

Varnish weird error

Are you getting a weird error from varnish that you’re having trouble working out? Is it cryptically saying that there’s an issue with your host definition in your VCL?

Expected ID got ';'
 (program line 165), at
 ('input' Line 20 Pos 30)
 .first_byte_timeout = 120;

You need to add a time unit to your timeout definitions — ‘s’ for seconds, ‘m’ for minutes.  Stupid cryptic error is dumbly cryptic. Grrr.

ARGH! A Rant about software developers

When software developers update their software to include new configuration options, they have a bunch of options to pick from, of how to handle installs running on the old options.

For example, you could

  1.  automatically update config
  2.  document a process to gracefully update config
  3.  Give human readable errors showing what changes you need to make to conifg
  4.  Break the application with a super cryptic error message, and refuse to start until config is wiped with ‘new’ default config.

Why do developers keep picking 4? At least 2 horribly complex bits of software in the past year have decided to do that to me, which cost me a few days of head-bashing-against-desk trying to work out what was broken time. 🙁

Varnish nsca logging on systemd system with x-forwarded-for

So, you have a Varnish server running systemd, which is behind a reverse proxy for SSL like nginx, and you can’t work out how to make varnishncsa log IP addresses from a specified header? Well, it’s a bit of a pain in the neck really. You need to override the systemd service file, which is like systemd’s version of the init scripts. Due to it being systemd, this is not just a case of editing a file…

For Debian, you can use the service file below, and paste it into

Once done, you need to reload systemd’s service listing itself;
$ systemctl daemon-reload

Congratulations, you now have varnishncsa logs including the visitor’s real IP address, as specified by Nginx. Change the name in “{X-Forwarded-For}” to change the header name, for example if you want CloudFlare’s view of the client’s IP address, use “CF-Connecting-IP”

 ExecStart=/usr/bin/varnishncsa -a -w /var/log/varnish/varnishncsa.log -D -P /run/varnishncsa/ -F '%%{X-Forwarded-For}i %%l %%u %%t "%%r" %%s %%b "%%{Referer}i" "%%{User-agent}i"'
 ExecReload=/bin/kill -HUP $MAINPID


mysql/mariadb failing to start under systemd

If SystemD MariaDB/MySQL is failing to start, make sure your logs directory is set correctly. Sometimes this is caused by failing to correctly make sure the directory is moved.

You’ll see an error log in the journal as follows;

Nov 27 18:24:55 db-a mysqld[28677]: 2017-11-27 18:24:55 140273229838208 [Note] /usr/sbin/mysqld (mysqld 10.2.10-MariaDB-10.2.10+maria~jessie-log) starting as process 28677
Nov 27 18:24:55 db-a systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
Nov 27 18:24:55 db-a systemd[1]: Failed to start MariaDB database server.
Nov 27 18:24:55 db-a systemd[1]: Unit mariadb.service entered failed state.