Logging boot messages to the console with SystemD and Digital Ocean

The problem

I recently ran into a situation where my boot times had gone through the roof. It was taking 6 seconds to boot through the kernel and nearly 3 minutes to boot through the userspace modules. Unfortunately, upon inspecting the log, it was unclear as to what exactly what was causing the delays, as the seconds were not being recorded, and since it was the usersapce modules taking a long time, the boot console wasn’t letting me see visually what was taking so long.

The Solution

Since Digital Ocean offers the “Recovery Console”, I decided to try to get SystemD to print its logs to the boot console. After a quick google, I found a reasonable suggestion to add a line or two to an unnamed location. While poking around in /etc/ looking for the proper place to insert this configuration, I found an easier way. Within /etc/systemd/ there is a file named journald.conf. It contains a series of default values for the service. Specifically, I found two that looked interesting:

#ForwardToConsole=no
#TTYPath=/dev/console

Just for fun, I decided to temporarily abandon my search for the aforementioned mysterious configuration location and try giving those two a shot. I changed them to the below values:

ForwardToConsole=yes
TTYPath=/dev/TTY1

Thankfully the TTY number is printed by Debian on the login screen, so I didn’t have to go figure that out. After a quick reboot, the log messages began to flow!

The culprit? Cloud-Config was generating ssh keys for some reason. I’ve opened a ticket with Digital Ocean support to see if this is intended behavior.