emonPi Random seed fix

Hi,

I noticed that systemd was failing to save the random seed across reboots. This is important for ssh and https connections, and a predictable random number generator could lead to compromised security, especially with long-lived keys like https keys.

To solve this we can put the random seed on the data partition:

pi@emonpi(rw):systemd$ cd /var/lib/systemd/
pi@emonpi(rw):systemd$ sudo mv random-seed ~/data/
pi@emonpi(rw):systemd$ sudo ln -s ~/data/random-seed 
pi@emonpi(rw):systemd$ rpi-ro
Filesystem is locked - Read Only access
type ' rpi-rw ' to unlock
pi@emonpi(ro):systemd$ sudo systemctl restart systemd-random-seed.service 
pi@emonpi(ro):systemd$ sudo systemctl status systemd-random-seed.service
â—Ź systemd-random-seed.service - Load/Save Random Seed
   Loaded: loaded (/lib/systemd/system/systemd-random-seed.service; static)
   Active: active (exited) since Mon 2017-04-03 14:22:57 BST; 4s ago
     Docs: man:systemd-random-seed.service(8)
           man:random(4)
  Process: 12851 ExecStart=/lib/systemd/systemd-random-seed load (code=exited, status=0/SUCCESS)
 Main PID: 12851 (code=exited, status=0/SUCCESS)

Apr 03 14:22:57 emonpi systemd[1]: Started Load/Save Random Seed.
pi@emonpi(ro):systemd$ 

It’s only written at startup and shutdown, so the extra wear on the SD card is minimal (with buffering it might even be zero).

This also has the significant side effect that systemd sees all services as “running” and therefore says the status of the system is “running” rather than “degraded” which makes the pedantic part of me much happier :slight_smile:

Incidentally, I looked into making a pull request for this, but I couldn’t see how the filesystem on the SD card is built from the git repository, so I had no way of testing it. Could someone point me at the documentation?

Thanks!
Bruce

pi@emonpi(ro):systemd$ ls -l /boot/emonSD*
-rwxr-xr-x 1 root root 1496 Apr 29  2016 /boot/emonSD-07Nov16
pi@emonpi(ro):systemd$ cat /etc/os-release 
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@emonpi(ro):systemd$
3 Likes

How about this document?

 
Last week I was looking at the node-red service and saw the same error. But it didn’t mean anything to me since I didn’t know how random was used.

pi@emonpi(ro):~$ systemctl

Hi Jon,

Yes, that’s the one. Thanks! Lots of manual steps in there, surely we can do better…

For a temporary fix you can

rpi-rw; sudo systemctl restart systemd-random-seed; rpi-ro

Bruce

1 Like

I believe the steps are more a checklist for Glyn as he builds each new SD card release (just a guess on my part). If you don’t find what you need, try a post directly to Glyn @glyn.hudson

I agree, but apart from my tendency to try to automate these things, in this case it being a list of manual steps for a human makes it hard for me to make a pull request to fix the bug I mentioned at the start… If I find the time I will send a pull request to Glyn automating at least some of the steps.

Bruce

the image is using the hardware random number generator built into the raspberries isn’t it?

HI @bwduncan

Thanks for letting us know about this.

Would running rpi-rw; sudo systemctl restart systemd-random-seed; rpi-ro each time the unit is booted fix the issue?

Yes the image is currently manually built. I would really love to improve on this. I did have a script to build most of the image, however since Debain changed quite a lot between Wheezy and Jessie I decided it was best to built it manually once then iterate.

The document that @Jon linked to above are the exact steps followed to build the image.

@david, yes, but it’s not available at early boot so things which generate keys might do so in a predictable way. In any case, we shouldn’t rely on it working, it’s better to have defense in depth and if we can make it work, why not?

@glyn.hudson That might run the risk of leaving the root in read-write mode, if something concurrently decides to open a file before we can rpi-ro again. It’s safer to move the seed file.

There are tools which automate building linux machines, such as preseed DebianInstaller/Preseed - Debian Wiki but I’m not sure if it’s worth the effort.

Bruce

1 Like

Hey I have had to fix this problem on another very similar system … the solution that worked for me is described here:

https://www.raspberrypi.org/forums/viewtopic.php?t=129037&p=888220

In general terms symlink the /var/lib/systemd/random-seed file to /tmp and then edit the systemd service file (/lib/systemd/system/systemd-random-seed.service), adding the following to the bottom:

RemainAfterExit=yes
ExecStartPre=/bin/echo "" >/tmp/random-seed

EDIT: This does assume you have a ramdisk mounted on /tmp but I think that is the case with emonpi?

Hi Ben,

You’re right that this fixes the problem, but it also degrades the security of the system by losing the random seed every time the machine is booted. On the emonpi we have a data partition which we can use to store the random seed, which is the right place to put it. If you do that then you only need the symlink and don’t need to edit the service file.

Bruce

1 Like

Bruce, you are right. That is a better idea. Thanks.

Forgive me if this is a daft question as I do not know the full workings of the random-seed, but wouldn’t creating a new random-seed increase security rather than reusing the same “random” seed (seems almost a contradiction in terms)

Understood, but if an SD image was fully read-only (ie no write partition) wouldn’t Ben’s solution still work?

I only bring this up as I use fully read-only, forwarding only images with no local emoncms just (original) emonhub, this is very much like the older “Rock Solid” image we had prior to the emonSD. Very robust and very reliable. But currently has no change to the random-seed.

Ben’s RAM version would fit with that image alternatively I could consider momentarily making the FS read&write as is done with the “NTP_backup” solution. But that would involve far more than just a symlink so I am looking to weigh up the gain of persisting the random-seed over creating in it RAM over leaving it as it is.

Hi Paul,

No such thing as a daft question.

The random seed is maintained the kernel, and is constantly updated from various sources to provide sufficient randomness. Obviously in a closed system this is a hard problem. Imagine two identical SD card images booted at exactly the same time: No randomness there! The random seed script simply saves the in-memory random seed at shutdown and loads it again at boot. There is no portable way to “create” a new random seed [1], especially not when the system has just been booted and has no interaction with the outside world.

What I’m trying to achieve here is a robust, simple system which is also resilient to being stuffed on the dirty internet. There are probably bigger issues (the default login password, for example) to be fixed, but the solution I posted solves a problem for me in a neat way.

Pragmatically, I’d say if your filesystem is truly read only and inaccessible from the outside, then you’re probably fine to just reuse the same random-seed file forever. Given that it will be updated as soon as the clock changes anyway it’s probably fine. Booting with no random seed at all would be less good.

Bruce

[1] OK, so the chipset on the RPi has a hardware random number generator, but using this to seed the kernel RNG is so far outside the scope of this thread!

Is that in the kernal, by the kernal, for the kernal, etc?

Jut trying to get a handle on how the RS plays its part in Linux security.

Ok thanks for the clarification. So in short we should at least try to avoid duplicating the random-seed across images, and not opt for Ben’s RAM option as that would mean booting with no random-seed at all.

Another solution (for me) might be to symlink the random-seed to the /boot partition which does remain RW, that might sound a little low security as it can be read on a windows PC, but that is only if you already have access to the micro usb card, otherwise the location is no less secure than the /data partition on the emonSD.

A minimal quick fix would be to just ensure the RO random-seed is different on each device by forcing an update to be saved.

Or if there is a command so save the random-seed we could just add it to the NTP_backup “at shutdown” routine so it is different at each boot.

Just thinking out loud here I suppose the RO random-seed (providing it’s unique to the device and not copied accross images) is no different to a un-graceful reboot, as the same random-seed would get used on consecutive reboots in that instance anyway unless there is a periodic save (like with NTP)

I believe I am probably changing the random-seed inadvertently with my image duplication process anyway as I always insert the duplicated card, mount as RW, change the hostname etc and reboot gracefully whilst still in RW mode so the random-seed is most likely persisted at that shutdown and never changes again.

The use of symlinks for this is a little unnerving though as that is why the “NTP_backup” solution came about, the emonSD used to store the current time at each hourly NTP update to the /data partition, but this was never found at boot time by crucial services, the symlinks seemed “not to work” early in the boot process for one reason or another.

So I guess my options are

  1. Leave things alone and have a unique but RO random-seed
  • Symlink the random-seed to the /boot partition (easy)
  • Add “saving” the random-seed at shutdown to ntp_backup (messy?)
  • Or create some elaborate “mount as RW and save seed” at shutdown (over engineered?)

Have I missed anything?

Haha sorry, In the kernel, by the kernel, for the kernel (and userspace). For those interested:

The kernel stores a pool of 4096 bits of random data, which is used by the kernel and by userspace programs. Each access to the random pool causes more randomness to be generated using a cryptographically-secure pseudo-random number generator (CSPRNG). If it helps, think of the CSPRNG as a hash function (SHA1) mixing the pool to provide a lot of random numbers. The kernel uses these random number for things like TCP sequence numbers and negotiating secret keys for network security (WEP, WPA, etc). Programs in userspace (i.e. not the kernel) can ask for random numbers and use them for cryptographic secret keys (think SSH, HTTPS, authentication tokens) and other non-security things (think UUIDs or waiting for network or hardware).

The random pool is empty at boot, and the kernel relies on userspace to fill it with random data. The random data which was in use when the system was last shut down is as good as any.

The random pool is also topped up by things like hardware RNGs (the raspberry pi has one but I don’t know if it’s used in this way), interactions with the outside world (typing on a keyboard, moving the mouse are excellent sources as humans are deliciously random, but probably unavailable on the emonpi) and network activity.

The kernel keeps track of how random it thinks the pool is (how much entropy it represents) and will refuse to give out random numbers to programs which need really random numbers, like really-long lived secret keys. In that case the program is made to wait until there is sufficient entropy available.

As I’ve said, I think the risk to an individual emonpi from predictable “random” numbers is probably much lower than software bugs, failing to keep up with security updates or public default passwords. I hope this doesn’t get into a bike-shedding discussion!

Bruce

Hi Paul,

Yes, the images handed out on the physical cards are probably different, but the ones downloaded from the website are not.

Symlinking into /boot will work, but is it any better than symlinking into ~/data? If you need access to the data on a partition, configure your systemd service to run after the filesystem has been mounted with After=local-fs.target. What exactly was the problem which lead to the creation of NTP_backup? I would argue for symlinking /etc/fake-hwclock.data into a writeable partition, for the same reasons as for random-seed.

Don’t worry about someone reading the card on a windows PC. If they have your card they already have your root password and can read or change anything they want on the card.

The random-seed command mentioned right at the start (in systemd-random-seed.service) loads and saves the random seed on boot and shutdown, so there is always a fresh seed (if the path is writeable!), even if there is an ungraceful restart.

That’s an accurate summary of the options. I would argue for symlinking into ~/data. Option 2a :slight_smile:

Bruce

Perhaps, if you have a data partition. I’m not in favor of adding a data partition just for this on my images.

ntp_backup was originally written for a fully read-only image with only 2 partitions, prior to emoncms being included, the “Read-only image time issues” discussion gives all the details.

That’s how the first emonSD cards were (ntp_update was dropped initially) and there were an enormous number of issues caused by the incorrect time being loaded at bootup (1970/01/01 00:00:00) due to /data/fake-hwclock.data not being read, reverting back to the previously fault-free ntp_backup resolved those issues instantly and has been reliable ever since.

This issue was prior to Jessie and systemd,

I understand and agree, this is no big threat and I only discuss it further out of curiosity and if things do happen to get improved along the way great. but no biggy I’m happy with the status quo or maybe a symlink.

1 Like

Hi @bwduncan,

Thanks a lot for this. I have included the changes you describe in the latest emonSD-26Oct17 release:

SD-card-build.md has also been updated.

1 Like