Mosquitto won't start on boot after raspbian and emonsd update

emonSD
,
15

Apparently so, I’ve often read tar is faster than cp and they are both faster than rsync for cold copying, rsync has the advantage when just copying the changes.

Absolutely, I wasn’t criticizing, in fact if we were working with a fixed quantity then a tar file would be better, but, yes for easy maintenance, a template would be easier.

Indeed, but in fact it would be no different to them getting “accidentally removed” on a normal read write OS.

Exactly, it’s the choosing the lesser evil, risk loosing files that get accidentally deleted by the user verses accumulating redundant files by just adding.

Correct. The idea of saving the logfiles from ram to disk periodically and at shutdown would limit the wear by keeping all the continuous streams of log entries in ram and writing to disk in batches, much like the buffered-write mode of emoncms.

Another suggestion back then was to use some logrotate functionality to rotate the logs out of ram onto disk so only the latest logs remain in RAM, this would improve ram management too.

Regards an alternative logfile location there are multiple options, it could either be sited on the writable partition, or the service that performs the log backup/cloning could make the OS writable for a moment, that’s how the fake-hwclock works. The advantage of this is that the files will be available earlier in the startup process. Originally the fake-hwclock was backed up to the /data partition but we found that often the Pi would boot with a date of 1/1/1970 as the /data partition wasn’t always mounted in time for when the file was needed during start-up. This isn’t something we would want happening with the saved logfiles (or template?). /var/log_backup might be an ideal place. in the past I have hard-linked /var/log to /var/log2 before making the OS RO and adding the tmpfs /var/log, then when the tmpfs /var/log hides the disk copy of /var/log, the original files are still available at /var/log2.

[edit] The other consideration with the logfile location is that not all read-only images have a writable partition, so the “momentarily mount rw” option would be more universal and perhaps even protect the logfiles from accidental deletion too if RO most of the time.

16

Nice work @jsroques. I like the .tar solution and service to extract into /var/log before the other services start on bootup. Using a tar will allow us to keep the empty files and folder on github. I’m assuming the tar honours the folder and file permissions?

I would prefer to keep the tar manual creation rather than allowing services to be able to automatically add their logs. This allows us to keep control over what’s being logged.

~/emonpi is the best place to store the file so it can be included in the emonPi repo:

Would you like me to add these changes into a dev branch of the emonPi repo so we can further test and refin?

Thanks for your help.

17

It only lets you keep control of what’s permitted to log correctly and not cause problems. For a user to add a package that needs or wants logging, any attempts to edit the log.tar will block future emonpi updates or get over written.

Manually adding the files or folders to a template folder will retain the ownership and permissions and allow users to add a log file if needed for other services. all you need to put in the emonpi updater is short list of commands (like were originally in rc.local) so that if the log_template folder doesn’t exist, it gets created with all the usual files and permisions etc. That way if someone messes it up, they can just delete the folder and update the emonpi to fix.

18

Hello,

I am not sure I understood what you meant (and I am not a git guru…) So you create a new branch so I can push the scripts and config file ? So I would put the “template” files structure in some subdir in emonpi. But where would you put the service file ?

Or you add the files yourself ? I can send you the config and tar files as is ?

Whatever seems easier for you.

Yes, the -p in the tar command (as in “preserve”) takes care of this.

Anyway, the 2 next things I can have a look at would be to make sure the folders get created before requiring services start up and launch a “backup” every so often. I won’t have time in the next few days but can work on this starting end of next week.

19

Hello,

I have been thinking and do some research about this and here are some results… and questions…

How do you update files outside emonpi directories ? Typically, systemd file ?

About execution order, it seems that using before= wilth a list of of log files depending services (that is apache, redis-server, mosquitto and supervisor) will make sure that the service creating the logs will run before them. I found a nice article here : systemd: Unit dependencies and order - Fedora Magazine

Also, it may be usefull to separate the “default” log package, the “user” log package and the log backups.

For “log package”

  1. Defauls log package : have a ~/emonpi/logtools/varlog.template directory, making it simpler to update and keeping it in ro fs limiting the risk of user messing it up

  2. User’s log package : only backup directory structure and create empty files with proper perms in ~/data/varlog.template as the fs is RW and volume is low. This way it is easy to recreate the user’s log structure if needed but won’t affect oem default log structure : if the user deletes its own ~/data/varlog.template it should not affect oem.

  3. Use an oem-varlog service which

  • copies both ~/emonpi/logtools/varlog.template and ~/data/varlog.template (using tar cpf - /path/to/template | tar xpSf - -C /var/log for instance)
  • makes a backup of the structure in ~/data/varlog.template
  1. Have a script in cron which backups the structure, let say every 10 minutes

For log backup

It is a bit trickier…

I don’t understand what you mean. And I though hard links worked for individual file only.

Anyway, I had a look at logrotate but backup cannot cross file system boundaries. Thus, it cannot be used to backup logs from tmpfs to another permanent fs.

Putting them in ~/data is a bit dangerous as it could fillup the fs

So, a backup could be done in ~/log.backup for instance which could be it’s own fs or, if not, remount / as rw, do the backup and remount as ro. May be rsync could be an efficient way (but kind of overkill?)

What do you think ? I’ll be testing some of that but I’ve had issues with my emonpi locking up my ssh session although emoncms seems to work…

1 Like
20

usually a script inside the emonpi folder will create/edit/replace any files outside the github maintained folders.

indeed that would work, but it is less than ideal having to list all the services inside that logfile creating service, it undermines having a template folder, you may as well just keep all the stuff that’s currently in rc.local but move it too the new service. The idea of the template was that the service would not need editing and woulds remain standard regardless of how many logfiles it was creating.

This is why I suggested doing it as some kind of runlevel based ordering “before anything else” type statement although that is very broad.

Yes, you’re right and jogged my memory on why I abandoned that route and just copied the /var/log folder to /var/somethingelse as part of the RO implementations. When I have since looked at some old scripts I could only see cp and no ln /var/log and for the life of me couldn’t recall why.

Really? Ok, I didn’t know that. I assumed it would be able to put them in /var/somethingelse if the FS was momentarily RW.

i’d be interested in how your tests go, I would be happy to work on this too as it is something I feel is important, the emonhub.log files keep getting rotated out ever hour by logrotate and any reboot results in all logs lost. improvements here would make debugging much easier as well as making the RO OS more robust and installation easier.

“log files” was the first thing to cross my mind when I saw this EmonCms inputs page not charging When OpenRemote is starting at startup post.

1 Like
21

I am interested also! Once per hour I rsync the log files to a USB drive so they don’t get lost.

22

Hello,

I just finished a first step after struggling a bit with systemd…
Everything is here : GitHub - jsiobj/emonpi at var-log-files

Restoring /var/log file structure at boot

I created a small script and associated service which restores the /var/log/ file structure at boot.

The service is configured to run quite early in the boot process, before the end of basic.target which is before most “apps” services (using WantedBy=basic.target) but after all fs are mounted (using After=local-fs.target)

I found a nice cp trick : --attributes-only which only copies directories and files with proper attributes but with no data, that is only create empty files. That is a quick and efficient way of backing up and restoring the structure.

Here is boot process logs summary

Dec  8 22:01:29 emonpi rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="522" x-info="http://www.rsyslog.com"] start
Dec  8 22:01:29 emonpi systemd-modules-load[112]: Inserted module 'i2c_bcm2708'
Dec  8 22:01:29 emonpi systemd-modules-load[112]: Inserted module 'i2c_dev'
Dec  8 22:01:29 emonpi systemd[1]: Mounted Configuration File System.
...
Dec  8 22:01:29 emonpi systemd[1]: Reached target Local File Systems.
...
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/apache2’ -> ‘/var/log/apache2’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/apache2/error.log’ -> ‘/var/log/apache2/error.log’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/emoncms.log’ -> ‘/var/log/emoncms.log’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/logrotate’ -> ‘/var/log/logrotate’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mosquitto’ -> ‘/var/log/mosquitto’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mqtt_input.log’ -> ‘/var/log/mqtt_input.log’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mysql’ -> ‘/var/log/mysql’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mysql/error.log’ -> ‘/var/log/mysql/error.log’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mysql.log’ -> ‘/var/log/mysql.log’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/ntp_update.log’ -> ‘/var/log/ntp_update.log’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/oem-varlog.flag’ -> ‘/var/log/oem-varlog.flag’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/openhab’ -> ‘/var/log/openhab’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/redis’ -> ‘/var/log/redis’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/redis/redis-server.log’ -> ‘/var/log/redis/redis-server.log’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/service-runner.log’ -> ‘/var/log/service-runner.log’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/supervisor’ -> ‘/var/log/supervisor’
Dec  8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/supervisor/supervisord.log’ -> ‘/var/log/supervisor/supervisord.log’
Dec  8 22:01:29 emonpi systemd[1]: Started Creates required directories and file in /var/log.
...
Dec  8 22:01:29 emonpi systemd[1]: Reached target Basic System.
Dec  8 22:01:29 emonpi systemd[1]: Starting Mosquitto MQTT Broker...
Dec  8 22:01:29 emonpi systemd[1]: Started Mosquitto MQTT Broker.
Dec  8 22:01:29 emonpi systemd[1]: Starting Node-RED graphical event wiring tool....
Dec  8 22:01:29 emonpi systemd[1]: Started Node-RED graphical event wiring tool..

So it seems to work as expected : log files are ready before apps services startup.

I have created a ~/emonpi/logtools directory where there is

  • log.template which contains the “default” set of log files
  • oem-varlog.install : a (very) simple install script for the service
  • oem-varlog.service : the systemd unit file
  • varlog-init.sh : the service script which is doing all the work (I finally created a script because of the limitations of systemd interpreter)

On startup, 2 directories (there structures and empty files) are copied :

  • ~/emonpi/logtools/log.template which is the default set of logs
  • ~/data/log.template which is (will be) the backed up file structure from actual /var/log
    That way, default log files are in the RO part so the user can’t mess it up and actual log files are stored in the RW fs. Using --attributes-only combine with -n (no overwrite) makes it a very light process.

But… as of now, backup of structure does not occurs on system shutdown or reboot :frowning: and I have hard time to track down the issue… as I have no persistent log !

Anyway… I’ll keep trying. If ou have any clue of the reason why it won’t work, please let me know.

Full log files backup

Digging a little bit more into logrotate, although it cannot use another fs to rotate logs, it has a postrotate feature which could be used to run some rsync for instance. It is to be noted that rotated files should not / can not be moved from their origin directory as it will probably prevent logrotate to work properly.

But thinking about it anyway, it is probably not possible to backup all log files on shutdown, especially syslog, as you may not be able to back it up late enough in the shutdown process. But still, it shoud work well for apps logs.

That’s all for today. Any comment is welcome.

JS

1 Like
23

Hello,

Here is the status of my tests

I finally found out what was going wrong with the service

In the unit file, to “restore” the /var/log file structure early enough (and especially before user services) I set defaultDependencies=no. But I did not realize that this would also remove the dependency that makes systemd to run ExecStop on shutdown. So I added Conflicts=shutdown.target which is solving this.

Also, I moved the unit file to /etc/systemd/system instead of /lib/systemd/system. As I understand it /lib is for “packaged” apps, not user defined services.

I’ll will now move on to test a full backup of logfile using postrotate feature of logrotate and rsync.

The updated files are there : GitHub - jsiobj/emonpi at var-log-files

And, it is not yet too late so best wishes to OpenEnergyMonitor community :smile:

JS

1 Like
24

Hello,

I just did some new tests and now log files are copied over ~/data on shutdown.

I tried to run this backup using postrotate option in logrotate but I could not have it to work. There are serveral logrotate.conf file and I am not sure which one to use. Actually, I could not find how logrotate was run… Is it from emoncms ?

JS

25

Logrotate is triggered from a cron job, usually a daily task, but I think it’s a hourly task on the emonSD. Try

ls -la  /etc/cron.*/logrotate

to find the cron entry. This is from a stock Raspbian Stretch

pi@raspberrypi:~ $ ls -la /etc/cron.*/logrotate
-rwxr-xr-x 1 root root 89 May  5  2015 /etc/cron.daily/logrotate

I think the main conf file is /etc/logrotate.conf while the best practice for other applications is to add a file to /etc/logrotate.d/ and logrotate compiles it’s configuration by parsing all the conf files.

pi@raspberrypi:~ $ ls -la /etc/logrotate.conf
-rw-r--r-- 1 root root 599 May  5  2015 /etc/logrotate.conf
pi@raspberrypi:~ $ ls -la /etc/logrotate.d
total 24
drwxr-xr-x   2 root root 4096 Nov 29 01:17 .
drwxr-xr-x 112 root root 4096 Feb 20 11:34 ..
-rw-r--r--   1 root root  173 Sep 13 16:47 apt
-rw-r--r--   1 root root   79 Apr 18  2017 aptitude
-rw-r--r--   1 root root  232 Jun 10  2015 dpkg
-rw-r--r--   1 root root  515 Jan 18  2017 rsyslog

you can get more info on logrotate via the command-line with

logrotate -?

or

man logrotate

Sorry for going a bit quiet on this, I have been following your posts with the intention of “giving it a spin” but I never quite get to do it. But from what I read it looks impressive what you’ve done so far.

26

Hello @pb66,

Ok… I did not know there was such directories ! I just used crontab -l to check…

So indeed, there is an hourly task :

pi@emonpi(ro):~$ ls -l /etc/cron.hourly/
total 8
-rwxr-xr-x 1 root root 207 Jan 22  2016 fake-hwclock
lrwxrwxrwx 1 root root  41 Jan 26  2016 logrotate -> /var/www/emoncms/scripts/logger/logrotate
-rwxr-xr-x 1 root root 300 Jan 22  2016 ntp-backup

Looking at the script, I can see which conf file is used

pi@emonpi(ro):~$ cat /var/www/emoncms/scripts/logger/logrotate
#!/bin/sh

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate -v -s /var/log/logrotate/logrotate.status /etc/logrotate.conf 2>&1 | tee /var/log/logrotate/logrotate.log

pi@emonpi(ro):~$ ls -l /etc/logrotate.conf
lrwxrwxrwx 1 root root 46 Apr 28  2016 /etc/logrotate.conf -> /var/www/emoncms/scripts/logger/logrotate.conf

It is using a specifc conf file. I find it strange though for the logrotate.conf to be located in /var/www/emoncms as it is more OS related stuff and not really website related.

Anyway, I’ll try to make this work and give the results.

Thank for the reply. And no worries for the delay… I had been busy myself and just restart to work on this a few days ago.

JS

27

Me too!

I had a quick look and found some related files in the emoncms repo, there’s a readme too.(emoncms/scripts/logger at master · emoncms/emoncms · GitHub) but in the emonPi repo, which is essentially the emonSD repo, there are also some logrotate files but there is a note in one saying

# NOT USED - SEE https://github.com/emoncms/emoncms/blob/master/scripts/logger/

why that is I cannot say. But there doesn’t seem to be any mention of logrotate in the build guide or update script so I’m not at all sure how and when it’s implemented. Or why the files are even in the emoncms repo, the “rc.local” file that is in that same directory is not used, the latest rc.local is in the emonPi repo. So I can only guess the rc.local is added by the emoncms logger install script and then overwritten by the emonpi update script.

Absolutely. I agree. I would like to eventually separate out all the “read-only” stuff to a separate install script/guide so that users can implement read-only or not as they choose and everything for “read-only” is in one place.

[edit]
Found a ref to the logger install script in the “low-write” install guide

28

Hello,

I finally did not use postrotate feature of logrotate as is only runs if it effectively rotate at least one file. So you’re not sure there will be a backup every hour. So I just added an hourly cron which runs every hour.

As of now, here is what is done :

  • At startup : Content of ~/emonpi/logtools/log.template and ~/data/logtools/log.template are copied over to /var/log/
  • On shutdown : Structure (directories and files with no content) is copied to ~/data/logtools/log.template so it can be restored on next boot
  • Every hour (crontab.hourly) : /var/log is archived to ~/data/logtools/log.archive

I put all scripts, service unit, templates… in a dedicated directory ~/emonpi/logtools

Everything is there as usual : emonpi/logtools at var-log-files · jsiobj/emonpi · GitHub

There are still a few things to do :

  • Thinking about it, on startup, not only the structure but the whole content should be restored. Or I need to “rotate” the log archive to that it is not erased with a new archive of empty files
  • I wrote a litlle install script but I don’t know where this should be run from. emonpiupdate script ?
  • Thinking (again) about all this… shouldn’t a persistent fs be used for log files ? :thinking: This would make it simple. On my dev pi, I added a usb key which I mounted on /var/log
29

I think this is to reduce the number of write cycles to the SD card.

Can the archiving be done as part of the rotate cron script so only one cron event is called? Is there a risk of the archiving and rotating scripts getting horribly out of sync.

Can you cascade logrotate? so rotate the hourly into a daily into a weekly?

30

I agree, but this is a difficult question as any new logs created during the prior boot up will be overwritten by either the blank file structure or the log files back up, I cannot see a way around that unless the logfiles are parsed, eg the pre-existing auto created logs content is appended to the end of the restored back up logs.

Again a tough one. I too like to see the logs persisted and what we are trying to achieve with backing up the ram logs every hour is a “low-write” persisted log, rather than several processes writing and rewriting log files in real time we are buffering the writes and “backing up” the logfiles periodically.

There is a risk here that something gets lost if the Pi is not shutdown gracefully in the 59th minuet of the hour, but the alternative is too increase the backup frequency to every 1min or 10 mins etc, which may add to the workload but the emoncms buffered write is already every 1min (AFAICR).

I would like to consider another feature for this logfile manager, and that is too have the option to add a remote location to stash logfiles, using rsync or similar, a duplicate set of logfiles could be maintained remotely, the frequenct that these are synced could be more frequent than the integral backups, thus less data lost in a crash instance.

This is a good option, I suggested this for the 3rd writable “data” partition on an emonPi, the advantage is that if the increased writes are fatal to the USB stick, it doesn’t effect the root filesystem, and It can be easily pulled out to backup or read in another PC.

But for now I think you have a good system evolving, just the initial log content to debate.

31

That issue could be worked around by using ramlog. Here’s an excerpt from the ramlog web page:

Ramlog act as a system daemon. On startup it creates ramdisk, it copies files from /var/log into ramdisk and mounts ramdisk as /var/log. All logs after that will be updated on ramdisk. Logs on harddrive are kept in folder /var/log.hdd which is updated when ramlog is restarted or stopped. On shutdown it saves log files back to harddisk so logs are consistent. Ramlog 2.x is using tmpfs by default, ramfs and kernel ramdisk are suppored as well. Program rsync is used for log synchronization.

1 Like
32

Good call Bill, I wonder why you haven’t mentioned it before :wink::wink::grin:

Some previous mentions
https://openenergymonitor.org/forum-archive/node/10773.html
https://openenergymonitor.org/forum-archive/node/11803.html
and it is used in DietPi too
https://openenergymonitor.org/forum-archive/node/12384.html

it hasn’t had a warm reception when Bill has previously suggested it and I’m not entirely sure why, perhaps it was considered too complex for simply writing the files to RAM. But now we are considering adding all this functionality, perhaps we should take another look.

I recall previously I wasn’t overly happy about the fact I can find no changes to the code (or the website) since 2010 and the info is fairly limited. that doesn’t mean it’s a non-runner for me, but I like to see activity and peruse the repos/issues. Plus I prefer installing via a repo such as debian or PECL or PIP etc rather than private zip’s, again not a show stopper.

I have since seen this project

which is still active (last updated 4 days ago). I haven’t looked at either in-depth so I cannot say if they would work for us or which is more suitable (or are there any others?)

Perhaps we can learn something from those projects even if we don’t fully embrace any one of them.

[Edit] - Just having a nose around the log2ram repo and see there are a couple of familiar faces using it @Tim (Current rsync usage may cause excessive SD card writes · Issue #45 · azlux/log2ram · GitHub) and Peter Scargill (Mosquitto and Apache · Issue #18 · azlux/log2ram · GitHub)

I have attached a copy of ramlog in a zipped folder for easier viewing. There is a file in there called “copying” and it states it’s ok to copy/distribute ramlog but not to modify it, I wonder if that includes repackaging it for systemd? I can see how/why log2ram might have started.

ramlog-2.0.0.zip (25.7 KB)

1 Like
33

I think this is what DietPi uses by default.

34

Indeed it is. PB made mention of that a couple of posts above this one. :wink: