Pause emoncms database updates?

Dunum · 7 April 2019 17:39

Hi
Installed emoncms on my Raspberry Pi / HDD the long way…
(emoncms/readme.md at master · emoncms/emoncms · GitHub)
…stopping short of installing emonhub. All is fine.
Now looking to (re)write the import / export scripts since there are hardcoded paths which don’t match this installation.
I note there is no feedwriter service (I assume this comes with emonhub) so the “sudo service feedwriter stop / restart” commands are irrelevant.
However, I have data arriving from http api messages directly from Arduino devices and from node-red (node-red-node-emoncms) on both local and remote Raspberry Pi’s. How can I pause updates to the feeds in the MySQL database during the backup (and restore?) process?
Thanks in advance.

borpin · 8 April 2019 05:59

Firstly, you cannot pause without losing data in this scenario. The Backup Module stops the feedwriter service, so incoming data will be stored in Redis until the service is restarted. There is no reason not to run feedwriter, even in your circumstance, even though the reduction of disk writes is not the driving factor. There has been some work around the service so it is very useful.

Have you installed the backup Module? It uses the $homedir property set in settings.php. Other than finding a good place to put it on my system, it did just work after modifying the config.

When did you do the install? I ask as there have been some changes to how service-runner and feedwriter services are setup.

Dunum · 10 April 2019 16:31

Hi Brian, thanks for your reply. Tried a few things (always dangerous!) and still have problems.
Yes - I had installed the Backup Module (same link that you posted) and have now enabled ‘$homedir’ property in ‘settings.php’ to ‘/home/pi’.
Completed the emoncms install (the link in my original post) by following the instructions at ‘Install Emonhub’. Changed cmdline.text to ‘dwc_otg.lpm_enable=0 console=tty1 root=/dev/sda2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait’. I assumed ‘root=/dev/sda2’ was correct (sda1 is ‘part /boot’).
All installs have been in the last couple of weeks (emoncms version 9.9.8).
Two immediate problems:

No sign of ‘feedwriter’. Server information panel on the ‘Administration’ page shows:
emonhub Active Running
service-runner Active Running
redis-server Active Running
mosquitto Active Running
The ‘Create Backup’ button on the ‘Backup’ page does nothing (nothing displayed in the panel below the button) - although the ‘Import Backup’ does!

Sorry to be a pain - any help much appreciated!

borpin · 10 April 2019 18:12

You probably need to manually install it. Instructions here - this is included with the emoncms repo so the files are already on your system.

Yes it fails silently in some circumstances. Depending on where you installed it to, did you need to edit the paths in the config.cfg file?

Try just running emoncms-export.sh from the command line and see what it does.

Dunum · 19 April 2019 12:11

Both suggestions worked. Many thanks for your help.

Dunum · 4 May 2019 17:36

Hi Brian. A couple of issues with feedwriter:

Although feedwriter seems to be working, I’ve noticed that each time the service starts I get an error in “/var/log/emoncms.log” saying “2019-05-02 19:21:38.830|ERROR|feedwriter.php|Starting feedwriter script”. Any idea what I’ve done wrong?
During the install of feedwriter I needed to change “Redis Low-write mode | ‘enabled’ => false” to true in settings.php. Was this correct? If so,maybe worth mentioning this in the installation instructions.

Any help much appreciated!

borpin · 5 May 2019 07:05

This is jut telling you it has started - I have no idea why it is logged as an ERROR, it should be INFO.

Not sure - @pb66?

pb66 · 5 May 2019 09:37

Yes that’s correct, I have no doubt it could be better documented.

There is also a setting for the interval between saves, if you have used the /emoncms/default.settings.php as a template rather than the /emoncms/default.emonpi.settings.php then that will be set to 10mins rather than the recommended 1min, so you might want to change that to 60 as well.

github.com

emoncms/emoncms/blob/9.9.8/default.emonpi.settings.php#L48-L52


      
          // Redis Low-write mode
          'redisbuffer'=>array(
              'enabled' => true      // If enabled is true, requires redis enabled and feedwriter service running
              ,'sleep' => 60          // Number of seconds to wait before write buffer to disk - user selectable option
          ),

github.com

emoncms/emoncms/blob/9.9.8/default.settings.php#L49-L53


      
          // Redis Low-write mode
          'redisbuffer'=>array(
              'enabled' => false      // If enabled is true, requires redis enabled and feedwriter service running
              ,'sleep' => 600          // Number of seconds to wait before write buffer to disk - user selectable option
          ),

Robert.Wall · 5 May 2019 10:45

Did you mean 600?

pb66 · 5 May 2019 10:51

No, 10mins is too long, nobody uses 10mins, the only place 10mins is set, is where feedwriter is “enabled = false” so ignored. Since he has had to enable the feedwriter (aka low-write) the 600 undoubtedly also needs reducing to 60 inline with the emonpi settings as that is the setting used when it’s active.

No, I have no idea why it’s 600 in the first place, I have brought it up before!

Dunum · 5 May 2019 12:03

Thanks all for your input. I had already changed the sleep time to 60 - this is self build with HDD, so 600 seemed far too long. Although I don’t need feedwriter to preserve SD lifetime, it will be useful during export (emoncms-export.sh) and import (emoncms-import.sh) activities.

pb66 · 5 May 2019 12:17

In that case I would bring it right down to 5 or 10 secs, in fact i would even try 1 or 0 seconds if it were me, but i do not know if the 0 or 1 would definitely work, hence the suggestion of a more relaxed 5 or 10s, there is no reason that wouldn’t work and no reason to buffer data any longer than that just to get the ability to pause writing data to a hdd.

[edit]

Also, just FYI, I have in the past, just unmounted my data folder (I have a partition purely for data) and emoncms has continued to buffer data in redis (I do not have feedwriter installed/running) until i remounted the data partition/folder. I have no idea how long that might work, it’s not a recognised method. But the advantage of this is that the characteristics of the feeds are not altered.

When feedwriter is enabled you cannot edit/post data older than the last entry. And more specifically with phpfina feeds, when multiple updates occur within the duration of a given fixed interval, with feedwriter enabled, the first presented is kept, whilst when not enabled, the last update is kept.

It’s a small thing, but it can cause confusion if you are not aware.

borpin · 5 May 2019 17:48

Feedwriter will also buffer when you do a backup so again, no data is lost.

It was originally designed (AIUI) to help reduce writes to SD Card and increase the life. It does more than that so is useful in any setup.

pb66 · 5 May 2019 19:16

Yes I get the reason he want’s to use feedwriter!

You forget that everything is (initially) emonpi/sd centric whether that’s appropriate or not. The feedwriter’s sole reason for existence was indeed to lower disk writes to a manageable level by batch writing. That is an understandable move to preserve the sdcard life, for sdcard users.

When the backup module was introduced, it was only for emonpi users, this was only because that was the target audience at that time and I imagine because the feedwriter offered a convenient way of pausing the writing of feeds for the purpose of backing up.

So in actual fact, the incidental backdoor approach to rolling out a solution for a set user base has actually blocked the development of a simple “pause” mechanism to halt feed writing for the duration of running a backup so that the backup module can be used by a wider audience without resorting to using feedwriter.

You are right it does serve this purpose, but only because the emonpi users alone have been catered for. The price of using the feedwriter is the change in characteristics I outlined. If the backup module was made available to everyone, you would not need to install feedwriter to use the backup module.

I have set up 2 virtual hosts for my emoncms installs, one for data and general access, and one for admin. When I want to do maintenance or backups etc, I just take down the first virtual host site and all my emonhub installs just sit on the data until I re-enable that vhost again. Let alone a backup, I can take a server down and completely rebuild it at my leisure over the course of a week, with zero loss of data, when it comes back up, all the buffered data is posted. this is handy for updates and other maintenance too.

The bottom line is, the fact feedwriter was “handy at the time” when the backup module was implemented and only emonpi users were considered, means if you want to use the backup module, yes you need the feedwriter too, regardless of whether you are on a sdcard. Once again non-emonpi users are at the back of the queue and need to make sacrifice to get emonpi features (ie the loss of ability to post earlier data and the first, not an amended fixed-value sticking).

Juerg1 · 7 June 2019 23:35

Not sure my issue is related to the above?

When I run a successful backup of my local emoncms with one EmonPi on emonSD-07Nov16, after the backup completed, there are no more data feeds stored per time slot, but the EmonPi display shows the correct current wattage draw. Once I reboot the EmonPi, all is back to normal, but the data from start of backup till reboot is lost.

Is that normal expected behaviour? I wasn’t like that some months ago I believe.

pb66 · 8 June 2019 13:32

That doesn’t sound right at all. But why are you rebooting? It shouldn’t be necisary and that will be why the data is permanently lost.

It sounds like the feed writer isn’t restarting after being paused for the duration of the backup. When did you last update the emonpi?

If it happens again, please check the admin page to see if the feedwriter is running, it should tell you how many data points are pending writing to disk too and/or if you have ssh access to the command line try

sudo systemctl status feedwriter

and if needed,

sudo systemctl restart feedwriter

Juerg1 · 8 June 2019 22:21

Hi @Paul,

Rebooted it because it’s the only thing I could think of after noticing the data is no longer stored. (I have no understanding of internals such as feedwriter prior to this thread).

Since I posted my issue, I have updated to 10.0.2. The issue occurred on 9.9.2 twice in the last 3-4 months, every time when making a backup.

Once I have fixed my other backup problem with # comments, I will run a new backup and follow your advice of checking the feedwriter status.

Please bear with me, and I will update you in the coming days.

Many thanks, you are of great help indeed to a novice like me J.

Juerg1 · 10 June 2019 04:51

Hi again pb66,

An update for you: Managed to run a backup (after removing all comments from the config.cfg file).

Completed successfully, but once again, after the backup the feedwriter did not restart:

pi@emonpi(ro):~$ sudo systemctl status feedwriter

● feedwriter.service - Emoncms feedwriter script

Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; disabled)

Active: inactive (dead)

Docs: emoncms/install-service-feedwriter.md at master · emoncms/emoncms · GitHub

Restarted it myself, no problem, but why does it not restart by itself (anymore?)

Many thanks in advance.