Problem with Feedwriter

Dunum · 2020-09-21T22:56:08.267Z

Hi
I have self build EMONCMS hosted on a Raspberry Pi running on a Western Digital HDD - which has been working well for a long time now and from a user point of view, continues to do so. Input data, feeds, dashboards, etc. all seem ok. However the Administration page reports:

“feedwriter Activating Auto-restartService is not running 136 feed points pending write”

The number of feed points pending write varies up and down over time.

“journalctl -f -u feedwriter” says:

Sep 21 23:34:45 Pegasus feedwriter[17202]: thrown in /var/www/emoncms/Modules/feed/engine/RedisBuffer.php on line 213
Sep 21 23:34:45 Pegasus feedwriter[17202]: Fatal error: Uncaught Error: Call to undefined method PHPFiwa::post_bulk_prepare() in /var/www/emoncms/Modules/feed/engine/RedisBuffer.php:213
Sep 21 23:34:45 Pegasus feedwriter[17202]: Stack trace:
Sep 21 23:34:45 Pegasus feedwriter[17202]: #0 /var/www/emoncms/Modules/feed/engine/RedisBuffer.php(176): RedisBuffer->process_feed_buffer(‘164’)
Sep 21 23:34:45 Pegasus feedwriter[17202]: #1 /var/www/emoncms/scripts/feedwriter.php(62): RedisBuffer->process_buffers()
Sep 21 23:34:45 Pegasus feedwriter[17202]: #2 {main}
Sep 21 23:34:45 Pegasus feedwriter[17202]: thrown in /var/www/emoncms/Modules/feed/engine/RedisBuffer.php on line 213
Sep 21 23:34:45 Pegasus systemd[1]: feedwriter.service: Main process exited, code=exited, status=255/n/a
Sep 21 23:34:45 Pegasus systemd[1]: feedwriter.service: Unit entered failed state.
Sep 21 23:34:45 Pegasus systemd[1]: feedwriter.service: Failed with result ‘exit-code’.

…which repeats continuously every minute.

Version low-write 10.2.6
Modules Administration | Dashboard v2.0.7 | EventProcesses | Feed | Graph v2.0.9 | Graph v2.0.9 | Input | CoreProcess | Schedule | Time | User | Visualisation
Git
URL GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data
Branch * stable
Describe 10.2.5-3-gf781bed0

“git pull” reports “Already up-to-date.”

Any help “fixing” this would be much appreciated.

borpin · 2020-09-22T07:49:20.368Z

Try this.

Service-runner stuck on Activating Auto-restart after full update Emoncms

Thanks. Worked for me. It needed to do a reboot after the “sudo apt-get upgrade”.

[edit]
Actually, I don’t think it will help.

Dunum · 2020-09-22T20:54:41.609Z

Hi Brian
Thanks for your reply.
FYI.

Not sure when the problem started but it certainly was working fine in the past. I have not (knowingly) changed the configuration or updated anything relating to emoncms.
In "/var/log/emoncms/emoncms.log I do see a few MQTT connection issues…
“2020-09-22 17:57:59.140|WARN|emoncms_mqtt.php|Not connected, retrying connection
2020-09-22 17:57:59.349|WARN|emoncms_mqtt.php|Connecting to MQTT server: Connection Accepted.: code: 0”
…three instances today, spread over 6 hours. I assumed that these were temporary network issues and not relevant to the problem in hand.
“settings.php” is unchanged from an old backup - so I don’t think there is any problem there.
I do use a “homebrewed” backup script which stops feedwriter while copying the database and feed folders, then restarts feedwriter. Been using this for some time with no problems.
Have attached a dump of emoncms server information - looks ok to me!

As per your suggestion, I tried:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-pip {SAYS: python3-pip is already the newest version (9.0.1-2+rpt2).}
pip3 install redis
sudo reboot

As per your “edit”, no change! However “sudo systemctl status service-runner” reports “Unit service-runner.service could not be found.”. Mine is a manual build - not sure if I ever had this installed or whether it has “disappeared”.

Also, ran phpMyAdmin, just to check the database. Initially, when I clicked on “feeds” and “input” tables both showed an error (something like “scroll down for more information which revealed a lot of text I couldn’t really understand”). But then ran “CHECK TABLE dashboard, feeds, graph, input, multigraph, rememberme, schedule, users” and all tables reported as OK! Subsequent clicks on “feeds” and “input” tables showed no error messages. Rebooting revealed that the problem with feedwriter persisted.

Admin page now reports…
“feedwriter: Activating Auto-restartService is not running7035 feed points pending write”
…but all still looks ok in inputs, feed lists, dashboards, etc!

Confused ¯_(ツ)_/¯

Server Information.txt (2.9 KB)

STOP PRESS!
re. Database issue discovered when using phpMyAdmin, was browsing “input” table, clicked on a column header to re-order the table and got the error again. File attached. Do I have a corrupt database?

2020-09-221369×930 104 KB

borpin · 2020-09-22T21:21:28.703Z

Easy one first

Dunum:

In "/var/log/emoncms/emoncms.log I do see a few MQTT connection issues…
2020-09-22 17:57:59.140|WARN|emoncms_mqtt.php|Not connected, retrying connection
2020-09-22 17:57:59.349|WARN|emoncms_mqtt.php|Connecting to MQTT server: Connection Accepted.: code: 0
…three instances today, spread over 6 hours. I assumed that these were temporary network issues and not relevant to the problem in hand.

Yes you often see these, just seems to not connect first time for some reason.

Dunum:

“feedwriter: Activating Auto-restartService is not running7035 feed points pending write”
…but all still looks ok in inputs, feed lists, dashboards, etc!

Take care, none of that data is written to disk! You will run out of memory eventually I suspect.

Just searched on post_bulk_prepare in the forum, looks like it may be a redis issue and the only solution is to flush the cache - that may dump all that data.

The other thing that occurs to me is that this may be the first signs of an SD card failure.

Dunum · 2020-09-23T18:32:11.451Z

OK, so some progress.
Looking at line 213 of RedisBuffer.php (which was reported as the fail point in the feedwriter log), it says…

“$this->feed->EngineClass($engine)->post_bulk_prepare($feedid,$time,$value,$arg);”

…I surmised that there may be a problem writing a specific feed to the database. So I shut down all sources of inputs to emoncms and rebooted. Once emoncms had restarted, the feedwriter service in emoncms Administration | Server Information was alive and well. I re-introduced some input sources and the feedwriter service was happy. Monitoring while I added more, I found that one source with 6 inputs recreated the feedwriter fault! After turning the source off, stopping feedwriter, flushing Redis, restarting feedwriter, all was well again!

Thinking that removing the offending inputs and related feeds using the web interface might clear the problem. I deleted the feeds but when trying to delete the inputs, although the message sequence suggested they had been deleted, they remained in the input list. When trying to edit any of the process lists assigned to the six inputs, I got the message “ERROR: Could not save processlist. undefined”. I was then probably trying to be a bit too clever but, I wound up phpMyAdmin and deleted the inputs directly in the “input” table browser page. The six input records duly disappeared.

Two problems:

When looking at the input list in emoncms input view, the inputs are still present (tried clearing my browser cache and also a complete power cycle but the inputs are still showing. Is emoncms storing database data outside of the database? Maybe a cache somewhere needs clearing?
I still get database error described in “STOP PRESS” in my last post. Presumably there IS a corruption. Is there some way to check / repair the integrity?

By the way,

The other thing that occurs to me is that this may be the first signs of an SD card failure.

My Pi has a hard disk drive, not a SD card. Will try to find a way to perform disk check later.

Sorry this is such a rambling post.
Any help would be much appreciated.

borpin · 2020-09-23T20:45:29.392Z

@TrystanLea is probably better placed to answer this.

Dunum · 2020-09-28T16:59:07.877Z

For anyone who may suffer from the same problems I had…

Emoncms input view shows non-editable inputs which are not present in the emoncms database…

Is emoncms storing database data outside of the database? Maybe a cache somewhere needs clearing?

…yes! When I executed sudo redis-cli FLUSHALL the “ghost” inputs disappeared.

2.Database error message when using phpMyadmin…

was browsing “input” table, clicked on a column header to re-order the table and got the message “Some errors have been detected on the server! Please look at the bottom of this window”

This seems to be a known problem with my version of phpMyadmin (4.6.6deb4+deb9u1). Fixable by masking the error, by adding a line to /etc/phpmyadmin/config.inc.php; or by upgrading phpMyadmin (and probably PHP). See https://stackoverflow.com/questions/27370372/phpmyadmin-pop-up-error-notice-keeps-appearing-when-clicking-on-columns-of-datab

I didn’t try either since I seldom use phpMyadmin.

Not sure what caused my original feedwriter problem but all seems ok now.