Feeds via API not updating after SD card update

carl_afgte45 · 28 May 2023 11:13

I have just updated my emonPI to the latest image (SD card purchased from shop) the import process went OK, sensors on the emonPI itself CT1,CT2 etc are working OK, but
external sensors I I have calling the PI are not updating.

I have a number of sensors round the house that make http calls to the emonPI e.g the immersion diverter:

http://192.168.1.57/input/post?node=elec_loads&fulljson={“divert”:494.7414749999999}&apikey=70619f2a2b25cb7d32**************
(api key have been obsfuriated)

These are getting a response of {“success”: true} - but the feeds are not updating in emon

I can even call the URL with API key in it from a browser and see the {“success”: true} but the feed does not update in the emon web screen

This affects all the feeds that log to the emonPI via API. Kassa plugs, weather station and a raspberyPI in the garage with some temperature sensors on the boiler

Am I missing something? or is there a log file where I can see what’s going wrong. The SD card is back on my desk, but I can mount it under linux using a card reader in my laptop to see the filesystem

I have replaced the original SD card and everything works fine again - feeds update.

vvvv emonPI info below vvvvv

—Server Information

Services
emonhub : Active Running
emoncms_mqtt : Active Running
feedwriter : Active Running - sleep 300s 380 feed points pending write
service-runner : Active Running
emonPiLCD : Active Running
redis-server : Active Running
mosquitto : Active Running
demandshaper : Not found or not installed
Emoncms
Version : low-write 11.3.20
Git :
URL : GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data
Branch : * stable
Describe : 11.3.20
Components : Emoncms Core v11.3.20 | App v2.7.2 | EmonHub Config v2.1.5 | Dashboard v2.3.3 | Device v2.2.2 | Graph v2.2.3 | Network Setup v1.0.2 | WiFi v2.1.1 | Backup v2.3.2 | Postprocess v2.2.7 | Sync v2.1.4 | Usefulscripts v2.3.10 | EmonScripts v1.6.18 | RFM2Pi v1.4.1 | Avrdude-rpi v1.0.1 | Emonhub v2.5.7 | EmonPi v2.9.5

Memory
RAM : Used: 22.80%
Total : 922.14 MB
Used : 210.25 MB
Free : 711.88 MB
Swap : Used: 0.00%
Total : 100 MB
Used : 0 B
Free : 100 MB

Disk
: - / : Used: 40.84%
Total : 5.78 GB
Used : 2.36 GB
Free : 3.11 GB
Read Load : n/a
Write Load : n/a
Load Time : n/a
/boot : Used: 19.52%
Total : 254.99 MB
Used : 49.78 MB
Free : 205.21 MB
Read Load : n/a
Write Load : n/a
Load Time : n/a
/var/opt/emoncms : Used: 2.43%
Total : 9.61 GB
Used : 239.16 MB
Free : 8.88 GB
Read Load : n/a
Write Load : n/a
Load Time : n/a
/var/log : Used: 3.73%
Total : 50 MB
Used : 1.87 MB
Free : 48.13 MB
Read Load : n/a
Write Load : n/a
Load Time : n/a

HTTP
Server : Apache/2.4.54 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL
Version : 10.5.15-MariaDB-0+deb11u1
Host : 127.0.0.1 (127.0.0.1)
Date : 2023-05-28 11:46:47 (UTC 01:00‌)
Stats : Uptime: 2088 Threads: 7 Questions: 6396 Slow queries: 0 Opens: 47 Open tables: 40 Queries per second avg: 3.063

Redis
Version :
Redis Server : 6.0.16
PHP Redis : 6.0.0-dev
Host : localhost:6379
Size : 387 keys (775.23K)
Uptime : 0 days

MQTT Server
Version : Mosquitto 2.0.11
Host : localhost:1883 (127.0.0.1)

PHP
Version : 8.1.12 (Zend Version 4.1.12)
Run user : User: www-data Group: www-data video Script Owner: pi
Modules : apache2handler calendar Core ctype curl date dom v20031129exif FFI fileinfo filter ftp gd gettext hash iconv json libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 8.1.12openssl pcre PDO pdo_mysql Phar posix readline redis v6.0.0-devReflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer xml xmlreader xmlwriter xsl Zend OPcache zlib
Pi
Model : Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)
Serial num. : 1D81EC23
CPU Temperature : 46.16°C
GPU Temperature : N/A (to show GPU temp execute this command from the console “sudo usermod -G video www-data” )
emonpiRelease : emonSD-10Nov22
File-system : read-write

Client Information

HTTP
Browser : Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0
Language : en-GB,en;q=0.5

Window
Size : 1280 x 881

Screen
Resolution : 1280 x 1024

Vster · 29 May 2023 00:32

It’s early and been a while since I did this but -
a) are the INPUTS updating?
b) your code snip says “divert”, the scene shot says “divert”, but the log print says 'immersion_diverter", something’s not right somewhere, they should all be the same.

borpin · 29 May 2023 07:07

What do you see in the Emoncms Log?

You might need to set the settings.ini debug setting to debug.

Do you mean if you use the URL in a browser, it does or doesn’t update the INPUT (not the FEED). It may be that the INPUT Processing has not imported correctly.

carl_afgte45 · 29 May 2023 21:09

Thanks for the help so far. I put the new SD back in this evening and did some testing. comparing to the original SD.

Thinking back there is a bit of history, the home made diverter meter has always called an input “immersion_divert”, and I created a feed of the same name - but when I tried to use the solar app it would fail, as the feed has to be called “divert” - you can read more of this on the forum thus the input is immersion_divert, but the feed is divert.

On the new SD with “import” the Inputs do indeed seem to update - you can see this in the screenshots, but it is not writing to feeds. there are two identical inputs “immersion_diverter” one updating, other not and showing NULL and another input “divert” (which was the feed name).

It has also messed up the input > feeds for lots of other inputs like my weather station or Kassa sockets - all make http:// calls to log data.

I am guessing the changing of the feed name from “immersion_divert” “divert” is not handled well by the import routine?

Out of time this evening, put the old SD card back, but what’s the best solution?

Ideally I want to “fix” it on the old SD card and then rerun the import routine to new SD. I think this will overwrite whats already been imported to the new SD?

borpin · 30 May 2023 07:09

Yes emoncms sometimes duplicates inputs.

Try just deleting the Input without the Input Processing. It is likely emoncms will just use the original one. If not, you will need to recreate the Input Processing from the old input to the new one.

@TrystanLea - is this a bug?

carl_afgte45 · 1 June 2023 20:58

Thanks for the various pointers, It’s now resolved.

What I did
With the original SD I deleted the “immersion_divert” input. I then changed the immersion diverter so it calls the emon API “divert”, on first use this creates an input “divert” and mapped this to the existing feed “divert”.

With everything working I shut down the emon, swapped to the new SD, putting old SD in card reader. and booted. Running the import from SD option - everything seems to work as expected. Inputs and feeds all come through and are updating

What I think the problem was
I had a feed name that was not the same as the input name “immersion_diverter” vs “divert” - there may also have been some reference to the old feed “immersion_diverter” in the database from before when it was changed. This has worked for for 12 months, but caused problems for the import from SD routines

borpin · 2 June 2023 10:28

No, it doesn’t have to be. You can create more than one Feed from the same input.

carl_afgte45 · 2 June 2023 12:55

I agree they don’t need to be the same for it to work - but when I did the old SD to new SD import process it went wrong when they were different but was OK when they were the same. This may not be the cause of the problem, but it’s the only thing that AFAIK was different between the two attempts.

TrystanLea · 3 June 2023 20:44

Seems like there was some kind of import related bug here. Perhaps a redis cache vs mysql database mismatch due to something in the import timing, it would be a hard one to track down im sure. Good to hear that you have this resolved @carl_afgte45.