API returning no values

Hi,
I’ve got a setup going now with multiple input nodes posting to several feeds in a self-hosted emoncms instance.
On the feeds page, I can see data going in and if I view a graph of feeds, I get some wiggly lines. However, hitting the API using feeds/data.json with the key, a valid feed ID and some sensible times (millisecond linux format) returns no data. If I “damage” the GET request, I get expected errors (invalid API key, non-existant feed id, too many data points - more than 8928) so it seems the API itself is working and can see datapoints, but won’t return them.


Get request:
http://xxx.xxx.xxx.xxx/feed/data.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&id=5&start=1579478400000&end=1579521016000&interval=60

I can see on the initial summary screen that the “feedwriter” service is not running - is that going to have a bearing on this? If so, any clues as to why it may have stopped and how to kick it into life again?

Many thanks,
Alyn.

Try seconds - Javascript returns time in miliseconds.

definitely miliseconds :man_facepalming:

How have you installed the system? If you can see data on a graph then data has reached the ‘feed’.

Generally I’d expect feedwriter to be running. It buffers data so reducing write load on the SD Card but graphs etc pull data off the written to file data.

Thanks Brian,
I did try seconds initially (because the time converter I found defaulted to seconds since 1970). When I didn’t get anything from the API I went looking and found an old archived thread on openenergymonitor.org that said to use millisconds (so add three zeros) so assumed that was correct.
archived thread
I’ve just tried seconds again and found the same issue.

Cheers,
Alyn

If you go into graphs, select a feed, look at the line output, then view the CSV output. If you put those times in do you get any data?

If you use the FeedAPI to just return the last value, what do you get?

I am guessing you have found the API help (top right of Feeds view)?

Hi (again) Brian,
I did a standard install on a LAMP stack running on a Ubuntu Core virtual HyperV server. Maybe feedwriter doesn’t need to run if it’s not on an SD card and it’s clever enough to know that.

Cheers,
Alyn

Which instructions? Some are very out of date. I’d have suggested using the EmonScripts.

If I use the times from the CSV view on the graph page, I do NOT get any data.

If I use the feed/list.json or feed/timevalue.json for the same feed ID, I DO get the expected value back.

I have indeed been through the feeds API help page (it’s how I got my API read key). timevalue, value and fetch all seem to work, but data.json does not. It’s only time parameters that are added, so I assume something is up with them.
As I said, I can get error messages from the API by including ridiculous values in the time parameters and/or interval parameter (so I can get the “too many values” error by either decreasing the start timestamp sufficiently, or decreasing the interval value). It’s just when the timestamp parameters are valid values, I get nothing back.

As for the install, I was following the instructions here, which took me to the EmonScripts page on github. So I think I’ve done as you suggest.

1 Like

If you are in a browser, and logged on in a different tab, you do not need the API key.

I have just tested the data.json API call and with a time in Milliseconds (sic), I am getting data back.

Ok great. What is the install config you used? I created the advice on an Ubuntu install, and I have it running here on a VM with Feedwriter enabled.

Can you click on the button Copy as Markdown next to Server Information on the Admin page and paste into a reply here please (no further formatting required).

As requested:

Server Information

Server Information

Services

  • emonhub :- Active Running
  • emoncms_mqtt :- Inactive Dead
  • feedwriter :- Inactive DeadService is not running197892 feed points pending write
  • service-runner :- Inactive Dead
  • redis-server :- Active Running
  • mosquitto :- Active Running
  • demandshaper :- Activating Auto-restart

Emoncms

Server

  • OS :- Linux 4.15.0-74-generic
  • Host :- emoncms | emoncms | (10.45.42.22)
  • Date :- 2020-01-20 12:57:41 UTC
  • Uptime :- 12:57:41 up 2 days, 20:19, 0 users, load average: 0.00, 0.00, 0.00

Memory

  • RAM :- Used: 23.93%
    • Total :- 1.84 GB
    • Used :- 450.95 MB
    • Free :- 1.4 GB
  • Swap :- Used: 0.00%
    • Total :- 2 GB
    • Used :- 0 B
    • Free :- 2 GB
      Write Load Period

Disk

  • / :- Used: 33.18%
    • Total :- 15.19 GB
    • Used :- 5.04 GB
    • Free :- 9.36 GB
    • Write Load :- n/a
  • /boot/efi :- Used: 1.18%
    • Total :- 510.98 MB
    • Used :- 6.01 MB
    • Free :- 504.98 MB
    • Write Load :- n/a

HTTP

  • Server :- Apache/2.4.29 (Ubuntu) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.1.43-MariaDB-0ubuntu0.18.04.1
  • Host :- localhost:6379 (127.0.0.1)
  • Date :- 2020-01-20 12:57:40 (UTC 00:00‌​)
  • Stats :- Uptime: 245913 Threads: 2 Questions: 293653 Slow queries: 0 Opens: 43 Flush tables: 1 Open tables: 37 Queries per second avg: 1.194

Redis

  • Version :-
    • Redis Server :- 4.0.9
    • PHP Redis :- 5.1.1
  • Host :- localhost:6379
  • Size :- 257 keys (22.78M)
  • Uptime :- 2 days

MQTT Server

  • Version :- Mosquitto 1.4.15
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.2.24-0ubuntu0.18.04.2 (Zend Version 3.2.0)
  • Modules :- apache2handler | calendar v7.2.24-0ubuntu0.18.04.2 | Core v7.2.24-0ubuntu0.18.04.2 | ctype v7.2.24-0ubuntu0.18.04.2 | curl v7.2.24-0ubuntu0.18.04.2 | date v7.2.24-0ubuntu0.18.04.2 | dom v20031129 | exif v7.2.24-0ubuntu0.18.04.2 | fileinfo v1.0.5 | filter v7.2.24-0ubuntu0.18.04.2 | ftp v7.2.24-0ubuntu0.18.04.2 | gd v7.2.24-0ubuntu0.18.04.2 | gettext v7.2.24-0ubuntu0.18.04.2 | hash v1.0 | iconv v7.2.24-0ubuntu0.18.04.2 | json v1.6.0 | libxml v7.2.24-0ubuntu0.18.04.2 | mbstring v7.2.24-0ubuntu0.18.04.2 | mosquitto v0.4.0 | mysqli v7.2.24-0ubuntu0.18.04.2 | mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 3591daad22de08524295e1bd073aceeff11e6579 $ | openssl v7.2.24-0ubuntu0.18.04.2 | pcre v7.2.24-0ubuntu0.18.04.2 | PDO v7.2.24-0ubuntu0.18.04.2 | pdo_mysql v7.2.24-0ubuntu0.18.04.2 | Phar v2.0.2 | posix v7.2.24-0ubuntu0.18.04.2 | readline v7.2.24-0ubuntu0.18.04.2 | redis v5.1.1 | Reflection v7.2.24-0ubuntu0.18.04.2 | session v7.2.24-0ubuntu0.18.04.2 | shmop v7.2.24-0ubuntu0.18.04.2 | SimpleXML v7.2.24-0ubuntu0.18.04.2 | sockets v7.2.24-0ubuntu0.18.04.2 | sodium v7.2.24-0ubuntu0.18.04.2 | SPL v7.2.24-0ubuntu0.18.04.2 | standard v7.2.24-0ubuntu0.18.04.2 | sysvmsg v7.2.24-0ubuntu0.18.04.2 | sysvsem v7.2.24-0ubuntu0.18.04.2 | sysvshm v7.2.24-0ubuntu0.18.04.2 | tokenizer v7.2.24-0ubuntu0.18.04.2 | wddx v7.2.24-0ubuntu0.18.04.2 | xml v7.2.24-0ubuntu0.18.04.2 | xmlreader v7.2.24-0ubuntu0.18.04.2 | xmlwriter v7.2.24-0ubuntu0.18.04.2 | xsl v7.2.24-0ubuntu0.18.04.2 | Zend OPcache v7.2.24-0ubuntu0.18.04.2 | zlib v7.2.24-0ubuntu0.18.04.2

Ok, here is your problem

DeadService is not running197892 feed points pending write

Can you post output of

sudo systemctl status feedwriter.service

[edit]
You also have some other service issues as well!

That’s for sure! I thought things were generally working though… Anyway, here’s the output -

● feedwriter.service - Emoncms feedwriter script
   Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; static; vendor present: enabled
  Drop-In: /lib/systemd/system/feedwriter.service.d
           └─feedwriter.conf
   Active: inactive (dead)
     Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-feedwriter.md

Trying to start the service gets:

● feedwriter.service - Emoncms feedwriter script
   Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; static; vendor preset: enabled)
  Drop-In: /lib/systemd/system/feedwriter.service.d
           └─feedwriter.conf
   Active: activating (auto-restart) (Result: exit-code) since Mon 2020-01-20 15:54:13 UTC; 5s ago
     Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-feedwriter.md
  Process: 2282 ExecStartPre=/bin/chown ${USER} ${LOG_PATH} (code=exited, status=1/FAILURE)
  Process: 2276 ExecStartPre=/bin/mkdir -p ${LOG_PATH} (code=exited, status=0/SUCCESS)

Jan 20 15:54:13 emoncms systemd[1]: feedwriter.service: Control process exited, code=exited status=1
Jan 20 15:54:13 emoncms systemd[1]: feedwriter.service: Failed with result 'exit-code'.

Not sure how familiar you are with Linux Services. I’d have expected to see some log entries at the bottom of that message. Could you try;

sudo systemctl restart feedwriter.service

and then the status command again please.

[edit]
Can you please

cat /lib/systemd/system/feedwriter.service.d/feedwriter.conf

It looks like the user set to run the service does not have permission to setup the logfile.

Did you edit the service file at all?

Hi Brian,
Sorry, thought I’d try to start the service and editted my reply after you’d asked this. The result of using “restart” instead of “start” is the same though. The /bin/chown ${USER}… process is failing.

Thanks,
Alyn

2 things.

I didn’t follow this up - did you do this before starting? https://github.com/openenergymonitor/EmonScripts/blob/master/install/readme.md#ubuntu

2nd did you see my edit (above) re the drop-in?

I think there has been an issue with the Install and it is probably user/permission problems.

Ah, that looks like the badger! The username in the .conf file was an “interim” one I’d used while setting up and then renamed to match our naming convention. I’ve editted the .conf to include the new name and the service started OK. It’s showing as green on the admin dashboard now too.

I’m also now getting JSON back from the query.

Thank you so much for your help this afternoon.

Kind Regards,
Alyn

Probably the same issue for the other services.