Emoncms on raspberry, no inputs from esp8266

chalkmaroon · 2019-05-07T15:22:02.897Z

tshark -Y http.request -T fields -e http.request.full_uri

and resetting the node gives me immediately the following output:

http://172.23.56.146/input/post.json?node=Kitchen&apikey=emonpicmsreadandwriteapikey&json={temperature:22.20,humidity:23.42,pressure:957.39}

The IP differs from my first post, since the node connects to a different router now.
For my humble knowledge it looks all good, but there are still no inputs, yet.
I also put the whole link from above, except the measured values, in the host line of the emoncms setting section,
but again, no success…

pb66 · 2019-05-07T16:06:06.395Z

If you cut and paste that request into a browser bar and enter, what response do you get?

Should it not be

http://172.23.56.146/emoncms/input/post.json?node=Kitchen&apikey=emonpicmsreadandwriteapikey&json={temperature:22.20,humidity:23.42,pressure:957.39}

I get a 404 on my emonSD image if I omit the /emoncms sub-directory.

chalkmaroon · 2019-05-07T17:38:59.110Z

Oh, I missed that, thanks !!
… and cannot try it out before tomorrow, because I forgot the sd card at work…

chalkmaroon · 2019-05-08T17:14:58.370Z

The request from tshark from my last post wasn’t from the node.
I ran it again and from the node comes nothing, but every 5 min an output, that mqtt server got 0 data…

I get a 404 on my emonSD image if I omit the /emoncms sub-directory.

So do I and with /emoncms I also get an error, but didn’t copy it, because this seems to be a dead end for me.
I do not understand what’s going on here and can’t get it to work. I’m not able to expand the code for posting a print line to get the url, too. Maybe I find some ohter code for the nodemcu and emoncms. I just wanted something neat to display temperature and humidity with a little bit of logging and the code that I have was exactly what I wanted, when using it with “emoncms.org” as host for the cms.
But since it is not possible for me to run emoncms locally, I’ll may check for something else.

Thanks all for your help, it was definitely educational
Cheers,
Oli

pb66 · 2019-05-08T20:50:05.065Z

chalkmaroon:

but didn’t copy it, because this seems to be a dead end for me.

That becomes a bit of a self-fulfilling prophecy, it will be a dead end if you do not pay attention to the error messages.

If you tell us what the error is we might be able to help.

If it works for emoncms.org, it should also work for local emoncms. Don’t worry too much about getting the nodemcu to print the url right now.

If you are getting an error and unable to post an input via the api in a browser then that needs resolving first.

Once we are able to post via an api, then we can retest the nodemcu. AFAIA all you may need to do is alter the const char* host = "192.168.0.133" to const char* host = "192.168.0.133/emoncms" IF we are able to get the url with /emoncms in it to work, but not without it.

chalkmaroon · 2019-05-09T08:34:28.859Z

Paul, your’re right!! Yesterday was frustrating and killed my motivation

Ok, a added /emoncms in the browser:
http://172.23.56.146/emoncms/input/post.json?node=Kitchen&apikey=emonpicmsreadandwriteapikey&json={temperature:22.20,humidity:23.42,pressure:957.39}
gives back in the browser window:
{"success": false, "message": "Format error, json value is not numeric"}

AFAIA all you may need to do is alter the const char* host = "192.168.0.133" to const char* host = "192.168.0.133/emoncms" IF we are able to get the url with /emoncms in it to work, but not without it.

I already tried that before, but got the same results:
no inputs and enter the link from above in the browser gives the same output:

{"success": false, "message": "Format error, json value is not numeric"}

pb66 · 2019-05-09T08:51:20.824Z

Good to hear the fight isn’t over yet!

So looking at the input api docs

image1345×415 25.3 KB

it would seem the format of the url isn’t right, unless you go fulljson and add all the quote marks, you would need to just remove .json from your url eg

http://172.23.56.146/emoncms/input/post?node=Kitchen&apikey=emonpicmsreadandwriteapikey&json={temperature:22.20,humidity:23.42,pressure:957.39}

chalkmaroon:

I already tried that before, but got the same results:
no inputs and enter the link from above in the browser gives the same output:

As I said, this isn’t important until you get it working in a browser, BUT it is good news that it is returning the same error, that potentially indicates that all will be well once you fix the url in the browser and introduce those edits (and the /emoncms) to your firmware.

chalkmaroon · 2019-05-09T09:34:38.258Z

pb66:

Good to hear the fight isn’t over yet!

And again, I’ll have to postpone my teaks until tomorrow…
I’ll post back my insights then immediately !

chalkmaroon · 2019-05-10T09:11:48.583Z

Guys, we’re getting closer:

I’ve added /emoncms in the firmware:
const char* host = "192.168.0.133/emoncms";

and putting this into the browser:
http://172.23.56.146/emoncms/input/post?node=Kitchen&apikey=emonpicmsreadandwriteapikey&json={temperature:22.20,humidity:23.42,pressure:957.39}

which gives me back
ok

AND when I’m checking the inputs in the emoncms menu then there are the 3 inputs with the values from the link above !!

So far so good, but I have still no idea, how to get the whole thing with real values into the cms…
Sorry, but at the moment, I’m more confused than in the beginning…

pb66 · 2019-05-10T09:25:01.593Z

So it all sounds good and hopefully the final piece is to remove the .json from the nodemcu firmware. Somewhere it will currently have something like /input/post.json? which needs changing to /input/post?, this might be part of a longer string including node= etc, but the change is the smae, whatever is there that includes .json remove the .json and it should work, if your unsure, post the section of code that needs amending or the whole sketch for us to look at.

chalkmaroon · 2019-05-10T09:58:18.768Z

pb66:

So it all sounds good and hopefully the final piece is to remove the .json from the nodemcu firmware. Somewhere it will currently have something like /input/post.json? which needs changing to /input/post? , this might be part of a longer string including node= etc, but the change is the smae, whatever is there that includes .json remove the .json and it should work, if your unsure, post the section of code that needs amending or the whole sketch for us to look at.

After a reboot I’ll get my 3 inputs, but no values and it’s not updating, the cms shows:
n/a NULL

Here’s the whole code

// Including the Libraries for WiFi Connection, HTTP Send and bme280 Sensor
#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BME280.h>

// Replace with your network details
const char* ssid = "mySSID";
const char* password = "RouterPasswd";
Adafruit_BME280 bme;

//emoncms settings, change node ID for every node
const char* host = "192.168.0.133/emoncms";
const char* nodeId   = "Kitchen";
const char* privateKey = "readandwriteapikey";
float h = 0;
float t = 0;
float p = 0;
int vcc;
ADC_MODE(ADC_VCC);
char temperatureCString[6];
char humiditiyString[6];
char pressureString[7];
const int httpPort = 80;

//Seconds to deep sleep 300 = 5Min
const int sleepTimeInSec = 300;

//Variables to store values


// ==============================================
// only runs once on boot then go deep sleep again
// ==============================================
void setup() {
  
  // Initializing serial port for debugging purposes
  Serial.begin(115200);
  delay(500);

  //Start sensor orginal bei 0x76
  bme.begin(0x76);

  //Connect WiFi
  //Fix  https://github.com/esp8266/Arduino/issues/2186
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");


  // Sensor readings may also be up to 2 seconds 'old', so we read twice
  h = bme.readHumidity();
  t = bme.readTemperature();
  p = bme.readPressure() / 100.0;
  delay(2500);
  h = bme.readHumidity();
  t = bme.readTemperature();
  p = bme.readPressure() / 100.0;
 
  //vcc = ESP.getVcc();//readvdd33();

  // Check if any reads failed and exit early (to try again).
  if (isnan(h) || isnan(t)) {
    Serial.println("Failed to read from bme280 sensor!");
    return;
  }

  Serial.print("Luftfeuchte: ");
  Serial.print(h);
  Serial.print(" %\t Temperatur: ");
  Serial.print(t);
  Serial.print(" *C ");
  Serial.print("%\t Luftdruck: ");
  Serial.print(p);
  
  //Send data to emoncms
  sendData();

  //go to deep sleep
  ESP.deepSleep(30e6);
}

// ==============================================
// Send Data to emoncms
// ==============================================
void sendData() {

  WiFiClient emonHTTPClient;

  //Connect by http
  const int httpPort = 80;
  if (!emonHTTPClient.connect(host, httpPort)) {
    return;
  }

  //Build the JSON for emoncms to send data
  String json = "{temperature:";
  json += t;
  json += ",humidity:";
  json += h; 
  json += ",pressure:";
  json += p; 
  //json += ",vcc:";
  //json += vcc;
  json += "}";

  //Build emoncms URL for sending data to
  String url = "/input/post?node=";
  url += nodeId;
  url += "&apikey=";
  url += privateKey;
  url += "&json=";
  url += json;

  // Send the HTTP Request to the emoncms server
  emonHTTPClient.print(String("GET ") + url + " HTTP/1.1\r\n" +
                  "Host: " + host + "\r\n" +
                  "Connection: close\r\n\r\n");
  delay(10);

  // Read all the lines of the reply from server and print them to Serial
  while (emonHTTPClient.available()) {
    String line = emonHTTPClient.readStringUntil('\r');
  }

  emonHTTPClient.stop();
}



void loop(void) {
}

Sorry, it’s not formatting correctly… [no worries, I’ve formatted it by adding ``` (3x backticks) on the lines before and after the code - pb66]

pb66 · 2019-05-10T10:34:50.971Z

chalkmaroon:

After a reboot I’ll get my 3 inputs, but no values and it’s not updating, the cms shows:
n/a NULL

If the firmware previously worked and the only changes you have made are editing the apikey, the server IP (incl adding /emoncms) and removing the .json. I don’t see why it wouldn’t work, this has been proven by the manual use of the url in the browser.

The only potential difference I can think of is if you are logged in when you try the url. Can you therefore try the url again in an incognito browser session (or just log out of emoncms) . This will also test the apikey is correct.

What reply appears in the nodemcu serial?

this bit of the firmware looks incomplete to me (but I’m no expert on esp8266)

  // Read all the lines of the reply from server and print them to Serial
  while (emonHTTPClient.available()) {
    String line = emonHTTPClient.readStringUntil('\r');
  }

to me it looks like it gets the reply, but doesn’t make any attempt to print it, try adding a serial.print line here eg

// Read all the lines of the reply from server and print them to Serial
  while (emonHTTPClient.available()) {
    String line = emonHTTPClient.readStringUntil('\r');
    Serial.print(line);
  }

but that is a guess and untested, if anyone can confirm or correct my guess, please do.

borpin · 2019-05-10T11:15:25.282Z

chalkmaroon:

// Send the HTTP Request to the emoncms server emonHTTPClient.print(String("GET ") + url + " HTTP/1.1\r\n" + "Host: " + host + "\r\n" + "Connection: close\r\n\r\n"); delay(10);

I suggest you leave the Host variable as just the IP addresss and add the emoncms path here

String url = "/emoncms/input/post?node=";

Then use tshark again to see what is actually posted.

borpin · 2019-05-10T11:20:04.662Z

BTW have you come across ESPHome? Very simply generic code for esp8266 based items.

Unless you want to do a bit of coding, it will probably be simpler to implement.

pb66 · 2019-05-10T12:05:08.984Z

borpin:

I suggest you leave the Host variable as just the IP addresss and add the emoncms path here

That makes perfect sense actually!

When I advised to add it to the url, we couldn’t see the whole sketch and I assumed a usual basic url formation. You are right, I have seen this before, the server actually is 172.23.56.146 not 172.23.56.146/emoncms. adding /emoncms to the server info (ie 172.23.56.146) was a lazy way of doing it that only works with full url string formation, not when the components are passed as separate args.

So @chalkmaroon both edits (adding /emoncms and removing .json) should happen in the same line ie what was originally

  //Build emoncms URL for sending data to
  String url = "/input/post.json?node=";

should now be

  //Build emoncms URL for sending data to
  String url = "/emoncms/input/post?node=";

and the IP left as just the IP.

Thanks Brian

chalkmaroon · 2019-05-10T13:24:00.069Z

The only potential difference I can think of is if you are logged in when you try the url. Can you therefore try the url again in an incognito browser session (or just log out of emoncms) . This will also test the apikey is correct.

This changes nothing.

pb66:

  //Build emoncms URL for sending data to
  String url = "/emoncms/input/post?node=";

and the IP left as just the IP.

I changed to
const char* host = "192.168.0.133";
and

String url = "/emoncms/input/post?node=";

Still get the same result, 3 inputs, but no values…

pb66:

What reply appears in the nodemcu serial?

14:13:05.181 -> rd
14:13:06.161 -> 
14:13:06.161 -> Connecting to RaspiRouter
14:13:06.672 -> ....
14:13:08.159 -> WiFi connected
14:13:10.634 -> Luftfeuchte: 31.94 %	 Temperatur: 24.64 *C %	 Luftdruck: 954.79

adding a print line:

  while (emonHTTPClient.available()) {
    String line = emonHTTPClient.readStringUntil('\r');
    Serial.print(line);
  }

Does nothing… The serial monitor output from above is already with this line…

borpin:

Then use tshark again to see what is actually posted.

tshark

pi@emonpi:~ $ sudo tshark -Y http.request -T fields -e http.request.full_uri
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eth0'
http://172.23.56.146/emoncms/device/list.json
http://172.23.56.146/emoncms/input/list.json
http://172.23.56.146/emoncms/device/list.json
http://172.23.56.146/emoncms/input/list.json
http://172.23.56.146/emoncms/device/list.json
http://172.23.56.146/emoncms/input/list.json
http://172.23.56.146/emoncms/device/list.json
http://172.23.56.146/emoncms/input/list.json
http://172.23.56.146/emoncms/device/list.json
http://172.23.56.146/emoncms/input/list.json
http://172.23.56.146/emoncms/device/list.json
http://172.23.56.146/emoncms/input/list.json
^C78 packets captured
pi@emonpi:~ $

I shortened it bit and don’t get confused with the different ip ! I’m switching between networks …

borpin:

BTW have you come across ESPHome? Very simply generic code for esp8266 based items.

Unless you want to do a bit of coding, it will probably be simpler to implement.

Thanks for the tip, I didn’t heard of it !

borpin · 2019-05-10T16:48:47.755Z

chalkmaroon:

I shortened it bit and don’t get confused with the different ip ! I’m switching between networks …

You need to run tshark on the machine that emoncms is installed on and you are looking for the URL that includes the input string.

chalkmaroon · 2019-05-10T16:57:59.401Z

This is from the emoncms machine (pi@emonpi)
These 2 lines are the only output and come every few seconds…

borpin · 2019-05-10T17:02:33.608Z

And how often does the ESP post the temperature data? Do you wait for it?

I see

http://192.168.7.xxx/emoncms/input/post?node=Kitchen&json={temperature:22.20,humidity:23.42,pressure:957.39}

as a tshark output.

Is the entry in your ‘host’ setting in the script the same as the IP your logged into when running tshark? If not, it will never see it.

[edit] I deleted the API Key data.
[edit2] I see these appearing as inputs on the emonPi.

chalkmaroon · 2019-05-10T18:00:07.573Z

I cannot believe it, now it works !!

I reflashed the emonpi image, in the firmware I replaced the api key and adjusted the firmware like you said:

pb66:

  //Build emoncms URL for sending data to
  String url = "/input/post.json?node=";

should now be

  //Build emoncms URL for sending data to
  String url = "/emoncms/input/post?node=";

and the IP left as just the IP.

Guys, you are awesome ! Thanks for the great help and patience !