EmonESP with emonTxV3.4 DiscreteSampling?

Hi Guys!
I just got my new EmonTx v3.4 package delivered today and I am setting it up to connect through EmonESP to my cms instance. I see that the standard emonTxV3_4_DiscreteSampling firmware now runs at 115200 baud and outputs the data in the format expected by the ESP firmware, great!

I’d like to have all the features of the standard firmware because I have a USA ACAC and a DS18B20 connected and also I feel like that’s where any new development will go, right? So the only problem is that the EmonESP firmware takes every line of serial data it gets and appends it to the GET url to emoncms.

So the question is: Shouldn’t the emonTx firmware prefix the lines of actual data (non-DEBUG info) with something so that the other side knows it is data and not simple logging info? Then the emonESP could use this to determine if serial data coming in is just logging or needs to be transmitted. Something like a NEMA GPS stanza maybe?
$EMTX,ct1:100,ct2:93,...*8C

The emonESP side can just look for things that begin with $EMTX, and then check the checksum at the end. If it all checks out, only then will it set gotData = true. This prefix and suffix will be stripped off before being saved. The code for the test API can stay the same because that comes directly from the user.

Thoughts?

What version emonTx firmware are you running? If you have version V2.5 onwards, the only data the emonTx V3 will print to serial will be data the emonESP can understand. The processing on the emonESP was intentionally kept to the minimum due to processor constraints.

Hey Glyn, thanks for responding.

The emonTx came with what appeared to be the same firmware from the compiled directory in git. I thought it had said v2.40 but I guess it was 2.30 (according to the git commit comment there). It was definitely Discrete Sampling RFM69CW though and it spit out the data in at 9600 baud and separated with spaces instead of “key:value,”

This is the firmware I compiled up (v2.60) and it does have the data portion in the right format but there’s a bunch of logging at startup that’s not data that the ESP tries to upload. Should I be on another branch?

That sounds correct, it outputs debug data at startup. After that, it should print only emonESP serial data.

Yeah it works a treat, it is just on startup the ESP sends all that log information to emoncms

47.198.192.198 - - [05/Jan/2017:08:07:14 -0500] "GET /emoncms/input/post.json?json={CT 1 Cal 90.90,psent:0,psuccess:0,freeram:28472}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 383 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:14 -0500] "GET /emoncms/input/post.json?json={CT 2 Cal 90.90,psent:1,psuccess:0,freeram:28120}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 383 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:14 -0500] "GET /emoncms/input/post.json?json={CT 3 Cal 90.90,psent:2,psuccess:0,freeram:28120}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 383 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:14 -0500] "GET /emoncms/input/post.json?json={CT 4 Cal 16.67,psent:3,psuccess:0,freeram:28120}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 383 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:15 -0500] "GET /emoncms/input/post.json?json={RMS Voltage on AC-AC  is: ~6V,psent:4,psuccess:0,freeram:28200}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 383 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:15 -0500] "GET /emoncms/input/post.json?json={AC-AC NOT detected - Apparent Pwr measure enabled,psent:5,psuccess:0,freeram:28048}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 383 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:15 -0500] "GET /emoncms/input/post.json?json={Assuming VRMS: 230V,psent:6,psuccess:0,freeram:28128}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 383 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:15 -0500] "GET /emoncms/input/post.json?json={Assuming power from batt / 5V USB - power save enabled,psent:7,psuccess:0,freeram:27912}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 383 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:16 -0500] "GET /emoncms/input/post.json?json={ct1:19109,ct2:0,ct3:0,ct4:0,vrms:491,pulse:0,t0:225,psent:8,psuccess:0,freeram:28056}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 160 "-" "ESP8266HTTPClient"
47.198.192.198 - - [05/Jan/2017:08:07:18 -0500] "GET /emoncms/input/list.json HTTP/1.1" 200 775 "http://capnbry.net/emoncms/input/view" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
47.198.192.198 - - [05/Jan/2017:08:07:26 -0500] "GET /emoncms/input/post.json?json={ct1:3767,ct2:0,ct3:0,ct4:0,vrms:498,pulse:0,t0:225,psent:9,psuccess:1,freeram:28032}&node=emonesp&apikey=a4919be232371523b604688954c6647e HTTP/1.1" 200 160 "-" "ESP8266HTTPClient"

What I was saying is, shouldn’t there be a simple prefix to actual data versus log spam that will be sent erroneously to emoncms? I don’t think it adds any complexity to the code to provide some validation that the serial data is actual data:

  else if (Serial.available()) {
    data = Serial.readStringUntil('\n');
    // Check for string integrity
    if (data.beginsWith("$EMTX,") {
      data.remove(0, 6);
      gotData = true;
    }
  }

Ok, good point. I can see how it would be useful to have extra debug info that emonESP ignored.

The emonESP code is in its infancy, I’ll look at merging this in during the next update. An even simpler solution could be for the emonESP to ignore any serial lines that begin with a non-numerical character.

UPDATE: forget that, it wouldn’t work since emonESP named serial key:value need to begin with a non-numerical character. Your solution is best :slight_smile:

Before setting a standard can you consider another option?

firstly setting a marker for packets to be ignored is potentially prone to error, if a packet slips through without a prefix it is assumed to be valuable well formatted data, what about coding errors and or corrupt packets etc, surely the thing to do is to prefix the known good packets as “good to use” much the same way JeeLib uses the “OK” prefix.

This leaves options for the future where errors, log messages and/or interactions between the softwares can be catagorised and used for various things rather than having just “not for sending” and sending what ever’s left, we should have “ok to send” and not send anything else, that “anything else” can in time be further catagorised without any backwards compatibility issues.

In a similar way I have a working PoC for serially or USB attached “Arduino” based devices that can pass log messages out to emonhub in among the data packets, currently I use L1 - L5 prefixes and emonhub just uses the prefix to translate into DEBUG, INFO, WARNING, ERROR or CRITICAL and passes them out to it’s own logfile. This avoids start up messages interfering with data streams and also gives a route out for error and debugging messages.

I intend to look at making a short list of log messages that reuse a handful of keywords to minimize the impact on the sketch file size and memory etc, but even numeric fault codes would be a huge step forward in getting log messages out of an emonTx or emonPi etc.

EDIT - re-reading Bryan’s post I think he is suggesting the same thing, prefixing good data not the bad!

The emonESP code may be in its infancy but it is great! I made the changes really quickly on my side and this is all the changes that are needed. The code above has “first thing in the morning, write code without a compiler” errors :smiley: The differences is the member is called “startsWith” and I missed a closing paren

  // If data received on serial
  else if (Serial.available()) {
    // Could check for string integrity here
    data = Serial.readStringUntil('\n');
    // Check for string integrity
    if (data.startsWith("$EMTX,")) {
      data.remove(0, 6);
      gotData = true;
    }
  }

On the emonTx side, the only difference is to change the first print of the data to include the prefix

  if (DEBUG==1) {
    Serial.print("$EMTX,ct1:"); Serial.print(emontx.power1);
    Serial.print(",ct2:"); Serial.print(emontx.power2);

I have been using emoncms for what seems like ever now. I can’t tell you how much money I’ve spent on different power monitoring devices. I’ve got a dozen zigbee outlet monitors (Alertme Smart Outlets) that I reverse engineered the protocol and have pumping in. I had a Black & Decker EM100B pulse-reading power monitor on the meter, which I reverse engineered the protocol and built hardware to receive the signals (superregenerative / superheterodyne receiver or RFM69 in variable floor OOK).

I just got solar panels so the pulse reader no longer is accurate and when I looked and saw that not only was there a way to give the emonTx wifi capabilities, that it also was low cost and had a slick little web interface I was sold. I am excited about installing my first official hardware to hook to emoncms. Thanks to all the Open Energy Monitor developers, both on the hardware and software side!