Looking forward to it, especially as I was thinking those were things I might add 
1 Like
Here’s a sneak peak of what’s in the MQTT branch:
It doesn’t yet post to separate MQTT topics, it currently posts the same string that gets’s sent to emoncms to an MQTT topic e.g. in this example CT1:3935,CT2:325,T1:12.5,T2:16.9,T3:11.2,T4:34.7. This is not very useful to subscribe to. I intend to get the ESP to phase the data and split it up into sperate topics e.g. emon/emonesp/ct1 > 3935 emon/emonesp/ct2 > 325 etc.
I would also like to add the ability to perform an avrdude upload over Wifi to a ATmega328 connected to the ESP’s UART. This will allow us to remotely upgrade an emonTx firmware from a RaspberryPi emonBase that’s on the same network. This should be possible with ESP8266AVRISP library that’s included in the Arduino ESP framework.
Update: it looks like serial upload from esp to avr is unpratical at the moment. Avr ISP upload looks like the best way to achieve pass through avr firmware update. This will require additional hardware connections. We will look at adding an esp on-board future emonTx (v4?) hardware update.
Would you like to help with either of these tasks? What experience do you have with ESP? I’m learning as I go along, this is my first ESP project.
1 Like
I can certainly help out I have a reasonable amount of experience with Microcontrolers in general and only a bit of ESP8266 experience, but I can Google for England
Unfortunately I am short on time 
but will do what I can. First thing I need to do is get some debug so I can work out why it is failing after some time. One of the examples was a Serial to Telnet gateway was thinking about adding something like that. Looks like this is best to do based on the mqtt branch.
Oh dear, sorry to hear it’s failing for you. All the test setups we have running (about 5 units) have been totally solid for months. Could it be a hardware issue? Good to get some more data points on this. Stable reliable logging is THE most important feature!
Debug so far has been done via the /status web page e.g. http://192.168.0.75/status and test data can be sent via a test API e.g.
http://192.168.0.75/test?serial=CT1:3935,CT2:325,T1:12.5,T2:16.9,T3:11.2,T4:34.7
Serial to Telnet gateway sounds interesting. Would not have a problem with this as long as it is quite lightweight and didn’t pose a security risk. As you know resources on the ESP are limited
Been thinking a bit more about getting some debug out of the EmonESP over the weekend, how do you feel about moving over to using websockets for retrieving the status instead of polling the EmonESP? As a WebSocket maintains a persistent connection to the EmonESP data can be pushed to the browser as it comes in.
This could then also be used to transport debug messages to the browser, ie replace all the Serial.print… with a call to send the data over the WebSocket.
The two methods are not mutually exclusive (apart from the debug unless you start messing around with buffering), you could try connecting via a WebSocket and if that fails revert to polling.
Interesting sounds like web sockets could be a better way to pass status messages from the esp to the web browser.
I would like to keep ‘status’ and ‘serial debug’ as being two sperate things. Since ‘status’ can be viewed from the web browser and potentially exposed to the internet while serial debug is definitely not going to be exposed to the internet. There is a potential security risk to exposing debug messages to the internet via web sockets?
Do you know how the memory and processing overhead on the esp for web sockets will compare to serial print and polling? As you know resources are quite tight on the ESP.
Thanks for the PR to extend the amount of EEPROM for MQTT password, I will get this merged together with my latest changes that allow the user to enter SSL fingerprint to enable SSL. This was previously hard coaded for emoncms.org but it has become apparent that the fingerprint will change every 3 months when the SSL certificate is renewed. It will be best to allow the user to enter the fingerprint if they want to use HTTPS, this does not limit them to using emoncms.org with ssl.
Is it somehow specific to the EmonTX HW? Is maybe a method used by the ESP-link more appropriate? Serial looks the simplest but something might be hindering its use here…
Yes, I’ve looked at ESP-Link, it’s a very nice bit of software development. However, reading the comments on users on ESP-Link it seems serial upload ESP > AVR is not particularly reliable. For remote flashing, reliability is important. I have not ruled it out. I would love for ESP > AVR serial upload to work. Currently, the ESP > AVR ISP upload method seems much more developed and reliable: Arduino/libraries/ESP8266AVRISP at master · esp8266/Arduino · GitHub
I see, thanks for the heads-up, good to know about the reliability aspect.
IMHO I would add both methods (maybe protected by build options). The serial option has a lot of benefits for the EmonESP/EmonTx setup as you don’t have to mess around opening the case. I think if the firmware is uploaded to the ESP first and then the complete programming process is managed via the ESP it should be reasonable reliable. I can see why the two options that use that use avrdude on the PC and pass the serial commands to the over the net could be unreliable.
Also bear in mind that the AVR probably does not have the space to have two copies of the firmware so there is always a risk of breaking the EmonTx no matter what update method you choose.
As an aside I have found a solution to getting debug out of the ESP, there are two UARTs on the ESP, pin #2 is the Tx of Serial1 so switching the trace to that serial port and connecting it to a second ESP running the WiFiTelnetToSerial example gives me remote debug. The added bonus of this method is that it works no matter what state the EmonESP is in, and given my issue looks to be related to the EmonESP loosing network connection this is a good thing.
Using this and enabling some of the ESP library debug, I think I have found the issue. It looks to be down to the HTTP server only being able to handle 1 client connection and as I had the EmonESP web page open this can cause it to not be able to fill up the receive buffers. After closing the web page on my PC it runs fine with no lock ups, (well kind of but I will get onto that). I had a chick look around and there are a few alternative web servers (eg GitHub - me-no-dev/ESPAsyncWebServer: Async Web Server for ESP8266 and ESP32, which also happens to support WebSockets). @glyn.hudson any objections to using an alternative web server?
I am also having issues with the latest mqtt branch with posts to emoncms.org not showing up. The EmonESP is getting back the HTTP/1.1 200 OK but nothing shows up in my inputs I am still investigating this. It is kind of working with Adafruit IO (via MQTT) however (with a patch I will send shortly) IO - Adafruit for some reason only MSG, P0, P1 and freemem are showing up, but that is a start.
I was wondering about this after Glyn’s post. A store-and-forward method should be as reliable as connecting a PC programmer directly. I’m yet to try that but I have my NodeMCU’s finally to test this (for another project) among other things.
As long as the ESP is accessible, there shouldn’t be a problem. It can reset and put it to programming mode any time to correct a completely broken AVR firmware.
Nice work debugging, I have not noticed any issue with the current web server, I have just tested opening the web page on my computer and mobile web browser at the same time and there seems to be no issue. Could this be a differance with running on an ESP-12 VS ESP-01? I would not be adverse to trying a different web server although if possible I would like to stick to using libraries build into the Arduino ESP or at least popular and well tested libraries.
I think this could be due to Emoncms.org only supporting numerical node ID’s. Try using an number instead of a name in the ‘node’ box. I have added a note on the web page in latest release. The latest version of emoncms running on EmonPi / emonBase does support passing a node name, I will try and poke @TrystanLea to update emoncms.org when he gets back from holiday 
https://github.com/openenergymonitor/EmonESP/blob/mqtt/src/data/home.html#L104
Try the latest version. Yesterday I added proper MQTT support i.e spitting out the MQTT data into sub topics. Ive started documenting MQTT & EmonESP API’s in the readme:
https://github.com/openenergymonitor/EmonESP/tree/mqtt#mqtt
I’m getting close to merging MQTT branch into master. The final thing to add is the ability to change the web page username and password via the web interface.
I moved over to an ESP-12 (Huzzah) as the OTA is kind of handy for dev. The failure I was seeing was it was working for maybe for a few mins then would stop for a while, see Emoncms - user login. so to reproduce you would just need to leave the EmonESP page open for a while.
I was using node 0 which was the ID used by the master branch and it used to work till I updated last night. I need to do some investigation, probably will try and capture the network traffic. Oh wait damn you HTTPS
Yeah, that is the version I am using, I had to make a change as Adafruit IO only supports a flat structure, PR to follow.
The fingerprint for emoncms.org has recently changed you can leave the fingerprint field blank if you want to post over HTTP instead it’s a shame the esp does not support full https
emoncms.org just tries to redirect to the HTTPS site I have a suspicion it may be related to URL length, but it is late so I will do some more debug tomorrow.
Update on the Adafruit IO nor reading all the feeds, this is a limit on Adafruits side, you can only have 10 feeds and I already had a few setup.
1 Like
Posting to emoncms.org via HTTP is definitely still possible, the majority of API traffic still comes from HTTP. Most users have not yet updated to use HTTPS. Only the web interface should re-direct to HTTPS.
Resolved this issue, both the errors I was seeing were related to the same thing. It turned out that there was an extra white-space char coming from the serial somewhere and that was messing up the request, for some reason when using HTTPS a 200 OK was returned but the data was not added and when using HTTP a 302 ‘error’ was returned, but I changed the code that reduced the size by 1 to use trim() instead and problem solved.
1 Like
Nice work, error checking on API into to Emoncms could definitely be improved…
@glyn.hudson how do you feel about splitting the firmware up into a few C files? I would propose;
- src.ino - normal setup and loop and anything else not in the below
- mqtt.cpp/h - MQTT stuff
- emoncms.cpp/h - code to push to EmonCMS
- webserver.cpp/h - Web server code
- eeprom.cpp/h - Code to read/write ‘EEPROM’ (although maybe just saving a JSON file in SPIFFS may be a better solution for config)
- ota.cpp/h - OTA update code (both emonESP and emonTx)
1 Like
Yes, that would be great. Was thinking that myself. src.ino is getting a bit large!
Would multiple .ino file have the same effect? I think platformio and Arduino both support multiple src files