Arduino Yun with TX shield updates only once each time it boots

Hello, I have a Yun+emonTx shield that had been monitoring some heat pumps for over a year and continued after I installed a home server and aimed it at that instead of emoncms.org. After getting it a nifty new case (and possibly updating libraries or Arduino board info) and reinstalling it, after each reboot it would only update the feeds once. It continues running the sketch, writing to serial, and flashing the LED at each update, but without any further activity visible on the server.

If I were very organized I would have made notes of any changes or code ā€œcleanupā€ I may have done when this started happening but I didnā€™t, and then I set it aside while I worked on other things so my memory is extra fuzzy. I just ran diff on an older version of my sketch that used emoncms.org and this one and didnā€™t notice anything more than trivial differences in comments and expected changes like the api key.

I wondered if the way the data and api strings were assembled was it since that could be different the first vs subsequent times, but the code looks ok to me and if I copy the string from the serial monitor and paste it into a browser, it is accepted as a new update.

The Yun stays online during this process, visible on the router and accessible through its linux interface.

Any ideas about what could be wrong?

Hereā€™s my sketch:

// based on http://forum.arduino.cc/index.php?topic=201925.0
#include "EmonLib.h" //for energy monitoring
#include <Bridge.h> //for Yun linux to Arduino connection
#include <OneWire.h> //for thermometer
#include <DallasTemperature.h> //for thermometer
#include <FileIO.h> //for writing to SD card
//#include <avr/wdt.h> // My energy monitor goes down every couple of days or after several months, so I added a watchdog timer. 
//At first try of the WDT, I could only get one data point so I diasbled it until I can troubleshoot.

#define DEBUG 1
#ifdef DEBUG
  #define DEBUG_PRINTLN(x)  Serial.println(x)
#else..
  #define DEBUG_PRINTLN(x)
#endif

// Create  instances for each CT channel
EnergyMonitor ct1; //ct1 is HPWH for 71
EnergyMonitor ct2; //ct2 is HPWH for 73
EnergyMonitor ct3; //ct3 is ASHP for 71
EnergyMonitor ct4; //ct4 is ASHP for 73

// On-board emonTx LED
const int LEDpin = 9;

//Setup thermometer
#define ONE_WIRE_BUS 4
OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature sensors(&oneWire);

const unsigned long postingInterval = 60000;
unsigned long lastRequest = 0;
String Timestamp = "";
String dataString =  "";
String APIKEY = "api key here";

void setup() {

  DEBUG_PRINTLN("Finished initialization, entering setup");

  // Initialize Bridge
  Bridge.begin();

  // Initialize Console
  Console.begin();

  // Start serial port
  Serial.begin(9600);

  // Start file system for SD card logging
  FileSystem.begin();

  // Start up the thermometer library
  sensors.begin();

  // 1st arg: pin number
  // 2nd arg: Calibration factor = CT ratio / burden resistance = (100A / 0.05A) / 33 Ohms = 60.606
  ct1.current(1, 20);
  ct2.current(2, 20);
  ct3.current(3, 25);
  ct4.current(4, 25);

  // (ADC input, calibration, phase_shift)
  ct1.voltage(0, 130, 1.7);
  ct2.voltage(0, 130, 1.7);
  ct3.voltage(0, 130, 1.7);
  ct4.voltage(0, 130, 1.7);

  // Setup indicator LED
  pinMode(LEDpin, OUTPUT);

  DEBUG_PRINTLN("Finished setup, entering delay to wait for wifi");

  // Hoping a delay allows the wifi connection to establish for more reliable operation
  delay(15000);
  
  //wdt_enable(WDTO_8S); // Setup watchdog timer, after the delay 
}

void loop() {

  DEBUG_PRINTLN("Entering loop");

  long now = millis();
  
  // call sensors.requestTemperatures() to issue a global temperature
  // request to all devices on the bus
  sensors.requestTemperatures(); // Send the command to get temperatures

  // Calculate all. No.of crossings, //time-out
  ct1.calcVI(20, 2000);
  ct2.calcVI(20, 2000);
  ct3.calcVI(20, 2000);
  ct4.calcVI(20, 2000);

  if (now - lastRequest >= postingInterval) {
   // wdt_reset(); //once here in the main loop 'if' and once at the start of each of these routines
    updateData();
    sendData();
    writeToCard();
    lastRequest = now;
  }

  DEBUG_PRINTLN("Finished loop, about to delay");

  //broke thirty second delay into 5-second chunks to keep watchdog happy. Could also disable it for this, but someone online
  //specifically suggested someone use the approach below instead of disabling it
 /* wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();*/
  delay(10000);
}

void updateData() {
  //wdt_reset();
  // convert the readings to a String to send it:
  digitalWrite(LEDpin, HIGH); //turn emon shield red LED on
//Timestamp moved to the write to card portion only because that has no other timestamp and the Emon server doesn't like the timestamp format
  dataString = "Temperature:"; //need to add a comma in front, and switch to += if I turn the timestamp back on
  dataString += sensors.getTempFByIndex(0);
  dataString += ",ct1.apparentPower:";
  dataString += 2*ct1.apparentPower;//these have 2x because the CTs are around 240v circuits while the voltage sensor is plugged into 120v
  dataString += ",ct2.apparentPower:";
  dataString += 2*ct2.apparentPower;
  dataString += ",ct3.apparentPower:";
  dataString += 2*ct3.apparentPower;
  dataString += ",ct4.apparentPower:";
  dataString += 2*ct4.apparentPower;
  digitalWrite(LEDpin, LOW); //turn emon shield red LED off
  DEBUG_PRINTLN("updateData: ");
  DEBUG_PRINTLN(dataString);
  DEBUG_PRINTLN("Finished updateData");
  
}

// this method makes a HTTP connection to the server:
void sendData() {
  //wdt_reset();
  // form the string for the API header parameter:
  String apiString = "apikey=";
  apiString += APIKEY;

  // form the string for the URL parameter:
  // specify the server address
  String url = "http://192.168.1.116/emoncms/input/post?";
  url += "node=Yun";
  url += "&csv=";
  url += dataString;
  url += "&";
  url += apiString;

  DEBUG_PRINTLN("sendData dataString: ");
  DEBUG_PRINTLN(dataString);
  DEBUG_PRINTLN("sendData url: ");
  DEBUG_PRINTLN(url);
    
  // Is better to declare the Process here, so when the
  // sendData function finishes the resources are immediately
  // released. Declaring it global works too, BTW.
  Process emoncms;
  Console.print("\n\nSending data to EmonCMS... ");
  emoncms.begin("curl");
  //emoncms.addParameter("-g"); //was commented in my old sketch, tried in this one when I was only getting one data post per boot
  emoncms.addParameter(url);
  emoncms.run();
  Console.println("upload to EmonCMS done!");
  DEBUG_PRINTLN("upload to EmonCMS done!");

  // If there's incoming data from the net connection,
  // send it out the Console:
  while (emoncms.available()>0) {
    char c = emoncms.read();
    Console.write(c);
    DEBUG_PRINTLN(c);
  }
}

void writeToCard() {
  //wdt_reset();
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  // The FileSystem card is mounted at the following "/mnt/FileSystema1"
  File dataFile = FileSystem.open("/mnt/sd/arduino/emon/datalog.csv", FILE_APPEND);

  Timestamp = getTimeStamp(); 
  dataString += ",Timestamp:";
  dataString += Timestamp;

  // if the file is available, write to it:
  if (dataFile) {
    dataFile.println(dataString);
    dataFile.close();
  }
  
  // if the file isn't open, pop up an error:
  else {
    Console.println("error opening datalog.csv");
    DEBUG_PRINTLN("error opening datalog.csv");
  }

  DEBUG_PRINTLN("Finished writeToCard");

}

// This function return a string with the time stamp
// from the bottom of this Yun specific tutorial https://www.arduino.cc/en/Tutorial/YunDatalogger
String getTimeStamp() {
  //wdt_reset();
  String result;
  Process time;
  // date is a command line utility to get the date and the time
  // in different formats depending on the additional parameter
  time.begin("date");
  time.addParameter("+%D-%T");  
  // parameters: D for the complete date mm/dd/yy
  // T for the time hh:mm:ss
  time.run();  // run the command

  // read the output of the command
  while (time.available() > 0) {
    char c = time.read();
    if (c != '\n')
      result += c;
  }

  return result;
  
  DEBUG_PRINTLN("Got TimeStamp");
  
}

So whatā€™s different about the way the browser sends the data?

Knowing nothing about your Yun, does Process emoncms require shutting down in any particular way before it dies as it goes out of scope?

Search the forum for ā€œAuthorization: Bearerā€. I had an issue with something similar and even added some notes to the Input API help page on your installation (top right of Inputs page).

Probably because the ā€˜browserā€™ is already authenticated.

Thanks Brian,

On the Feed API Help page, it looks like any of those three ways to provide the apikey would apply in a given situation and I was already using middle one: appending the post parameter (and using the address ā€¦/emoncms/input/post/ā€¦). I tried switching to the -H one you mention, but couldnā€™t get it to work. I think that the Arduinoā€™s attempt may not work because I might be mixing methods now (I know these words are involved, donā€™t know anything about them: curl, GET, post, JSON, etc.) If I log the browser out of the server, then paste the string, the server responds: ā€œUsername or password emptyā€ Whereas, if I go back to the string I had been using, and use that in a browser logged out, the server says ā€œokā€. It may be that I didnā€™t set up the -H parameter correctly, but it also seems like the POST parameter should work for the Arduino more than once.

Thereā€™s both the question of whatā€™s different between the browserā€™s input and the Arduinoā€™s, but maybe closer to the problem: whatā€™s different between the first post the Arduino makes when it reboots and every one after that? When I test the browser, I use a newer string, not the first one.

So then, to Robertā€™s postā€¦Iā€™ve been treating the emoncms commands as complete back boxes and donā€™t know if they have changed since I set this up years ago, or if thereā€™s ways to change how they are used that might solve this. I mean short of asking the Arduino to reboot after each post.

I have no idea how the sketch works or what the Process command does, but the error you are seeing is because what you are doing is not authenticating correctly.

If the string generated is a curl command, if you have another linux machine on your network, you could try the curl command directly from that.

Can you post the actual url ā€˜commandā€™ used (with munged API key)?

Just to be clear, you are using the Input API (note ā€˜inputā€™ in the URI) and not the Feed API.

Hereā€™s the string, copied from the serial monitor after the Arduino wakes up, this one added a new point to the server:

http://192.168.1.116/emoncms/input/post?node=Yun&csv=Temperature:69.24,ct1.apparentPower:0.00,ct2.apparentPower:2.09,ct3.apparentPower:7.43,ct4.apparentPower:49.60&apikey=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

And hereā€™s one a couple of minutes later, that did not generate a new data point on the server:

http://192.168.1.116/emoncms/input/post?node=Yun&csv=Temperature:69.35,ct1.apparentPower:0.00,ct2.apparentPower:0.00,ct3.apparentPower:0.00,ct4.apparentPower:0.07&apikey=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

In the serial monitor, I confirmed the APIKEY is the same in both strings.

It doesnā€™t seem like the emoncms process should be doing anything differently when it first starts up vs the next update, since itā€™s called within sendData() in the loop() so I would think it is called fresh each time, but maybe it does need to be shut down? I hadnā€™t seen that in any documentation. Granted I really got my head in this when I started in 2016 or earlier, but have only skimmed since then. Seems like a number of things have changed and thereā€™s abandoned methods and archived docs.

On the Input API page, it still looks like Iā€™m using one of the methods, the third one in this case, my appending the APIKEY to the end of the url. It says a POST parameter is recommended instead. I played around adding an emoncms.addParameter() before the emoncms.run(), but wasnā€™t sure about the format. I tried to duplicate what it shows on the Input API page including the quotes, since the POST parameter option has quotes and the others donā€™t. In that case, the serial monitor was showing the serverā€™s reply about no username and password. In the normal failure modes, that does not appear. I tried switching emoncms.begin(ā€œcurlā€) to emoncms.begin(ā€œGETā€) and that failed to post or to get an error message from the server. But even in that case, if I took the url from the serial monitor, it still worked from the browser, though it correctly rejected an incorrect APIKEY when I accidentally pasted the following line along with the url. I did learn through this part that ā€œemoncmsā€ as a process is just the arbitrary name that the person I copied the code from used. I had previously thought it was part of emonlib and hadnā€™t paid it any attention.

It is odd that it works once.

I forgot to ask what version of EmonCMS you are running.

Looking at the Input API, the data structure you are using is a mix of JSON and CSV although it does seem to work when I tested it.

All I can suggest is using tshark (search here for some examples) to see what is actually being sent on the network.

Other than that Iā€™m out of ideas - sorry. These sketches are not really my area :slight_smile:

Thanks Brian, Iā€™m using 10.1.10. I realize I need to change my approach from looking for a simple fix to some breakage, to reconsidering how the sketch sends data. I had thought it was a part of the emoncms system, like another project I have that handles the communication with the server, but I see in the examples that the sending it completely different depending on the type of transmission.

I think I was initially using JSON, but switched to CSV when I had a previous problem. This is another reason to sort of start over.

Hello hereā€™s an updated of sorts. Iā€™ve tried a few Yun specific communication libraries, HTTPClient and BridgeHTTPClient without luck. I went back to my oldest version of the sketch, when it worked for months at a time, and that brought me back to the Process/curl method in my original post, but now I knew it was a Yun-specific process. Unfortunately I didnā€™t find much documentation. But I did get the serial monitor to write out the command below. Except for the ā€œcurlā€ which is not in the printout, itā€™s within the .begin().

curl --data "node=Yun&data={"Temperature":68.56,"ct1.apparentPower":0.00,"ct2.apparentPower":0.00,"ct3.apparentPower":33.09,"ct4.apparentPower":19.47}&apikey=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn" "http://192.168.1.116/input/post"

If I paste the above into my browser it works. If I remove the apikey, the server rejects it whether or not another tab in the browser is logged in. If I paste it into a ssh session in the Linux side of the Yun, it works.

An Arduino forum post suggested this function can struggle when the string is long and that user said they try to keep their strings less than 10 characters so that could be it. But why did it work in the past? And then why once per boot? I guess the latter is easier to understand if itā€™s a memory problem. Two other people suggested a different Yun Bridge/Process command: runShellCommand(). If nothing else, itā€™s one line instead of three. Once I set that up, which is also a little more clear because ā€œcurlā€ goes in with the rest of the string all at once, I was back to where I started: one successful post of data, then the Yun keeps running, but the server doesnā€™t see any new data, nor do its responses appear in the serial monitor. And still, if I paste one of the later unsuccessful commands from the serial monitor into the browser, it works.

Looking at the stats from the Linux side of the Yun, memory stays flat with 28% free and the short term load average moves between 0.45 and 0.25 so the overalll system isnā€™t resource constrained, but maybe the Arduino side is? At the end of compiling, the Arduino IDE says the sketch takes up 68% of program storage space and global variables use 55% of dynamic memory, leaving 1150 bytes for local variables. Is it possible that is exhausted between the first and second data updates? But if that were it, why would it have worked in the past? Flailing around in the dark now (as if I wasnā€™t before), I set the three strings to be blank at the start of the loop in case any cruft from a previous run was holding up memory before the program reached the point where it resets them for the current run. But no noticeable difference.

Going back to the original curl process and thinking about runShellCommand() in my last post got me very close. Today I tried again and compared the string to the Input API help page and noticed that unlike the fulljson version of the GET command, the JSON POST does not use quotes around the variables. I changed that (and maybe made another change I canā€™t remember) and it works! Iā€™m happy it works, but that still makes no sense: why would it work once with unneeded quotes then not after that? Anyway, this is good to have it sorted out now when another heating season is here and when I want to duplicate this device to monitor three new loads. Hereā€™s the working sketch for anyone (including me!) who is searching with a related problem in the future:

// based on http://forum.arduino.cc/index.php?topic=201925.0
#include <EmonLib.h> //for energy monitoring
#include <Process.h> //for Yun
#include <Bridge.h> //for Yun linux to Arduino connection
#include <OneWire.h> //for thermometer
#include <DallasTemperature.h> //for thermometer
#include <FileIO.h> //for writing to SD card
#include <avr/wdt.h>

#define DEBUG 1
#ifdef DEBUG
  #define DEBUG_PRINTLN(x)  Serial.println(x)
#else
  #define DEBUG_PRINTLN(x)
#endif

// Create  instances for each CT channel
EnergyMonitor ct1; //ct1 is HPWH for 71
EnergyMonitor ct2; //ct2 is HPWH for 73
EnergyMonitor ct3; //ct3 is ASHP for 71
EnergyMonitor ct4; //ct4 is ASHP for 73

// On-board emonTx LED
const int LEDpin = 9;

//Setup thermometer
#define ONE_WIRE_BUS 4
OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature sensors(&oneWire);

const unsigned long postingInterval = 60000; //one minute
unsigned long lastRequest = 0;
String Timestamp = "";
String dataString =  "";
String APIKEY = "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn";

void setup() {

  DEBUG_PRINTLN("Finished initialization, entering setup");

  // Initialize Bridge
  Bridge.begin();

  // Start serial port
  Serial.begin(9600);

  // Start file system for SD card logging
  FileSystem.begin();

  // Start up the thermometer library
  sensors.begin();

  // 1st arg: pin number
  // 2nd arg: Calibration factor = CT ratio / burden resistance = (100A / 0.05A) / 33 Ohms = 60.606
  ct1.current(1, 20);
  ct2.current(2, 20);
  ct3.current(3, 25);
  ct4.current(4, 25);

  // (ADC input, calibration, phase_shift)
  ct1.voltage(0, 130, 1.7);
  ct2.voltage(0, 130, 1.7);
  ct3.voltage(0, 130, 1.7);
  ct4.voltage(0, 130, 1.7);

  // Setup indicator LED
  pinMode(LEDpin, OUTPUT);

  DEBUG_PRINTLN("Finished setup, entering delay to wait for wifi");

  // Hoping a delay allows the wifi connection to establish for more reliable operation
  delay(15000);
  
  wdt_enable(WDTO_8S); // Setup watchdog timer, after the delay 
}

void loop() {

  DEBUG_PRINTLN("Entering loop");

  long now = millis();
 
  // call sensors.requestTemperatures() to issue a global temperature
  // request to all devices on the bus
  sensors.requestTemperatures(); // Send the command to get temperatures

  // Calculate all. No.of crossings, //time-out
  ct1.calcVI(20, 2000);
  ct2.calcVI(20, 2000);
  ct3.calcVI(20, 2000);
  ct4.calcVI(20, 2000);

  if (now - lastRequest >= postingInterval) {
    wdt_reset(); //once here in the main loop 'if' and once at the start of each of these routines
    updateData();
    sendData();
    writeToCard();
    lastRequest = now;
  }

  DEBUG_PRINTLN("Finished loop, about to delay");

  //broke thirty second delay into 5-second chunks to keep watchdog happy. Could also disable it for this, but someone online
  //specifically suggested someone use the approach below instead of disabling it
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  //delay(30000);
}

void updateData() {
  wdt_reset();
  // convert the readings to a String to send it:
  digitalWrite(LEDpin, HIGH); //turn emon shield red LED on
//Timestamp moved to the write to card portion only because that has no other timestamp and the Emon server doesn't like the timestamp format
  dataString = "{Temperature:"; //need to add a comma in front, and switch to += if I turn the timestamp back on
  dataString += sensors.getTempFByIndex(0);
  dataString += ",ct1.apparentPower:";
  dataString += 2*ct1.apparentPower;//these have 2x because the CTs are around 240v circuits while the voltage sensor is plugged into 120v
  dataString += ",ct2.apparentPower:";
  dataString += 2*ct2.apparentPower;
  dataString += ",ct3.apparentPower:";
  dataString += 2*ct3.apparentPower;
  dataString += ",ct4.apparentPower:";
  dataString += 2*ct4.apparentPower;
  digitalWrite(LEDpin, LOW); //turn emon shield red LED off
  DEBUG_PRINTLN("updateData: ");
  DEBUG_PRINTLN(dataString);
  DEBUG_PRINTLN("Finished updateData");
  
}

// this method makes a HTTP connection to the server:
void sendData() {
  wdt_reset();
  // form the string for the API header parameter:

  //OP
  String apiString = "apikey=";
  apiString += APIKEY;

  // form the string for the URL parameter:
  // specify the server address
  String url = "curl --data \"node=Yun&data=";
  url += dataString;
  url += "}&";
  url += apiString;
  url += "\" \"http://192.168.1.116/input/post\"";

  Process p;
  Serial.print("\n\nSending data... ");
  p.runShellCommand(url);
  DEBUG_PRINTLN("Wrote to EmonCMS ");
  DEBUG_PRINTLN(url);

  while (p.running());

  // If there's incoming data from the net connection,
  // send it out the Console:
  while (p.available()>0) {
    char c = p.read();
    Console.write(c);
    DEBUG_PRINTLN(c);
  }
}

void writeToCard() {
  wdt_reset();
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  // The FileSystem card is mounted at the following "/mnt/FileSystema1"
  File dataFile = FileSystem.open("/mnt/sd/arduino/emon/datalog.csv", FILE_APPEND);

  Timestamp = getTimeStamp(); 
  dataString += ",Timestamp:";  dataString += "}";
  dataString += Timestamp;

  // if the file is available, write to it:
  if (dataFile) {
    dataFile.println(dataString);
    dataFile.close();
  }
  
  // if the file isn't open, pop up an error:
  else {
    Console.println("error opening datalog.csv");
    DEBUG_PRINTLN("error opening datalog.csv");
  }

  DEBUG_PRINTLN("Finished writeToCard");

}

// This function return a string with the time stamp
// from the bottom of this Yun specific tutorial https://www.arduino.cc/en/Tutorial/YunDatalogger
String getTimeStamp() {
  wdt_reset();
  String result;
  Process time;
  // date is a command line utility to get the date and the time
  // in different formats depending on the additional parameter
  time.begin("date");
  time.addParameter("+%D-%T");  
  // parameters: D for the complete date mm/dd/yy
  // T for the time hh:mm:ss
  time.run();  // run the command

  // read the output of the command
  while (time.available() > 0) {
    char c = time.read();
    if (c != '\n')
      result += c;
  }
  return result;
  DEBUG_PRINTLN("Got TimeStamp");
}

So this was for legacy reasons. (note I did not test this for fulljson and curl - actually I am not surprised it doesnā€™t work and the API help should say JSON like as it is not a valid JSON object).

In a valid JSON object, the ā€˜nameā€™ must be double quoted. A ā€˜versionā€™ of JSON was implemented quite some time ago in EmonCMS that did not use quotes.

For backward compatibility, in order to provide the ability to send valid JSON to Emoncms, a new API method was created (by me :grin:) called ā€˜fulljsonā€™ that accepted valid JSON. This was only tested for the HTTP API and not curl.

If the object is a valid JSON string (check it using jsonlint.com) then use ā€˜fulljsonā€™ if it isnā€™t, use the ā€˜jsonā€™ call.

I thin you should be able to change the

--data "node=1&data={power1:100....

to

--data "node=1&fulljson={"power1":100....

and it might work.

Be useful if you could try it :grinning:

Cheers

When I said ā€œwork onceā€ I didnā€™t mean I think itā€™s weird it worked years ago and not now, software evolves, I was referring to the topic of this thread: working once per boot. Itā€™s the same version of server software reacting to what the Arduino sends, and I wouldnā€™t think the quotes would make the difference since it works then doesnā€™t work with the same string within minutes.

If using fulljson is not a mixing of vintages or methods, and is more proper and the way future EmonCMS will go, Iā€™m happy to use it, but it worked with ā€œdata=ā€ and unquoted variable names. I did test quoting the variable names, changing ā€œdataā€ to ā€œfulljsonā€, confirming the string is valid JSON (thanks for suggesting that step and the link), then my server says the string is not valid JSON and would not take any posts. Iā€™m only testing the part between and including the curly brackets as valid JSON, is the server testing a longer part? I changed it back, and Iā€™m back to the one update per minute, fixed system that I arrived at this morning. So my issue seems to be fixed, but is there are problem with EmonCMS parsing?

Ok thanks - if you are getting that, the API is reading the ā€˜fulljsonā€™ as the type of input.

but is not able to decode it which is odd if you have checked the string is a valid JSON object.

Seems there might be or it might be some misplaced quotes. I wonder if you might need to URL encode the string? Iā€™ve had issues like this before but cannot remember the whys and wherefores of URL encoding. You could edit that line in emoncms to dump the $datain object to see what string Emoncms actually got.

The more I look at it and think about it, I am sure the issue is the quotes required in the JSON object.

This may help (if you are interested) http - How do I POST a buffer of JSON using libcurl? - Stack Overflow

This is what I see in the serial monitor:

curl --data "node=Seeed&fulljson={"Temperature":-196.60,"ct1.apparentPower":2.27,"ct2.apparentPower":2.23,"ct3.apparentPower":3.25,"ct4.apparentPower":2.94}&apikey=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn" "http://192.168.1.116/input/post"

I can paste that entire thing, as is (with an apikey), into two linux devices and both will post data that shows up on the server. That whole thing isnā€™t valid JSON, but the {part} is. Because of that valid JSON, and following the Input API help page, I donā€™t think thereā€™s misplaced quotes.

Iā€™ve just had a go and the command works for me as well.

All I can think is that this is something to do with how the CPP actually sends the string. I still think you may need to urlencode it.

I would quite like to get to the bottom of it if you are OK with that - Iā€™m not running anything I can test it with.

If you edit that line and add the data item, what do you get back from the server?

return "Input in not a valid JSON object : ".$datain;

Try this using the fulljson data item if you could.

Hi Brian, I hadnā€™t touched any of the emoncms files yet, so at first I wasnā€™t sure where to add ā€œ$datainā€, but I figured it out. Sending the server a string with ā€œfulljson=ā€ results in this:

2019-11-27 13:27:14.573|ERROR|input_controller.php|{"success": false, "message": "Input in not a valid JSON object{Temperature:-196.60,ct1.apparentPower:2.44,ct2.apparentPower:2.50,ct3.apparentPower:2.73,ct4.apparentPower:2.65}"} for User: 2

I looked into urlencode a little, but all I found was a Python library. The Yun has linux running so I could put a script on that side and call it from the runShellCommand in Arduino, but Iā€™m surprised C canā€™t do it. But I guess thatā€™s not that C canā€™t do it, itā€™s that C canā€™t do it easily for a non-programmer, while Python may have a handy function or wrapper. What does url encoding do in this situation? I also found people saying not to use strings on Arduino and to use character arrays instead. But I think thatā€™s related to the Arduino performance, but not necessarily to the problem here.

Iā€™m reasonably certain it is something to do with the double quotes around the keys.

From the command line, using curl (with a comma missing so it will fail),

root@DietPi-V6-HA:~# curl --data 'node=1&fulljson={"power1":100"power2":200,"power3":300}&apikey=26135609fde7afcacbc40c74ff96xxxxxx' "http://192.168.7.77/input/post"

I get as a responseā€¦

{"success": false, "message": "Input in not a valid JSON object : {'power1':100'power2':200,'power3':300}"}

Note how this differs from your response - your response has no quotes around the keys.

Using tshark to look at what is received by the emoncms server;

sudo tshark -Y http.request.method==POST -T fields -e http.file_data

I see the following data when using the above curl command

node=1&fulljson={"power1":100"power2":200,"power3":300}&apikey=26135609fde7afcacbc40c74ff968

Clearly curl from the command line is sending the quotes.

If you could try that tshark command on the receiving server (you will have to install tshark), and see what the CPP curl is sending I am pretty certain the quotes will be missing.

You could try ā€˜escapingā€™ the quotes round the keys and see if that helps.

@damonlane Did a bit more digging. I think you need to escape the double quotes round the key names as you build the JSON string.

Hi Brian, you are right that the quotes are gone by the time the string gets to the server; hereā€™s what tshark returned:

node=Seeed&fulljson={Temperature:-196.60,ct1.apparentPower:2.07,ct2.apparentPower:2.12,ct3.apparentPower:2.68,ct4.apparentPower:3.01}&apikey=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

But I thought I was escaping the quotes, hereā€™s part of the Arduino sketch:

  dataString = "{\"Temperature\":";
  dataString += sensors.getTempFByIndex(0);
  dataString += ",\"ct1.apparentPower\":";
  dataString += 2*ct1.apparentPower;//these have 2x because the CTs are around 240v circuits while the voltage sensor is plugged into 120v
  dataString += ",\"ct2.apparentPower\":";
  dataString += 2*ct2.apparentPower;
  dataString += ",\"ct3.apparentPower\":";
  dataString += 2*ct3.apparentPower;
  dataString += ",\"ct4.apparentPower\":";
  dataString += 2*ct4.apparentPower;
  ...
  url += dataString;
  url += "}&";

Serial.println(url) in the code prints the string below in the serial monitor, making it look like the quotes made it out of the Arduino C/C++ code and should have been sent to the server.

20:50:04.036 -> curl --data "node=Seeed&fulljson={"Temperature":-196.60,"ct1.apparentPower":2.20,"ct2.apparentPower":2.46,"ct3.apparentPower":3.00,"ct4.apparentPower":3.07}&apikey=nnnnnnnnnnnnnnnnnnnnnnnnnnn" "http://192.168.1.116/input/post"

Does losing quotes in translation have to do with urlencoding? None of the quotes show up in tshark though they do in the finished url. Hereā€™s the spot in the Arduino sketch where the url is assembled:

  String url = "curl --data \"node=Seeed&fulljson=";
  url += dataString;
  url += "}&";
  url += apiString;
  url += "\" \"http://192.168.1.116/input/post\"";