Community
OpenEnergyMonitor

Community

Arduino Yun with TX shield updates only once each time it boots

Tags: #<Tag:0x00007f5b71676500>

Hello, I have a Yun+emonTx shield that had been monitoring some heat pumps for over a year and continued after I installed a home server and aimed it at that instead of emoncms.org. After getting it a nifty new case (and possibly updating libraries or Arduino board info) and reinstalling it, after each reboot it would only update the feeds once. It continues running the sketch, writing to serial, and flashing the LED at each update, but without any further activity visible on the server.

If I were very organized I would have made notes of any changes or code “cleanup” I may have done when this started happening but I didn’t, and then I set it aside while I worked on other things so my memory is extra fuzzy. I just ran diff on an older version of my sketch that used emoncms.org and this one and didn’t notice anything more than trivial differences in comments and expected changes like the api key.

I wondered if the way the data and api strings were assembled was it since that could be different the first vs subsequent times, but the code looks ok to me and if I copy the string from the serial monitor and paste it into a browser, it is accepted as a new update.

The Yun stays online during this process, visible on the router and accessible through its linux interface.

Any ideas about what could be wrong?

Here’s my sketch:

// based on http://forum.arduino.cc/index.php?topic=201925.0
#include "EmonLib.h" //for energy monitoring
#include <Bridge.h> //for Yun linux to Arduino connection
#include <OneWire.h> //for thermometer
#include <DallasTemperature.h> //for thermometer
#include <FileIO.h> //for writing to SD card
//#include <avr/wdt.h> // My energy monitor goes down every couple of days or after several months, so I added a watchdog timer. 
//At first try of the WDT, I could only get one data point so I diasbled it until I can troubleshoot.

#define DEBUG 1
#ifdef DEBUG
  #define DEBUG_PRINTLN(x)  Serial.println(x)
#else..
  #define DEBUG_PRINTLN(x)
#endif

// Create  instances for each CT channel
EnergyMonitor ct1; //ct1 is HPWH for 71
EnergyMonitor ct2; //ct2 is HPWH for 73
EnergyMonitor ct3; //ct3 is ASHP for 71
EnergyMonitor ct4; //ct4 is ASHP for 73

// On-board emonTx LED
const int LEDpin = 9;

//Setup thermometer
#define ONE_WIRE_BUS 4
OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature sensors(&oneWire);

const unsigned long postingInterval = 60000;
unsigned long lastRequest = 0;
String Timestamp = "";
String dataString =  "";
String APIKEY = "api key here";

void setup() {

  DEBUG_PRINTLN("Finished initialization, entering setup");

  // Initialize Bridge
  Bridge.begin();

  // Initialize Console
  Console.begin();

  // Start serial port
  Serial.begin(9600);

  // Start file system for SD card logging
  FileSystem.begin();

  // Start up the thermometer library
  sensors.begin();

  // 1st arg: pin number
  // 2nd arg: Calibration factor = CT ratio / burden resistance = (100A / 0.05A) / 33 Ohms = 60.606
  ct1.current(1, 20);
  ct2.current(2, 20);
  ct3.current(3, 25);
  ct4.current(4, 25);

  // (ADC input, calibration, phase_shift)
  ct1.voltage(0, 130, 1.7);
  ct2.voltage(0, 130, 1.7);
  ct3.voltage(0, 130, 1.7);
  ct4.voltage(0, 130, 1.7);

  // Setup indicator LED
  pinMode(LEDpin, OUTPUT);

  DEBUG_PRINTLN("Finished setup, entering delay to wait for wifi");

  // Hoping a delay allows the wifi connection to establish for more reliable operation
  delay(15000);
  
  //wdt_enable(WDTO_8S); // Setup watchdog timer, after the delay 
}

void loop() {

  DEBUG_PRINTLN("Entering loop");

  long now = millis();
  
  // call sensors.requestTemperatures() to issue a global temperature
  // request to all devices on the bus
  sensors.requestTemperatures(); // Send the command to get temperatures

  // Calculate all. No.of crossings, //time-out
  ct1.calcVI(20, 2000);
  ct2.calcVI(20, 2000);
  ct3.calcVI(20, 2000);
  ct4.calcVI(20, 2000);

  if (now - lastRequest >= postingInterval) {
   // wdt_reset(); //once here in the main loop 'if' and once at the start of each of these routines
    updateData();
    sendData();
    writeToCard();
    lastRequest = now;
  }

  DEBUG_PRINTLN("Finished loop, about to delay");

  //broke thirty second delay into 5-second chunks to keep watchdog happy. Could also disable it for this, but someone online
  //specifically suggested someone use the approach below instead of disabling it
 /* wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();
  delay(5000);
  wdt_reset();*/
  delay(10000);
}

void updateData() {
  //wdt_reset();
  // convert the readings to a String to send it:
  digitalWrite(LEDpin, HIGH); //turn emon shield red LED on
//Timestamp moved to the write to card portion only because that has no other timestamp and the Emon server doesn't like the timestamp format
  dataString = "Temperature:"; //need to add a comma in front, and switch to += if I turn the timestamp back on
  dataString += sensors.getTempFByIndex(0);
  dataString += ",ct1.apparentPower:";
  dataString += 2*ct1.apparentPower;//these have 2x because the CTs are around 240v circuits while the voltage sensor is plugged into 120v
  dataString += ",ct2.apparentPower:";
  dataString += 2*ct2.apparentPower;
  dataString += ",ct3.apparentPower:";
  dataString += 2*ct3.apparentPower;
  dataString += ",ct4.apparentPower:";
  dataString += 2*ct4.apparentPower;
  digitalWrite(LEDpin, LOW); //turn emon shield red LED off
  DEBUG_PRINTLN("updateData: ");
  DEBUG_PRINTLN(dataString);
  DEBUG_PRINTLN("Finished updateData");
  
}

// this method makes a HTTP connection to the server:
void sendData() {
  //wdt_reset();
  // form the string for the API header parameter:
  String apiString = "apikey=";
  apiString += APIKEY;

  // form the string for the URL parameter:
  // specify the server address
  String url = "http://192.168.1.116/emoncms/input/post?";
  url += "node=Yun";
  url += "&csv=";
  url += dataString;
  url += "&";
  url += apiString;

  DEBUG_PRINTLN("sendData dataString: ");
  DEBUG_PRINTLN(dataString);
  DEBUG_PRINTLN("sendData url: ");
  DEBUG_PRINTLN(url);
    
  // Is better to declare the Process here, so when the
  // sendData function finishes the resources are immediately
  // released. Declaring it global works too, BTW.
  Process emoncms;
  Console.print("\n\nSending data to EmonCMS... ");
  emoncms.begin("curl");
  //emoncms.addParameter("-g"); //was commented in my old sketch, tried in this one when I was only getting one data post per boot
  emoncms.addParameter(url);
  emoncms.run();
  Console.println("upload to EmonCMS done!");
  DEBUG_PRINTLN("upload to EmonCMS done!");

  // If there's incoming data from the net connection,
  // send it out the Console:
  while (emoncms.available()>0) {
    char c = emoncms.read();
    Console.write(c);
    DEBUG_PRINTLN(c);
  }
}

void writeToCard() {
  //wdt_reset();
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  // The FileSystem card is mounted at the following "/mnt/FileSystema1"
  File dataFile = FileSystem.open("/mnt/sd/arduino/emon/datalog.csv", FILE_APPEND);

  Timestamp = getTimeStamp(); 
  dataString += ",Timestamp:";
  dataString += Timestamp;

  // if the file is available, write to it:
  if (dataFile) {
    dataFile.println(dataString);
    dataFile.close();
  }
  
  // if the file isn't open, pop up an error:
  else {
    Console.println("error opening datalog.csv");
    DEBUG_PRINTLN("error opening datalog.csv");
  }

  DEBUG_PRINTLN("Finished writeToCard");

}

// This function return a string with the time stamp
// from the bottom of this Yun specific tutorial https://www.arduino.cc/en/Tutorial/YunDatalogger
String getTimeStamp() {
  //wdt_reset();
  String result;
  Process time;
  // date is a command line utility to get the date and the time
  // in different formats depending on the additional parameter
  time.begin("date");
  time.addParameter("+%D-%T");  
  // parameters: D for the complete date mm/dd/yy
  // T for the time hh:mm:ss
  time.run();  // run the command

  // read the output of the command
  while (time.available() > 0) {
    char c = time.read();
    if (c != '\n')
      result += c;
  }

  return result;
  
  DEBUG_PRINTLN("Got TimeStamp");
  
}

So what’s different about the way the browser sends the data?

Knowing nothing about your Yun, does Process emoncms require shutting down in any particular way before it dies as it goes out of scope?

Search the forum for “Authorization: Bearer”. I had an issue with something similar and even added some notes to the Input API help page on your installation (top right of Inputs page).

Probably because the ‘browser’ is already authenticated.

Thanks Brian,

On the Feed API Help page, it looks like any of those three ways to provide the apikey would apply in a given situation and I was already using middle one: appending the post parameter (and using the address …/emoncms/input/post/…). I tried switching to the -H one you mention, but couldn’t get it to work. I think that the Arduino’s attempt may not work because I might be mixing methods now (I know these words are involved, don’t know anything about them: curl, GET, post, JSON, etc.) If I log the browser out of the server, then paste the string, the server responds: “Username or password empty” Whereas, if I go back to the string I had been using, and use that in a browser logged out, the server says “ok”. It may be that I didn’t set up the -H parameter correctly, but it also seems like the POST parameter should work for the Arduino more than once.

There’s both the question of what’s different between the browser’s input and the Arduino’s, but maybe closer to the problem: what’s different between the first post the Arduino makes when it reboots and every one after that? When I test the browser, I use a newer string, not the first one.

So then, to Robert’s post…I’ve been treating the emoncms commands as complete back boxes and don’t know if they have changed since I set this up years ago, or if there’s ways to change how they are used that might solve this. I mean short of asking the Arduino to reboot after each post.

I have no idea how the sketch works or what the Process command does, but the error you are seeing is because what you are doing is not authenticating correctly.

If the string generated is a curl command, if you have another linux machine on your network, you could try the curl command directly from that.

Can you post the actual url ‘command’ used (with munged API key)?

Just to be clear, you are using the Input API (note ‘input’ in the URI) and not the Feed API.

Here’s the string, copied from the serial monitor after the Arduino wakes up, this one added a new point to the server:

http://192.168.1.116/emoncms/input/post?node=Yun&csv=Temperature:69.24,ct1.apparentPower:0.00,ct2.apparentPower:2.09,ct3.apparentPower:7.43,ct4.apparentPower:49.60&apikey=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

And here’s one a couple of minutes later, that did not generate a new data point on the server:

http://192.168.1.116/emoncms/input/post?node=Yun&csv=Temperature:69.35,ct1.apparentPower:0.00,ct2.apparentPower:0.00,ct3.apparentPower:0.00,ct4.apparentPower:0.07&apikey=nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

In the serial monitor, I confirmed the APIKEY is the same in both strings.

It doesn’t seem like the emoncms process should be doing anything differently when it first starts up vs the next update, since it’s called within sendData() in the loop() so I would think it is called fresh each time, but maybe it does need to be shut down? I hadn’t seen that in any documentation. Granted I really got my head in this when I started in 2016 or earlier, but have only skimmed since then. Seems like a number of things have changed and there’s abandoned methods and archived docs.

On the Input API page, it still looks like I’m using one of the methods, the third one in this case, my appending the APIKEY to the end of the url. It says a POST parameter is recommended instead. I played around adding an emoncms.addParameter() before the emoncms.run(), but wasn’t sure about the format. I tried to duplicate what it shows on the Input API page including the quotes, since the POST parameter option has quotes and the others don’t. In that case, the serial monitor was showing the server’s reply about no username and password. In the normal failure modes, that does not appear. I tried switching emoncms.begin(“curl”) to emoncms.begin(“GET”) and that failed to post or to get an error message from the server. But even in that case, if I took the url from the serial monitor, it still worked from the browser, though it correctly rejected an incorrect APIKEY when I accidentally pasted the following line along with the url. I did learn through this part that “emoncms” as a process is just the arbitrary name that the person I copied the code from used. I had previously thought it was part of emonlib and hadn’t paid it any attention.

It is odd that it works once.

I forgot to ask what version of EmonCMS you are running.

Looking at the Input API, the data structure you are using is a mix of JSON and CSV although it does seem to work when I tested it.

All I can suggest is using tshark (search here for some examples) to see what is actually being sent on the network.

Other than that I’m out of ideas - sorry. These sketches are not really my area :slight_smile:

Thanks Brian, I’m using 10.1.10. I realize I need to change my approach from looking for a simple fix to some breakage, to reconsidering how the sketch sends data. I had thought it was a part of the emoncms system, like another project I have that handles the communication with the server, but I see in the examples that the sending it completely different depending on the type of transmission.

I think I was initially using JSON, but switched to CSV when I had a previous problem. This is another reason to sort of start over.