Starting out - Raspberry PI3b / Arduino Uno R3 => local Emoncms

Cheers Robert,

Thanks for the welcome…

Fortunately I have good flexibility - PI One / Arduino are currently in the garage immediately under consumer box / next to the meter cupboard.

PI Two (which I incorrectly referred to as EmonHub) is setup successfully running EmonCMS (although it is twiddling its thumbs wondering why I bothered :wink: . It is currently connected via ethernet which I understand to be preferred. It’ll be located in the garage too which is where the routers are located so wired connection or WiFi are both possible.

I’ll take a look at your link - Thanks

Is there a particular reason you wanted to run two Pi’s? You can certainly run all of the components on a single Pi (equivalent to the EmonPi device available from the shop).

Hi Greebo,

No reason - more of a lack of misunderstanding and wanting to “run before I can walk”.

I did try removing PI One (Arduino IDE / Python EmonHub) and connecting PI Two (EmonCMS SD Card Build) direct to the Arduino - the plan being for the EmonCMS to take its serial feed direct from the Arduino via the USB cable but that didn’t work. I suspect it was because EmonHub wasn’t properly configured to accept the serial feed so the Arduino wasn’t being asked for TX anything so it just sat there.

I did eventually get it all (sort of) working between the 2 PIs. Emonhub on PI One has been busily scrolling data / log on its PuTTY window and EmonCMS sporadically receives the data. It collected data for half an hour last night and then stalled - I restarted everything both PIs and it resumed, graphing on EmonCMS through to about 5am and then it stopped. The serial output is pushing data out at approximately 3 lines per second according to the PuTTY window but the EmonCMS seems to fall behind quickly.

Emonhub (Pi One) is set to receive this data on node 5 but I’ve noticed in EmonCMS that I’ve now seem to have spurious data across a number of nodes / feeds so all is not right at theEmonCMS end - I’ll trim down its EmonHub config (just in case).

Can you supply more detail on what is running on the Arduino and how it is connected to the Pi?

Just a slightly modifed version of the “voltage and current” sketch taken from GitHub - openenergymonitor/EmonLib: Electricity monitoring library - install in Arduino IDE's libraries folder then restart the IDE (sketch copied below) and Emonhub. Connected to the PI via USBcable (/dev/ttyACM0)

#include <EmonLib.h>

// EmonLibrary examples openenergymonitor.org, Licence GNU GPL V3

#include “EmonLib.h” // Include Emon Library
EnergyMonitor emon1; // Create an instance

const byte emonPi_nodeID=5; // Added based on old sketch emonpi/emonPi_discrete_sampling_test.ino at master · openenergymonitor/emonpi · GitHub

void setup()
{
Serial.begin(9600);
Serial.println(“Arduino Uno R3 Startup”);
Serial.print("nodeID "); Serial.println(emonPi_nodeID);

emon1.voltage(2, 227.590.965, 1.7); // Voltage: input pin, calibration, phase_shift
emon1.current(1, 111.1
0.552); // Current: input pin, calibration.
}

void loop()
{
emon1.calcVI(20,2000); // Calculate all. No.of half wavelengths (crossings), time-out
// emon1.serialprint(); // Print out all variables (realpower, apparent power, Vrms, Irms, power factor) remarked out 1st Feb
Serial.print(emonPi_nodeID); Serial.print(’ ‘);
Serial.print(emon1.realPower); Serial.print(’ ‘);
Serial.print(emon1.apparentPower); Serial.print(’ ‘);
Serial.print(emon1.Vrms); Serial.print(’ ‘);
Serial.print(emon1.Irms); Serial.print(’ ');
Serial.println(emon1.powerFactor);

float realPower = emon1.realPower; //extract Real Power into variable
float apparentPower = emon1.apparentPower; //extract Apparent Power into variable
float powerFActor = emon1.powerFactor; //extract Power Factor into Variable
float supplyVoltage = emon1.Vrms; //extract Vrms into Variable
float Irms = emon1.Irms; //extract Irms into Variable
}

Two misunderstandings there, I think. Firstly, it looks as if the Arduino wasn’t sending the data in the format that emonHub expects, and secondly, that’s exactly what happens: the Arduino/emonTx/emonTH are expected to send the data, emonHub/emonCMS doesn’t ask for it. That’s fundamental to the OEM philosophy.

Here’s a slice of code from an emonTx sketch, showing the data and more importantly, the format in which it’s sent:

#ifdef EMONHUBTX3EINTERFACER
      // ---------------------------------------------------------------------
      // Key:Value format, used by EmonESP & emonhub EmonHubTx3eInterfacer
      // ---------------------------------------------------------------------
      Serial.print(F("MSG:")); Serial.print(emontx.Msg);
        
      if (CT1) { Serial.print(F(",P1:")); Serial.print(emontx.P1); }
      if (CT2) { Serial.print(F(",P2:")); Serial.print(emontx.P2); }
      if (CT3) { Serial.print(F(",P3:")); Serial.print(emontx.P3); }
      if (CT4) { Serial.print(F(",P4:")); Serial.print(emontx.P4); }
      
      if (CT1) { Serial.print(F(",E1:")); Serial.print(emontx.E1); }
      if (CT2) { Serial.print(F(",E2:")); Serial.print(emontx.E2); }
      if (CT3) { Serial.print(F(",E3:")); Serial.print(emontx.E3); }
      if (CT4) { Serial.print(F(",E4:")); Serial.print(emontx.E4); }
      
      if (emontx.T1!=30000) { Serial.print(F(",T1:")); Serial.print(emontx.T1*0.01); }
      if (emontx.T2!=30000) { Serial.print(F(",T2:")); Serial.print(emontx.T2*0.01); }
      if (emontx.T3!=30000) { Serial.print(F(",T3:")); Serial.print(emontx.T3*0.01); }

      Serial.print(F(",pulse:")); Serial.println(emontx.pulse);  
#else

... ['standard' format]

So you need to copy that format for whatever data you want to send.
You did follow up with the change to emonHub for the serial interfacer?

I think emonCMS choked on that.

The old emonLib sketches used to pause for 10 s between sampling the power, so a snapshot was sent every 10 s, which emonCMS can easily keep up with. The modern emonLibCM (Zip download here: EmonLibCM - Version 2.2.2) samples continuously and sends the average for that period - we keep to 10 s (actually a bit less at 9.85 s because of the way emonCMS works - but in principle you can set any sensible period, the limits are in the documentation that comes with the library (along with example sketches).

1 Like

Hi Robert,

I’m starting think I’m out of my depth - I could just about cope with Python and an ADC1115 chip which was outputting reasonable data with no AC monitoring. Now my head is now really hurting as there is so much I don’t currently understand - I would normally break things down into smaller bite size chunks but I’m not even sure what to go and chew on first.

I guess my first mouth full would be storing the raw data at the rate it is produced in a format I can view and manipulate - ideally in CSV, or a data feed that a spreadsheet can play with. I’m not sure if the SQL database in EmonCMS does this already and if I can just tap into it?

Then I could slow down and take my time to take myself through the various topics to get to a fully functioning emon setup which I also understand (to some degree).

I guess what’s frustrating is that the emocms works for a while with the data formatted as it is before it seems to “choke”. I’ll take a closer look at the emonTx extract you have provided - if I bottom the data format, I guess the one PI approach (running the emonpi/emoncms image) connected by USB to the arduino? (that’s me jumping ahead again).

In terms of the interfacer… - this is the extract from my emonhub config for SerialDirect

[[SerialDirect]]
Type = EmonHubSerialInterfacer
[[[init_settings]]]
com_port = /dev/ttyACM0 # or /dev/ttyAMA0 or/dev/ttyUSB0 etc
com_baud = 9600 # to match the baud of the connected device
[[[runtimesettings]]]
pubchannels = ToEmonCMS,

My emonhub config only covers interfacing for SerialDirect, MQTT (no idea if it should), and emoncmsorg (configured for my internal emoncms with api key)

Thank you for being patient, I’m far from giving up

Forgot to add the nodes section of my emonhub config…

[[5]]
nodename = emonPi
[[[rx]]]
names = powerreal,powerapp,vrms,irms,powerfactor
// datacodes = h, h, h, h, h
scales = 1,1,1,1,1,1
units = W,W,V,A,p

In terms of interfacer - I’ve followed the straight directserial interfacer examplar rather than the directserial-serialtx3e - is that the issue then, please say yes

Uh oh…

I’ve updated my serial printing, hopefully in accordance with the EMONHUBTX3EINTERFACER example (there is only one instance running EnergyMonitor emon1)

Serial.print(F(“MSG:”)); Serial.print(emonPi_nodeID);
Serial.print(F(",realPower:"));Serial.print(emon1.realPower);
Serial.print(F(",apparentPower:"));Serial.print(emon1.apparentPower);
Serial.print(F(",Vrms:"));Serial.print(emon1.Vrms);
Serial.print(F(",Irms:"));Serial.print(emon1.Irms);
Serial.print(F(",powerFactor:"));Serial.println(emon1.powerFactor);

I appreciate that there’s a lot to learn. Don’t worry, we’re used to it. I think you’re very close to having something you can work with.

Starting with the last post first, you’re getting the values passed through to emonHub, but it’s objecting to the decimal in the floating-point values. If you dig through our sketches, we only ever transmit integers, so what we do when it’s necessary to have a fractional value (for a temperature, say) is multiply by 10 or 100 at source and use "scales = " in emonHub to recover the true value.

If you look at the interfacers and the sketches that go with them, the code snippet I gave you is for the “3e” interfacer. This accepts “name:value” pairs - I think that means it doesn’t need/use a “nodes” definition in emonHub to give names to the incoming data.
The code for the plain direct-serial interfacer is space-separated values, which means emonCMS doesn’t have names, emonHub adds them according to the ‘nodes’ entry it has.

I think you’ll struggle to do much in real time, unless you slow the data rate down. You’re probably sending at 3 sets of data per second because you didn’t know better. :wink: EmonLib is like that because of history: the emonTx started out being battery-powered, so to get a reasonable life, it sampled for 200 ms, then went to sleep with just a timer running for 10 s, then woke up and did it again. So if you stay with emonLib, at the end of the main loop, you need to add 9 s or a bit more of delay.

emonCMS doesn’t use the SQL for the data, only for the metadata. The real data is in a home-brew database called PHPFINA or PHPTIMESERIES. These are documented (sorry, your learning curve just got a big step in it) in ‘Learn’ and you choose which one to use when setting up the Feed on the Inputs page in emonCMS. emonCMS will give you dashboards with the present data, and graphs and histograms for historic data.

No, I think that’s the sensible and obvious way ahead. If you want a shortcut, for the price of an SD card you can download a working RPi OS with emonHub & emonCMS ready to run - you’ll just need to connect to your Wi-Fi or Ethernet and edit emonhub.conf again. Then you can get to know emonHub & emonCMS and you’ve got a standard system which will make it a lot easier for us to help you. Then, when you know what you’re doing, if you really want to delve into the nitty-gritty, swap SD cards and carry on.

1 Like

Once more, thank you for spending the time to respond…

Ultimately, I want (primarily) what everyone wants - that is a working energy monitor with a reasonably high degree of accuracy. My secondary want is to pick up new knowledge and skills as I go along.

My “PI Two” has the July 20 emonscripts / emonpi / emoncms (etc) image on it already, running the emonCMS locally. So my next question is how to set up my arduino - at the outset, I was religiously following the learning centre to the point I was getting (what looked like reasonably decent) data outputted to the Arduino serial monitor using the emonlib library and “voltage and current” example ino sketch.

So… which arduino libraries should I use and which corresponding arduino sketch as the basis? And once this is running, is it a case of re-installing the emonscripts SD card image on the PI connected to it? I take it I’ll still have a some configuring to do… If there is an idiots guide - please point me in that direction so that I’m not hogging your time (I’m sure I’ll have plenty more questions)

Ultimately I’d like to add other sensors to it - the next will be the pulse counter (planning to build hardware from scratch) to help with occasional recallibration to ultimately have the meter as close to synced with the actual dumb meter in the meter cabinet.

I’m reading our water meter too currently - this is just dumping info to a CSV file which I plan to reconfigure and post to a dashboard too (hopefully emonCMS though I know its not technically energy).

Thanks once again

The easy way to the first part conflicts somewhat with the second, because taking our “plug and play” SD card is the easy way, but you won’t learn as much as building the system yourself.

You have a Pi that’s getting data. I suggest continue with that for a while. Add a delay to your Arduino sketch, so that the data comes in every 9.8 s or so (as long as it’s under 10 s and you are logging in emonCMS every 10 s, that’s fine).
Send integer values, then emonHub won’t spit them out. The error will even out over time, or there are software tricks you can use to carry the rounding/truncation error over into the next reading.

Stay with what you have for now. Get it working first. We use emonLibCM in the sketch that the emonTx comes with, we’ll be moving before long to the same in the emonPi. Both are well proven and accurate long-term, but the discrete sample emonLib won’t give accurate results short-term if you have an electric hob for example that switches on and off rapidly, whereas the CM library does what it says - Continuously Monitors. It’s your decision, but I’d prove it “works”, then switch to the CM library. It’s totally different, but the user interface is hopefully simpler. Don’t try to understand the inner workings just yet though.

I’m going to stop there because I think adding the pulse inputs to the Pi is the next level, and it’s pretty much separate from getting the analogue energy values in.

1 Like

Cheers Robert - I will have a ponder and possibly a play. I did have a look at the emonlibCM library earlier, its got some really neat features in it.

Thanks

UPDATE:
Had a bit of a play about…
I was getting a lot of null data in the csv view, and the delays on the feeds were getting progressively longer. So I slowed the arduino down (got a lovely yellow LED flashing away merrily now). I then noticed on the emonhub output (arduino end) that the httpemoncms service was buffering each of the lines being outputted from the serial interface: SerialDirect line 1, line 2, line 3, line 4 etc… and then all of those were buffered by the httpemoncms interface and then sent - at this point the emonCMS would reduce its feed delay and it would once more show green - the delay would then increase through amber before peaking at 30 - 40s (by which time the next buffered collection would be received.

So I’ve reduced the interval in the httpemoncms interface from 30s to 3s to keep the delay generally below 5s and therefore in the green. Furthermore, no more “null” data in the CSV output on the graphing page.

I don’t know how emonCMS processes the data it receives - does it matter that there could be 5 lines of data in the buffered feed? Does it use all the data to produce one datapoint at 10s or does it just take one line from the feed?

Eitherway - happy I’ve got some output, feeds are green and no nulls. I’ll leave it running for a while to see how stable it is.

Thanks

Others here know a lot more about emonCMS than I do, but I think what you’ve seen is part of the mechanism that reduces the number of writes to the SD card, so doing what you’ve done might well have spoiled that mechanism. And I might be wrong. I think we need @borpin to help me out.

There is a help item somewhere on how the timeseries data is fed but can’t find it. @pb66 will correct any eroneous things I may say here :slight_smile:

At the emoncms end, in simple terms, the interval you set the Feed to, is how often the value in the input is recorded. If you are sending input values once a second, and the feed is set to 10s, you will get the last value of the input in that 10s window. If the Input rate is slower than the feed rate, you will get nulls as the input has not updated in that window. It is generally better to have the input running quicker than the feed. If this is very ‘slight’, you get drift so once every x times you miss an input value as 2 input values occur within the same window so only the second goes to the feed.

The HTTP emonhub interface does ‘buffer’ this was designed to help with intermittent network and to reduce the load on the server at the other end. However, at the emoncms end it takes the timestamps so you could still see the effect above. This HTTP buffer does mean that the Inputs page do go from green to amber or red depending on the buffer size.

Secondly, if you have REDIS running, this again buffers the data - this is to reduce write load on the SD Card. This doe though mean if you are looking at a graph, the data is not live (it waits until it has been written to disk). Not sure about the dashboard widgets though.

I think I got that right…

In the context of what you are explaining, yes you do ++get every 10th value++ but in actual fact, when running low-write emoncms ie feedwriter, you would in fact get the first value in any 10s window.

Feedwriter will only update the feed once and then not update further, ie not over write existing values. Whilst standard emoncms would in fact write the given feed value window 10x, once each time the 1s input is updated so it is only then that you end up with the last value jn eazh 10s window.

But yes overall that sounds pretty much it for fixed timeseries feed data.

Ah, had not realised that - thanks.

Can’t remember if the timestamp is window start or window end?

Thanks all (back at work today and have only just sat down to catch up). I’ll continue to tinker with the values and will read up on emoncms. Its been running for almost 24h, I don’t appear to have any errors in the feed data being sent and the feeds look healthy (green) at the emoncms end.

The integrations are way way off (I appreciate I’ve still got some callibration to do at the arduino end as I’ve got deltas between the apparent and real power at lower current draws.

Thanks again.