Starting out - Raspberry PI3b / Arduino Uno R3 => local Emoncms

Getting Started
8

Two misunderstandings there, I think. Firstly, it looks as if the Arduino wasn’t sending the data in the format that emonHub expects, and secondly, that’s exactly what happens: the Arduino/emonTx/emonTH are expected to send the data, emonHub/emonCMS doesn’t ask for it. That’s fundamental to the OEM philosophy.

Here’s a slice of code from an emonTx sketch, showing the data and more importantly, the format in which it’s sent:

#ifdef EMONHUBTX3EINTERFACER
      // ---------------------------------------------------------------------
      // Key:Value format, used by EmonESP & emonhub EmonHubTx3eInterfacer
      // ---------------------------------------------------------------------
      Serial.print(F("MSG:")); Serial.print(emontx.Msg);
        
      if (CT1) { Serial.print(F(",P1:")); Serial.print(emontx.P1); }
      if (CT2) { Serial.print(F(",P2:")); Serial.print(emontx.P2); }
      if (CT3) { Serial.print(F(",P3:")); Serial.print(emontx.P3); }
      if (CT4) { Serial.print(F(",P4:")); Serial.print(emontx.P4); }
      
      if (CT1) { Serial.print(F(",E1:")); Serial.print(emontx.E1); }
      if (CT2) { Serial.print(F(",E2:")); Serial.print(emontx.E2); }
      if (CT3) { Serial.print(F(",E3:")); Serial.print(emontx.E3); }
      if (CT4) { Serial.print(F(",E4:")); Serial.print(emontx.E4); }
      
      if (emontx.T1!=30000) { Serial.print(F(",T1:")); Serial.print(emontx.T1*0.01); }
      if (emontx.T2!=30000) { Serial.print(F(",T2:")); Serial.print(emontx.T2*0.01); }
      if (emontx.T3!=30000) { Serial.print(F(",T3:")); Serial.print(emontx.T3*0.01); }

      Serial.print(F(",pulse:")); Serial.println(emontx.pulse);  
#else

... ['standard' format]

So you need to copy that format for whatever data you want to send.
You did follow up with the change to emonHub for the serial interfacer?

I think emonCMS choked on that.

The old emonLib sketches used to pause for 10 s between sampling the power, so a snapshot was sent every 10 s, which emonCMS can easily keep up with. The modern emonLibCM (Zip download here: EmonLibCM - Version 2.2.2) samples continuously and sends the average for that period - we keep to 10 s (actually a bit less at 9.85 s because of the way emonCMS works - but in principle you can set any sensible period, the limits are in the documentation that comes with the library (along with example sketches).

1 Like
9

Hi Robert,

I’m starting think I’m out of my depth - I could just about cope with Python and an ADC1115 chip which was outputting reasonable data with no AC monitoring. Now my head is now really hurting as there is so much I don’t currently understand - I would normally break things down into smaller bite size chunks but I’m not even sure what to go and chew on first.

I guess my first mouth full would be storing the raw data at the rate it is produced in a format I can view and manipulate - ideally in CSV, or a data feed that a spreadsheet can play with. I’m not sure if the SQL database in EmonCMS does this already and if I can just tap into it?

Then I could slow down and take my time to take myself through the various topics to get to a fully functioning emon setup which I also understand (to some degree).

I guess what’s frustrating is that the emocms works for a while with the data formatted as it is before it seems to “choke”. I’ll take a closer look at the emonTx extract you have provided - if I bottom the data format, I guess the one PI approach (running the emonpi/emoncms image) connected by USB to the arduino? (that’s me jumping ahead again).

In terms of the interfacer… - this is the extract from my emonhub config for SerialDirect

[[SerialDirect]]
Type = EmonHubSerialInterfacer
[[[init_settings]]]
com_port = /dev/ttyACM0 # or /dev/ttyAMA0 or/dev/ttyUSB0 etc
com_baud = 9600 # to match the baud of the connected device
[[[runtimesettings]]]
pubchannels = ToEmonCMS,

My emonhub config only covers interfacing for SerialDirect, MQTT (no idea if it should), and emoncmsorg (configured for my internal emoncms with api key)

Thank you for being patient, I’m far from giving up

10

Forgot to add the nodes section of my emonhub config…

[[5]]
nodename = emonPi
[[[rx]]]
names = powerreal,powerapp,vrms,irms,powerfactor
// datacodes = h, h, h, h, h
scales = 1,1,1,1,1,1
units = W,W,V,A,p

11

In terms of interfacer - I’ve followed the straight directserial interfacer examplar rather than the directserial-serialtx3e - is that the issue then, please say yes

12

Uh oh…

I’ve updated my serial printing, hopefully in accordance with the EMONHUBTX3EINTERFACER example (there is only one instance running EnergyMonitor emon1)

Serial.print(F(“MSG:”)); Serial.print(emonPi_nodeID);
Serial.print(F(",realPower:"));Serial.print(emon1.realPower);
Serial.print(F(",apparentPower:"));Serial.print(emon1.apparentPower);
Serial.print(F(",Vrms:"));Serial.print(emon1.Vrms);
Serial.print(F(",Irms:"));Serial.print(emon1.Irms);
Serial.print(F(",powerFactor:"));Serial.println(emon1.powerFactor);

image
image906×450 79.2 KB

13

I appreciate that there’s a lot to learn. Don’t worry, we’re used to it. I think you’re very close to having something you can work with.

Starting with the last post first, you’re getting the values passed through to emonHub, but it’s objecting to the decimal in the floating-point values. If you dig through our sketches, we only ever transmit integers, so what we do when it’s necessary to have a fractional value (for a temperature, say) is multiply by 10 or 100 at source and use "scales = " in emonHub to recover the true value.

If you look at the interfacers and the sketches that go with them, the code snippet I gave you is for the “3e” interfacer. This accepts “name:value” pairs - I think that means it doesn’t need/use a “nodes” definition in emonHub to give names to the incoming data.
The code for the plain direct-serial interfacer is space-separated values, which means emonCMS doesn’t have names, emonHub adds them according to the ‘nodes’ entry it has.

I think you’ll struggle to do much in real time, unless you slow the data rate down. You’re probably sending at 3 sets of data per second because you didn’t know better. :wink: EmonLib is like that because of history: the emonTx started out being battery-powered, so to get a reasonable life, it sampled for 200 ms, then went to sleep with just a timer running for 10 s, then woke up and did it again. So if you stay with emonLib, at the end of the main loop, you need to add 9 s or a bit more of delay.

emonCMS doesn’t use the SQL for the data, only for the metadata. The real data is in a home-brew database called PHPFINA or PHPTIMESERIES. These are documented (sorry, your learning curve just got a big step in it) in ‘Learn’ and you choose which one to use when setting up the Feed on the Inputs page in emonCMS. emonCMS will give you dashboards with the present data, and graphs and histograms for historic data.

No, I think that’s the sensible and obvious way ahead. If you want a shortcut, for the price of an SD card you can download a working RPi OS with emonHub & emonCMS ready to run - you’ll just need to connect to your Wi-Fi or Ethernet and edit emonhub.conf again. Then you can get to know emonHub & emonCMS and you’ve got a standard system which will make it a lot easier for us to help you. Then, when you know what you’re doing, if you really want to delve into the nitty-gritty, swap SD cards and carry on.

1 Like
14

Once more, thank you for spending the time to respond…

Ultimately, I want (primarily) what everyone wants - that is a working energy monitor with a reasonably high degree of accuracy. My secondary want is to pick up new knowledge and skills as I go along.

My “PI Two” has the July 20 emonscripts / emonpi / emoncms (etc) image on it already, running the emonCMS locally. So my next question is how to set up my arduino - at the outset, I was religiously following the learning centre to the point I was getting (what looked like reasonably decent) data outputted to the Arduino serial monitor using the emonlib library and “voltage and current” example ino sketch.

So… which arduino libraries should I use and which corresponding arduino sketch as the basis? And once this is running, is it a case of re-installing the emonscripts SD card image on the PI connected to it? I take it I’ll still have a some configuring to do… If there is an idiots guide - please point me in that direction so that I’m not hogging your time (I’m sure I’ll have plenty more questions)

Ultimately I’d like to add other sensors to it - the next will be the pulse counter (planning to build hardware from scratch) to help with occasional recallibration to ultimately have the meter as close to synced with the actual dumb meter in the meter cabinet.

I’m reading our water meter too currently - this is just dumping info to a CSV file which I plan to reconfigure and post to a dashboard too (hopefully emonCMS though I know its not technically energy).

Thanks once again

15

The easy way to the first part conflicts somewhat with the second, because taking our “plug and play” SD card is the easy way, but you won’t learn as much as building the system yourself.

You have a Pi that’s getting data. I suggest continue with that for a while. Add a delay to your Arduino sketch, so that the data comes in every 9.8 s or so (as long as it’s under 10 s and you are logging in emonCMS every 10 s, that’s fine).
Send integer values, then emonHub won’t spit them out. The error will even out over time, or there are software tricks you can use to carry the rounding/truncation error over into the next reading.

Stay with what you have for now. Get it working first. We use emonLibCM in the sketch that the emonTx comes with, we’ll be moving before long to the same in the emonPi. Both are well proven and accurate long-term, but the discrete sample emonLib won’t give accurate results short-term if you have an electric hob for example that switches on and off rapidly, whereas the CM library does what it says - Continuously Monitors. It’s your decision, but I’d prove it “works”, then switch to the CM library. It’s totally different, but the user interface is hopefully simpler. Don’t try to understand the inner workings just yet though.

I’m going to stop there because I think adding the pulse inputs to the Pi is the next level, and it’s pretty much separate from getting the analogue energy values in.

1 Like
16

Cheers Robert - I will have a ponder and possibly a play. I did have a look at the emonlibCM library earlier, its got some really neat features in it.

Thanks

17

UPDATE:
Had a bit of a play about…
I was getting a lot of null data in the csv view, and the delays on the feeds were getting progressively longer. So I slowed the arduino down (got a lovely yellow LED flashing away merrily now). I then noticed on the emonhub output (arduino end) that the httpemoncms service was buffering each of the lines being outputted from the serial interface: SerialDirect line 1, line 2, line 3, line 4 etc… and then all of those were buffered by the httpemoncms interface and then sent - at this point the emonCMS would reduce its feed delay and it would once more show green - the delay would then increase through amber before peaking at 30 - 40s (by which time the next buffered collection would be received.

So I’ve reduced the interval in the httpemoncms interface from 30s to 3s to keep the delay generally below 5s and therefore in the green. Furthermore, no more “null” data in the CSV output on the graphing page.

I don’t know how emonCMS processes the data it receives - does it matter that there could be 5 lines of data in the buffered feed? Does it use all the data to produce one datapoint at 10s or does it just take one line from the feed?

Eitherway - happy I’ve got some output, feeds are green and no nulls. I’ll leave it running for a while to see how stable it is.

Thanks

18

Others here know a lot more about emonCMS than I do, but I think what you’ve seen is part of the mechanism that reduces the number of writes to the SD card, so doing what you’ve done might well have spoiled that mechanism. And I might be wrong. I think we need @borpin to help me out.

19

There is a help item somewhere on how the timeseries data is fed but can’t find it. @pb66 will correct any eroneous things I may say here :slight_smile:

At the emoncms end, in simple terms, the interval you set the Feed to, is how often the value in the input is recorded. If you are sending input values once a second, and the feed is set to 10s, you will get the last value of the input in that 10s window. If the Input rate is slower than the feed rate, you will get nulls as the input has not updated in that window. It is generally better to have the input running quicker than the feed. If this is very ‘slight’, you get drift so once every x times you miss an input value as 2 input values occur within the same window so only the second goes to the feed.

The HTTP emonhub interface does ‘buffer’ this was designed to help with intermittent network and to reduce the load on the server at the other end. However, at the emoncms end it takes the timestamps so you could still see the effect above. This HTTP buffer does mean that the Inputs page do go from green to amber or red depending on the buffer size.

Secondly, if you have REDIS running, this again buffers the data - this is to reduce write load on the SD Card. This doe though mean if you are looking at a graph, the data is not live (it waits until it has been written to disk). Not sure about the dashboard widgets though.

I think I got that right…

20

In the context of what you are explaining, yes you do ++get every 10th value++ but in actual fact, when running low-write emoncms ie feedwriter, you would in fact get the first value in any 10s window.

Feedwriter will only update the feed once and then not update further, ie not over write existing values. Whilst standard emoncms would in fact write the given feed value window 10x, once each time the 1s input is updated so it is only then that you end up with the last value jn eazh 10s window.

But yes overall that sounds pretty much it for fixed timeseries feed data.

21

Ah, had not realised that - thanks.

Can’t remember if the timestamp is window start or window end?

22

Thanks all (back at work today and have only just sat down to catch up). I’ll continue to tinker with the values and will read up on emoncms. Its been running for almost 24h, I don’t appear to have any errors in the feed data being sent and the feeds look healthy (green) at the emoncms end.

The integrations are way way off (I appreciate I’ve still got some callibration to do at the arduino end as I’ve got deltas between the apparent and real power at lower current draws.

Thanks again.

23

Presumably at very low currents?
That’s because all c.t’s are inherently inaccurate at low currents, “low” being relative to the rated current and a function of cost. I have plans to do something about it, but there’s no timescale.

24

Define very low currents…the household’s quiescent power usage was measured on an old energy monitor to be approx 300W (fridges).

According to the emoncms. apparent and real respectively, are approx. 430W and 250W (UK Voltage). Not properly read up about it but I’d assumed low current to be less than that - I’ve not tweaked the power coefficient yet, just the voltage and current.

During lockdown, with no kW draw devices, a crude energy monitor would show around 500 - 600W avg if the boys have their PCs on. Increases to 750W avg during the evening with TV / lights etc… Maximum draw in a day could be 9 - 12kW, say if there was one shower and a dryer on, 20kW if both showers and the dryer were on.

image
image1125×565 134 KB

25

Define what you’re prepared to spend for a higher quality c.t. :wink: 300 W is pushing your luck with a cheap (= YHDC SCT-013-00) 100 A c.t., accuracy is only specified above 10 A for that particular model.

I think you probably need to go through the full calibration procedure, and if you’re really interested in matching your meter reads, then be prepared to do very fine tweaks to the calibration until you get it “close enough” over an extended period of time (a few weeks). Otherwise, calibrate at around the current that you’re concerned about, but be prepared for the 10 - 20 kW loads to be wrong.

If you look at the c.t. test report in ‘Learn’, you’ll see how the phase error rockets up as the current drops. That’s what is causing most of the trouble.

26

Everyday’s a school day and I’m at the bottom of a steep learning curve. Had read that test report, admittedly I’d focussed on the first part.

That was the plan re: the full calibration - I’ll keep tweaking for a long while yet, both the calibration, the software and probably the hardware too. I wouldn’t be surprised if I end up learning that I need multiple current sensors, each with specific ranges of accuracy and the emonpi decides which sensor to use based on the current being measured :thinking:

27

I would (be surprised), because how are you going to switch the primary current between them? Each c.t has a maximum current rating, above which saturation occurs, and that can lead to overheating. Manufacturers always recommend using the c.t. that has the lowest rated current which is more than the maximum that it will see in service. Quite honestly, I’d describe that idea as ‘hopeful’. You’d need a 100 A contactor, driven from your Arduino, to put a shunt circuit around each of the lowest-rated c.t’s, and the resistance of that loop - including the contacts when they get old and worn - would need to be less than the resistance of the conductor through the c.t., by a factor that was at least the ratio of the currents. I can’t see it being achievable.

Also bear in mind that it’s always safe to short-circuit a c.t., and you should do if its burden is disconnected. It might be safe to open-circuit it, but that depends on the integrity of a protection diode - even if one is fitted, which is not necessarily the case. The only type that it’s safe to leave open-circuit is the voltage-output type.