Emontx3-continuous - New continuous monitoring firmware

Hardware emonTx
1

I’d like to announce a new firmware for the emonTx3 which does continuous sampling. It provides high rate continuous sampling (19230 samples per second), uses all four channels and provides extensive information about your power and energy usage.

The firmware is here: https://github.com/CraigMarkwardt/emontx3-continuous

I’ve been using it for the past few months and I have experienced accuracies of better than 1% compared to my official utility bills. I know there are several other attempts at doing continuous monitoring, but felt that my approach was significantly different and added enough capabilities, that it was worth publishing to the community. In particular, the fast sampling, true reactive power sampling, and continuous utility energy reporting are things I had not seen before.

Please enjoy! More details below, and a full report can be found on the github main page.

Summary

This is emontx3-continuous, a firmware for emonTx energy monitors using an Arduino microcontroller. Its key benefits are that it provides high rate (19230 samples/second) continuous sample of your power usage and reports cumulative energy use like a utility meter. It reports your active and reactive power usage to allow you to distinguish between resistive, capacitive and inductive loads in your household. It also reports your mains voltage, frequency and mains quality. It is designed to work with the “EmonESP” ESP8266 wifi output, transformer input, and 5V USB.

This firmware is recommended for users who:

  • intend to connect via EmonESP (not the built-in RFM69)
  • have an AC transformer input to measure mains voltage
  • want to achieve maximum accuracy through calibration (which requires rebuilding the firmware using the Arduino IDE)

This firmware is not recommended for users who:

  • use an emonBase (via the on-board RFM69); because RFM69 support is not possible
  • don’t have the AC-AC voltage sensor adapter; because the features of emonTx3-continuous require the mains voltage input.

Features

The emontx3-continuous firmware has the following features.

  • Continuously samples inputs at 19230 samples/second, or 3846 samples/second for each input channel, with full 10-bit ADC resolution
  • Intended to be used with “EmonESP” wifi transmitter
  • This sample rate is equivalent to 77 samples per 50 Hz AC waveform and 64 samples per 60 Hz AC waveform, which allows to capture most fast power transients such as switch-mode power supplies.
  • Supports all four emonTx current transformer inputs
  • Requires AC-AC voltage sense input, so that mains voltage can be measured
  • For each input channel reports, RMS current, power factor, active, reactive and apparent power. Active power is power in-phase with the mains voltage such as incandescent bulbs; and reactive power is 90 degrees out-of-phase with mains voltage (typically AC motors, compressors and fans).
  • Reports total cumulative active and reactive energy usage. This is cumulative and builds up over time, just like a utility meter.
  • Reports the AC mains voltage, mains frequency to three digits of accuracy, and the AC crest factor, which can be used to diagnose AC power faults.
  • Automatically determines mains AC frequency at startup (50 Hz or 60 Hz)
  • Is fully calibratable. Voltage, current and phase offset calibration factors are availaable.
  • Adjusts for sample time offsets beteween voltage and current, with accuracy of better than 1%.
  • Should be compatible with 3-phase power systems. The phase offset variables can be set to align all inputs to one voltage leg.
  • Measures pulse input to allow utility meter pulse input (not tested).
  • Reports other diagnostic information such as uptime, version, and internal buffer sizes.

Features not supported

  • the firmware uses features specific to the Atmel hardware, and will be difficult to port to other architectures.
  • the RFM69 transmitter on-board the emonTx is not supported; the transmitter software library is not compatible with the methods used by emonTx3-continous to achieve high sample rates (it is busy-wait driven instead of interrupt-driven).
  • the standard emonTx temperature/humidity sensor accessory is not usable because the DallasTemperature library is not compatible with the methods used by emonTx3-continous to achieve high sample rates (it is busy-wait driven instead of interrupt-driven).
  • a mains AC-AC transformer is required. No-voltage input is not supported. Without the AC-AC transformer, accuracy will be much lower and you may as well use the standard emonTx firmware.
  • because of the AC-AC transformer requirement, you will also need to power the unit separately with 5VDC (with the USB input, for example)

Available Readings

The emontx-continuous provides many readings about your home power system. Since it is different and more advanced than the standard emonTx firmware, do not expect the readings to be exactly the same. You will need to develop new charts and visualizations for all the data you are producing. Here is a description of the data fields.

Readings are produced with a 4-character name, such as “vrms” which identify the reading, and may be an integer or floating point value. Note that some readings start with an underscore, such as _ever. These are intended to be MQTT “retained” readings that are kept for late clients, and are typically reserved for readings which are diagnostic in nature.

Mains Voltage

emontx-continuous provides readings about your mains voltage, frequency and quality. This can inform you about the quality of mains voltage you are receiving from your utility. These values are reported every 10 seconds.

  • vman - mains AC voltage dip switch setting, in volts. Either 120 or 240.
  • vrms - mains AC voltage, in volts.
  • vfrq - mains AC frequency, in Hertz.
  • vcrs - mains AC crest factor, as a fraction. This is a diagnostic of your mains voltage quality. Crest factor is defined as Vmax/Vac where Vmax is the maximum voltage and Vac is the AC (rms) voltage. For typical AC voltage, the crest factor is 1.414; deviations indicate non-sinusoidal voltage condition.

Power Usage

emontx-continuous reports power usage of each of the four inputs. It reports the rms current, active and reactive power, and power factor. These readings are produced every 30 seconds, and represent the average power usage over the past 30 second interval.

  • irmsN - for current sensor N, rms current usage in Amps.
  • pacN - for current sensor N, active power usage in Watts.
  • preN - for current sensor N, reactive power usage in Watts. A positive value indicates capacitive load, negative indicates inductive.
  • powN - for current sensor N, fractional power factor. Power factor is defined as powN = pacN / papN, where papN = (vrms*irmsN) is the apparent power usage.

Cumulative energy monitoring

emontx-continuous reports cumulative energy usage just like an energy meter. It is the true continuous sum of energy usage without interruptions, averaged over the full waveform of every AC cycle. The reported values are produced every 60 seconds, and represent the sum of all available input channels.

  • _enac - total cumulative active energy usage, in kWh, for all four input channels combined.
  • _enre - total cumulative reactive energy usage, in kWh, for all four input channels combined. Your power company typically does not bill you for this energy usage.

Pulse counter

If you have a utility meter with LED pulser, you can retrieve these pulses with a pulse sensor.

  • pulse - The reported value is total number of pulses received since startup.

Diagnostics

These diagnostics are typically not useful for reporting home energy usage, but are useful for diagnosing the state of the device itself.

  • _ever - emonTx firmware version, as a 4-digit integer number.
  • _uptm - total system uptime, in seconds, since last reset.
  • _adcd - maximum ADC ring buffer depth. A diagnostic which indicates possible processing overload.
  • _novr - number of ADC samples lost due to ring buffer overflow. Any value different than zero indicates processor overload.
  • vdel - correction factor for out-of-phase voltage readings, as a fractional quantity.

More information on how it works and how to use it on the github page.

Reformatted post for readability and corrected RFM96 to RFM69 - BT, Moderator

3 Likes
2

Very nice! I’d not heard of those Atmel maths inlines. How do they compare with the code gcc-avr generates?

3

Thanks. There is no such thing as 16 x 16 = 32 integer multiplication within the standard Atmel C library, just 32 x 32 = 32. So in principle the inlines 16 x 16 = 24 inlines are 2x to 3x faster.

I did some tests a while ago, but don’t have the results anymore, sorry.

When I just use standard C multiply-and-sum there is a noticeable overall difference, but it’s not huge. There’s lots of other floating point math going on in the firmware too, which dominates.

4

Hi Craig,

Nice work.

How is the accuracy when measuring low current?
Of particular interest, accuracy at low current with a low .e.g. < 0.5 power factor.

5

I knew it could’ve been improved yet this is beyond expectation. Nicely done. I’ll be using this soon for displaying the energy use at an artist residency space, appreciated.
Version control is such a good idea, and best done sooner rather than later in the grand scheme of things I think. Version control for selective parsing methods for the data perhaps… Especially, I thought, in relation to RFM69 and emonBase, would be useful I think.
Using the huzzah module though, would it be possible with your sketch to issue commands or settings (like calibration setting, or different modes of operation) back to the arduino over wifi, would this be compatible with your program?

6

Nice work @cbmarkwardt! and thanks a lot for sharing. Im currently testing continuous sampling firmware developed by @Robert.Wall. Its not yet released (we are about 2 weeks testing away) but it would be great for you both to compare notes.

Thanks again for sharing and for detailing all the functionality.

7

Mine runs at the full 10-bit resolution of the ADC, and it’ll handle the RFM69 and temperature sensors. I won’t have time to look at @cbmarkwardt’s for some while, but I do wonder whether the extra speed is necessary, or even advisable, considering the limited amount of energy that’s legally present in the upper harmonics.

8

Hi @Bill.Thomson, I don’t think the new firmware will be any more or less accurate than the old firmware in that respect. It really matters what you mean by low current, but for the SCT-013-000 current transformers, 0.1 Amp (AC) is kind of the measurement tolerance “dead zone.” (quantization error)

Within that tolerance zone, no firmware will be able to effectively measure a current, much less distinguish power factor. As a rough estimate, the minimum power factor one can distinguish is 1 - (0.1 Amp)/(Current). So for a desired power factor of 0.5 that would be 0.5 = 1 - (0.1 Amp)/(Current), giving Current = 0.5 Amp. Very rough though.

9

@danbates, the data comes out as serial data, which the emonESP sends along to either MQTT or emonCMS for reporting. I believe any data that comes out that way should appear OK within either system. I’ve only tested with MQTT though, and it works fine.

10

@Robert.Wall, I was going for what could be described as “true rms” capability, usually approximately defined as 100 samples per waveform. It was a goal which I didn’t quite achieve, whether it was advisable or not. And for people that wish to diagnose their home energy system, testing the harmonic content would be of interest.

If one is doing continuous sampling with Atmel’s interrupt-driven free-running mode, anything that monopolizes the CPU for too long could cause missed data. If you were able to communicate with the RFM69 and temperature devices, then great!

11

I understand the way the data comes out, but I’m talking about data going IN to the device… I know, this is not standard for any emonTX… but I see benefit to uploading settings to the onboard ATMEGA in the future… I think this is especially possible with the ESP add-on to the TX… only the issue exists with the bent/swapped RX/TX connections, which is slightly fiddly.
I wondered if your sampling mode would be compatible with the ATMEGA receiving instructions on the RX line, or if there’s a compatibility issue for the same reason you’ve decided not to make available use of RFM. I don’t know. At the moment I don’t think there’d be a problem with it.

1 Like
12

@danbates, the current standard firmware does use the serial console for some basic input. Essentially for setup activities.

For my published firmware, the serial input is ignored. That’s not to say that serial input is excluded from being used, but the setup needs were not there, so there is nothing implemented in my firmware to receive serial data or commands.

The firmware will probably work on an AtMega 2560. I built it uploaded it to a standalone Arduino Mega that I recently purchased. However, the unit wasn’t connected to any real energy monitoring inputs, so it was not feasible to test in detail.

Craig

13

We were working on something similar to this a few years ago and advised against it. We ended up going with a system that samples every 1sec, which was still a major increase in sample rate.

I’m following this thread.

14

Great work! I am using this on an Arduino Nano that is connected to an emonESP which is posting the data via MQTT to emoncms successfully.

Is there any way to exclude CT inputs from the _enac total? I currently only have connections on A0(voltage), A1 (ct1), A2 (CT2). I am currently getting bogus readings on the un-used inputs (A3,A4) which I would prefer to not make it to the total.

15

I must point out that emonLibCM can have any or all current inputs selected at will as part of the configuration.

16

@WhichWayWazzit, thanks for your report. It’s the first report from someone other than me!

The firmware is intended to automatically determine which CT inputs are active. It polls them one at a time at startup for a few seconds to see which inputs have signal. How do you have your CTs connected to the unit? The standard emonTx3 has a pull-down resistor connected when the 3.5mm jack is disconnected, so the resulting signal is zero. The firmware notices the lack of signal.

If you don’t have a pull-down on some channels, then yes you will get undefined signals from the ADC on those inputs.

Do you have the capability of building the firmware yourself? I can make an improvement to ignore certain inputs, but since there are a lot of possible combinations of inputs I can’t build firmwares for all possibilities.

Craig

17

Thanks for the suggestion! I am quite new to electronics but will give the pull down a try this evening. This gave me some more background reading materials and get now that the ADC is shared amongst all the analog pins and is muxed to the pinouts, because of this, we can see noise and other oddities as the unused pins are floating and often take similar readings as the nearby pins in use. A pull down resistor would allow the pins to stabilize near zero and the firmware would then deactivate them on startup.

I am likely in the minority here as I am using Arduino Nano. I also understand the emonTX have integrated pull down resistors into the CT input plugs that are active whenever pin is vacant (now I know why :slight_smile: ). I still think it would be handy to have a software solution to skip unused inputs if the user chooses. Would removing the inputs yield better sampling rates on the in-use pins if they didn’t have the poll the unused pins?

Yes I am able to build new firmware versions using arduino IDE. Thanks again!

18

OK, why don’t you try doing the pull-downs. I added an issue on github to add a software disable.

Faster sampling with fewer input channels? Naively yes, but in practice no. I did try this and it did not work as desired. In fact there are vestiges of the code in there to completely disable reading an ADC channel if it is inactive. The problem is that there is a fixed overhead of math to do every set of measured samples. Going much beyond my current setup resulted in needing a very large ring buffer of samples, just to keep up while the math is being done.

It could be done, but it would be a rather big rewrite.

19

Pull downs did the trick for the most part, IRMS2 and IRMS3 were reporting small-ish values (~0.15), therefore, I also set the ICAL value for the unused inputs to 0.00000.

20

@WhichWayWazzit, very nice, congratulations. This is rewarding for me as well since it is the first successful deployment outside of my own house! :slight_smile:

I do have on the list to have an “ignore” capability. I’m working on another all-consuming project just now, but will come back to this firmware.

Craig