How to feed data from alternate hardware to emoncms for storage?

I have an experimental Raspberry Pi add on board which is a 10 channel A/D converter for handling 1 AC/AC input and 9 current clamps. I would like to generate the data from this in a form suitable for feeding into Emoncms for storage and analysis. Is there a definition of the interface between the measurement hardware and the rest of the system that I could use so I can write the code to do this. I understand there may be a upper rate threshold of 1 sample per second, is this per clamp, as I would like to feed in 9 current and 1 voltage measurement per interval. Is this a hard limit or is it possible to go faster for example every 100ms or even every 1ms?

You can look at the code for an emonTx, to see what’s presently being sent, or you can look at the emoncms.org inputs API to see what you’re allowed to send (several formats are available). The rate limit to emoncms.org applies per block of data. There’s no limit if you run emoncms on your own RPi, other than the processor etc speed.

I’d seriously question whether 1 ms data rate is meaningful - that’s well less than 1 cycle of mains. All our software calculates rms current and voltage and average power in real time, but reports the averages at prescribed intervals - our default is 10 s, but unless you need to analyse for system disturbances, I’d question the need for data at intervals of less than a cycle, and probably less than several cycles.
Hopefully a stupid question - have you worked out how much storage you’ll need at whatever data rate you choose?

Robert, thanks for the prompt response. Any clues about where in the code? I had looked at the inputs API, but I wasn’t sure if that was the right place, so thanks for confirming that.

My interest is disaggregation, and I have previously collected over a year’s worth of data at 1 second granularity which runs at at 2 Mb per month, so we are talking in gigabytes here. however, I am unlikely to go to a 1ms rate except to look closely at device signatures, 1s is generally OK but you do get a fair number of overlapping events which makes life more difficult so 100ms might be useful. I agree anything less that a cycle brings its own problems, and I am pretty sure that the Raspberry Pi would hit a bottleneck somewhere.

The data structure is called PayloadTX, so the definition of that / declaration of emontx is what to search for.

typedef struct {int power1, power2, power3, power4, Vrms, T1, T2, T3; } PayloadTX;
PayloadTX emontx;

That, (preceded by the node ID if going by radio) is sent as a byte stream to emonHub (local emonCMS), which interprets it according to the node definition in emonhub.conf

Otherwise, for a serial connection, a search for “Serial.print(“ct1:”)” should get you to the right area.

I wasn’t to guess that. In that case, rather than storing all the data, I’d suggest storing the last few seconds on a per sample basis and immediately analysing that for changes. When anything is detected, then you can go ahead and store that buffer for the duration of the disturbance, plus a bit either end, for later analysis as necessary, because as I understand it, inrush etc can be a useful parameter in distinguishing various appliances & loads. Otherwise you’re probably only interested in steady-state readings every few seconds. emonCMS isn’t going to be suitable for the per-sample data, you’ll probably need to design your own database for that, but it will be suitable for the steady-state averages.

Thanks for the pointer, that’s helpful.
On the method, I don’t think that works for several reasons:

  • The aggregated data stream is almost never stable, particularly in a house with fridge, freezer, computers, router etc.
  • Start and end points of a device can be widely separated in time.
  • There is a lot of overlapping and stacking of device power signatures

You are correct that having real and reactive power readings is very useful to create a two dimensional space such as cluster diagrams for identifying devices, There are many academic papers on the subject, some of them have covered high frequency measurement for identifying devices, but at the time I didn’t have any high frequency data to look at so I largely ignored them.

Most of the academic stuff is focused on rates of 1 sample per second or slower, I suspect that is because that is where the bulk (power utility) data is. Personally I found that it was necessary to go to 1s data to separate out what happening, and even that is a bit slow. Think how many things get turned on/off each time you enter or leave room, humans tend to cluster the events!

I do already have my own database & graphing, and I can import my existing data into NILMTK. Emoncms looks like a reasonable option for looking at trend information etc. I just wanted to explore how far I could push the technology.

Do you know if there are others experimenting with disaggregation on the forum?

None at present spring to mind. However, there’s some discussion in the “Archived” forums, so a search there might turn up some useful leads.

This provided useful information

As did this

If your addon board is directly connected to the PI then you can also look to write code to read from your addon board and write over the values to MQTT - emoncms will then pick those up automatically.

It is, so that sounds like a great idea. Any guidance or examples I can follow?

Since I prefer working in Java a good option seems to use a Java MQTT client such as https://eclipse.org/paho/clients/java/ then use it something like:

MqttClient adcClient = new MqttClient(“tcp://127.0.0.1:1883”, “ADC_Client”, persistence);
MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);
adcClient.connect(connOpts);
MqttMessage message = new MqttMessage(content.getBytes());
message.setQos(qos);
adcClient.publish(“emon/myADC”, message);
…
adcClient.disconnect();

The next question would be what format would the message content take?

@stuart Having read up on this a bit, if I write code to publish data from my ADC shield to Emoncms via MQTT, I don’t think I need Emonhub. Is this correct?

Yes think that’s correct

I now have data available and can feed it in any format via MQTT see examples below (noise input only). What is the format of the MQTT message data required by Emoncms?

1.017 VA
0.113 W
0.545 VA
0.372 W
0.573 VA
0.431 W
0.857 VA
0.166 W
0.717 VA
0.739 W
0.872 VA
0.098 W
1.621 VA
0.435 W
1.052 VA
0.406 W
0.528 VA
0.494 W
0.539 VA at 27-Aug-2017 22:34:48
0.086 W at 27-Aug-2017 22:34:48
1.170 VA at 27-Aug-2017 22:34:49
0.300 W at 27-Aug-2017 22:34:49
1.059 VA at 27-Aug-2017 22:34:50
0.735 W at 27-Aug-2017 22:34:50
0.507 VA at 27-Aug-2017 22:34:51
0.132 W at 27-Aug-2017 22:34:51
0.464 VA at 27-Aug-2017 22:34:52
0.260 W at 27-Aug-2017 22:34:52
0.788 VA at 27-Aug-2017 22:34:53
0.345 W at 27-Aug-2017 22:34:53
0.335 VA at 27-Aug-2017 22:34:54
0.129 W at 27-Aug-2017 22:34:54
0.451 VA at 27-Aug-2017 22:34:55

This an example of what is passing through MQTT

pi@rpi3gjw:~ $ mosquitto_sub -t /emon/PMon10/Whole_House/Voltage
238.488
238.743
238.050
238.607
238.441

I am getting this error in emoncms

2017-08-28 09:59:36.106|ERROR|phpmqtt_input.php|No matching MQTT topics! None or null inputs will be recorded!



Any ideas about what I am doing wrong?

I am now getting some data through to phpmqtt_input.php which I can see in the log (level set to info) see below

2017-08-30 13:08:47.679|INFO|phpmqtt_input.php|emon/PMon10/Lounge_End_Plugs RealPower -0.189
2017-08-30 13:08:47.693|INFO|phpmqtt_input.php|emon/PMon10/Whole_House ApparentPower 50.369
2017-08-30 13:08:47.695|INFO|feed_model.php|insert_data() feedid=8 updatetime=1504098527 feedtime=1504098527 value=ApparentPower 50.369 arg=
2017-08-30 13:08:47.696|INFO|PHPFina.php|post() id=8 timestamp=1504098527 value=0 padding=
2017-08-30 13:08:47.697|INFO|feed_model.php|insert_data() feedid=8 updatetime=1504098527 feedtime=1504098527 value=0 arg=join
2017-08-30 13:08:47.697|INFO|PHPFina.php|post() id=8 timestamp=1504098527 value=0 padding=join
2017-08-30 13:08:47.700|INFO|phpmqtt_input.php|emon/PMon10/Whole_House RealPower -3.505
2017-08-30 13:08:47.703|INFO|feed_model.php|insert_data() feedid=8 updatetime=1504098527 feedtime=1504098527 value=RealPower -3.505 arg=
2017-08-30 13:08:47.703|INFO|PHPFina.php|post() id=8 timestamp=1504098527 value=0 padding=
2017-08-30 13:08:47.704|INFO|feed_model.php|insert_data() feedid=8 updatetime=1504098527 feedtime=1504098527 value=0 arg=join
2017-08-30 13:08:47.705|INFO|PHPFina.php|post() id=8 timestamp=1504098527 value=0 padding=join
2017-08-30 13:08:47.714|INFO|phpmqtt_input.php|emon/PMon10/Upstairs_Lighting ApparentPower 1.448
2017-08-30 13:08:47.722|INFO|phpmqtt_input.php|emon/PMon10/Upstairs_Lighting RealPower 0.067
2017-08-30 13:08:47.741|INFO|phpmqtt_input.php|emon/PMon10/Outside_Lighting ApparentPower 0.965
2017-08-30 13:08:47.746|INFO|phpmqtt_input.php|emon/PMon10/Outside_Lighting RealPower 0.063
2017-08-30 13:08:47.759|INFO|phpmqtt_input.php|emon/PMon10/Downstairs_Lighting ApparentPower 1.053
2017-08-30 13:08:47.767|INFO|phpmqtt_input.php|emon/PMon10/Downstairs_Lighting RealPower -0.036
2017-08-30 13:08:47.781|INFO|phpmqtt_input.php|emon/PMon10/Extension_Lighting ApparentPower 0.760
2017-08-30 13:08:47.788|INFO|phpmqtt_input.php|emon/PMon10/Extension_Lighting RealPower 0.016
2017-08-30 13:08:47.802|INFO|phpmqtt_input.php|emon/PMon10/Cooker ApparentPower 15.785
2017-08-30 13:08:47.809|INFO|phpmqtt_input.php|emon/PMon10/Cooker RealPower -1.594
2017-08-30 13:08:58.635|INFO|feed_model.php|EngineClass() Autoloaded new instance of ‘PHPFina’.
2017-08-30 13:08:58.636|INFO|PHPFina.php|get_data() feed=9 st=1504098228 end=1504098548 int=1 sk=1 lm=1 pts=1251 st=1504097277
2017-08-30 13:09:01.629|INFO|feed_model.php|EngineClass() Autoloaded new instance of ‘PHPFina’.
2017-08-30 13:09:01.630|INFO|PHPFina.php|get_data() feed=9 st=1504094931 end=1504098551 int=4 sk=1 lm=1 pts=1251 st=1504097277

I am not certain that the message Tag Key Value, example:
emon/PMon10/Whole_House ApparentPower 50.369
is being handled, as no non-zero values appear.

Any guidance would be appreciated.

further info, the feeds look like this


but the data looks like this:

OK I think I finally have something that works!
MQTT

  • topic = “emon/PMon10/Whole_house”
  • payload = “99.99”

Note only 3 levels in the topic, 4 doesn’t seem to work, no leading or trailing “/” or " ". " " is legal but is confusing as taken as part of the name.

and finally …

Hurray!

1 Like

Hello,

Did you managed to identify the consuming devices based on their signatures?
I want to implement something like this, use the emon to collect the data and then process it on a more “powerfull” central server/database. Is this the way you’re doing this?

Do you know if the kind of data collected by Emon is similar to those used by sense or similar projects capable of identify the devices?

Thanks,

Hi Bruno,
No, it’s a non trivial problem for a whole house, which is why I wanted to monitor each circuit on the consumer unit separately. Unfortunately it isn’t going to be easy to fit the 9 current clamps to my consumer unit without using an electrician as the wires are not accessible. There are many academic papers on this, the most successful approach currently seems to be to use AI, but for that you need a lot of data and computing power, so not really an option for individual use.