Could you provide me with a little info as to what you have setup to report to the emonpi?
Even a screenshot of the expanded input menu will be helpful.
Sorry for the delay in getting back to you. The aim is to start monitoring equipment and spot patterns in usage of the machine. For example a washing machine or a motor, then use either machine learning or programming logic work out the state of the machine. 10 seconds is not enough, we need at least 1 second and probably more. The picture on this original post has the settings available from the menus for feeding information and the lowest is 5 seconds. I need more. Hope that helps.
Thanks
In order to be of assistance, I would need to know what is feeding data
into EmonCMS. The easiest way would be if you could post a screenshot of
your input menu showing where all your data is being posted from.
Since these units are highly customizable I have no way of telling what
your setup is, thus no way to tell if such a data rate is actually
possible.
Assuming you have an emonPi you would need to edit the firmware sketch so that it reports it’s data each second, once that is compliled and installed you could use a phptimeseries (variable interval) feed instead of a phpfina (fixed interval) feed which will store data with timestamps down to 1sec resolution.
phptimeseries data uses just over twice the disc space of phpfina (9byte v 4byte per datapoint), at 1s resolution that will use disc space at 9 bytes per second per feed.
I still do not know if this will give you the accuracy you need though, due to the timestamps being converted to ints in the emoncms inputs, a swing of 1 sec is only 10% at 10secs, but it is effectively almost +/- 100% when you are using a 1 sec update rate, for example 1519078074.000 to 1519078074.999 would be seen as the same timestamp (0 secs) despite actually being 0.999 secs apart and 1519078074.999 to 1519078076.001 would be seen as 2 full secs despite only being 1.002 seconds apart.
Thanks for getting back to me so quick. I only have one input at the moment which is Services as can be seen below:
We are using EmonPi. Is there any reason I can’t say go to 10 readings a second or more by editing the firmware sketch? Is there anyway to prevent them converting to ints easily?
Would it be easier to amend the emon library to get faster updates
I’ve just found this from looking at the learning section of the website
Another point about the sketch in the emonPi - are you using it as a RFM receiver to handle the data from other nodes (not counting itself, that is)? If you are, you might not be able to go as fast as 10 per second, as the sketch also services the RFM69 receiver, and it needs time to do that. If you’ll never use it to receive other nodes, you will probably be best to remove the RFM parts completely. And if you do try to go to 10 samples per second, you’ll need to cut the measuring period from our standard 200 ms and make it something like 2 or 3 cycles (40 or 60 ms).
Thanks. No we don’t use radio. It’s not very reliable in our environment (lots of steel components in a factory floor). Would i be better using the emon library with network shield than use the existing emon pi. I have all the components and can then send the data using mqtt via the lan.
I doubt that you would see much difference. The emon part uses the serial interface to send the data to the Pi part, and you’d do much the same if you had an Arduino and an Ethernet Shield. But if the Ethernet Shield required any degree of processing power from the Arduino, you might in fact be worse off. I’m not an Arduino expert so I’m only suggesting things you need to check.
Your real problem if you need 10 samples per second is the way emonLib works. I’ve had the upcoming emonLibCM reporting serially at that rate when running on an emonTx, so it should run on the ATMega in the emonPi.
Thanks for the quick response. What im thinking at the moment is to use the emonlib library and basic arduino uno as it strips back to basics and just serial print to screen. Once thats working, then go ahead and starting adding the network shield and mqtt library. That way i know which bit is breaking and which bit isnt. Do you have any comments on what ive suggested? e.g. good idea/bad idea and why 
That’s fine, you can do that and experiment with the timings of both the sampling and of the serial prints as if the payloads are too big the serial port could become the bottle neck if you go too fast.
If you use the standard serial output format of nodeid val1 val2 val3 etc and add a config to the emonhub.conf you can push this data into emonhub via a USB port and it will give you the option to upload to emoncms.
This is where you might introduce more issues, you cannot have accurate data by timestamping it on arrival over a network connection at these sample rates, if you were passing data every 60s, a few seconds either way makes very little difference, so you need to add a RTC and timestamp each packet before sending it over the network.
I do not think you will get reliable sub 1sec interval data if your arduino uno is trying to timestamp and publish each frame as well as sampling.
If you stick to a simple serial connection and use all available resource for sampling you may achieve your goal easier.
That’s beyond my pay grade, I too would like emoncms inputs to use floats for processing timestamps, primarily to make existing posting intervals more accurate, but yes it would allow for shorter sample periods too. I haven’t attempted to work out how this might be implemented as I assume it to be quite involved, hope fully I’m wrong. This is not the first time this has come up.
I’ve been tuned out of this discussion because it didn’t seem like anything IoTaWatt would be appropriate for. Now that the data export handlers are going async, this might be something that could be provided in the upcoming MQTT service for IoTaWatt.
The first issue is quality of data at 1 second intervals. A fully loaded IoTaWatt would only have two samples per channel each second. A partially loaded unit on the otherhand, say 7 channels (8 with voltage) would average about four samples per second (five at 60Hz). So I think the data might be viable.
The second issue is whether the data export is strictly real-time or delivered as in the Emoncms or influxDB server support. If historical integrity isn’t an issue, and I don’t believe it is in these other solutions being discussed, then a real-time async MQTT stream could be produced at 1 second intervals with each data point based on 2-5 samples.
At this point, I’m just coming up to speed on the MQTT services that are in use with Emoncms. Can anyone elaborate on the QoS being used and specifically what circumstances might warrant QoS 1? Right now, I’m thinking that real-time QoS 0 would be pretty easy.
With respect to timing, the packets would probably go out at 1 second intervals +/- a few 10ms (50Hz), provided a connection can be maintained.
As an aside, the time issue for time series storage may be better accomplished with influxDB. It can easily handle ms resolution, and can reduce resolution over time to limit the storage needs over time.
Thanks. Good to know.
Thanks ill stick with a simple serial connection.
Do I need to stamp it on the Arduino? Can’t I just stamp it on either on the way into the pi or if send it to MQTT i can put into influx (a time series database) and that can stamp it? I’m looking for patterns in the electricity usage, a few seconds out wont matter too much providing its consistently out. Hope that makes sense.
Thanks for the post. I’ve not looked at IOTawatt before. Like I said in my previous post, im not sure accuracy (in terms of time) is the most important, just that it is consistent in its posting. For my purpose, im looking for patterns in usage and if its out a few milliseconds that’s fine. I currently send all data from emonpi to influx and display in grafana, i just need more resolution. I also find wireless not the best method in time critical solutions as wireless is more likely to be lost and higher latency ect unless it is time stamped and buffered on the arduino (or esp), and then passed to say influx database with the data and corresponding time.
OK Alex, I can see you have some pretty specific requirements. FYI/ IoTaWatt is currently only 5 second resolution, but it does upload to influxDB with timestamp over WiFi. I suppose a $20 AP right next to it would get you to hardwire pretty reliably - pretty much the opposite of what most people are trying to do, but I can see your reluctance to use WiFi.
Have fun plugging that Arduino stuff together. I’m not willing to do anything custom for your project, but I do think that the IoTaWatt has the potential to do what you want, and if the MQTT feed ends up being a real-time feed, then I believe it will be what you are looking for.
Good luck with your project.
Not if you are just using serial, if you use emonhub it will timestamp on arrival, those timestamps will be to the nearest millisecond, I expect the latency of a constant serial stream to be fairly consistent and pretty short too, but I guess that also depends on how hard you push the MCU too.
It’s only if you use a network connection you should need to timestamp before sending so as to be sure any network delays are not affecting the timestamps. When working at these speeds it won’t take much to distort the picture. I am talking about the frame to frame timing, the actual time/date is of little consequence other than identifying what was being monitored at any given time.
I do not think it would take much to get emonhub to post to a influxdb either. It’s something I keep thinking about but never get the chance to explore. I like influx/grafana and think it is probably the ideal tool for the task at hand.
Thanks for all your comments. I’ve downloaded the emon.lib and just running the code on Arduino Uno to see what throughput I can get (I don’t have the CT clip attached yet) . I am only getting 4 readings a second, ideally I want at least 8 but would be happy with 1 or 2 more readings:
Time in mlsec: 245, Time in sec: 0, 19161.06 83.31
Time in mlsec: 489, Time in sec: 0, 9249.77 40.22
Time in mlsec: 736, Time in sec: 0, 4292.21 18.66
Time in mlsec: 987, Time in sec: 0, 2712.41 11.79
Time in mlsec: 1232, Time in sec: 1, 2394.64 10.41
Time in mlsec: 1481, Time in sec: 1, 2288.04 9.95
Time in mlsec: 1731, Time in sec: 1, 2307.08 10.03
Time in mlsec: 1979, Time in sec: 1, 2561.42 11.14
Time in mlsec: 2228, Time in sec: 2, 2443.51 10.62
Time in mlsec: 2477, Time in sec: 2, 2496.14 10.85
Time in mlsec: 2726, Time in sec: 2, 2494.32 10.84
Time in mlsec: 2976, Time in sec: 2, 2391.45 10.40
Time in mlsec: 3226, Time in sec: 3, 2494.75 10.85
Time in mlsec: 3475, Time in sec: 3, 2420.95 10.53
Time in mlsec: 3725, Time in sec: 3, 2483.36 10.80
Time in mlsec: 3974, Time in sec: 3, 2462.69 10.71
Time in mlsec: 4224, Time in sec: 4, 2483.08 10.80
Time in mlsec: 4472, Time in sec: 4, 2419.62 10.52
Time in mlsec: 4722, Time in sec: 4, 2417.25 10.51
Time in mlsec: 4971, Time in sec: 4, 2505.22 10.89
Here’s the code I am running (I will take the time serial prints out after testing as i know they take processor cycles too):
// EmonLibrary examples openenergymonitor.org, Licence GNU GPL V3
#include "EmonLib.h" // Include Emon Library
EnergyMonitor emon1; // Create an instance
unsigned long time; // to store time
void setup()
{
Serial.begin(115200);
emon1.current(1, 111.1); // Current: input pin, calibration.
}
void loop()
{
double Irms = emon1.calcIrms(1480); // Calculate Irms only#
//double Irms = emon1.calcIrms(1480/2); // Calculate Irms only
time = millis(); // Returns the number of milliseconds since the Arduino board began running the current program
Serial.print("Time in mlsec: ");Serial.print(time);Serial.print(", "); // time in millseconds
Serial.print("Time in sec: ");Serial.print(time/1000);Serial.print(", "); //time in seconds
Serial.print(Irms*230.0); // Apparent power
Serial.print(" ");
Serial.println(Irms); // Irms
}
I’ve removed the delay from emonlib.cpp as it didnt seam to be adding any value my purpose. See below:
void EnergyMonitor::serialprint()
{
Serial.print(realPower);
Serial.print(' ');
Serial.print(apparentPower);
Serial.print(' ');
Serial.print(Vrms);
Serial.print(' ');
Serial.print(Irms);
Serial.print(' ');
Serial.print(powerFactor);
Serial.println(' ');
// delay(100);
}
The only other way i can see to increasing the readings is either :
-
to decrease the sample rate - which is this line -
double Irms = emon1.calcIrms(1480)but that will reduce quality of the readings, right??? can anyone expand on what this means?
p.s. i have halfed IRMS to 740 and i do get twice as many readings. -
or use a quicker processor than the uno, maybe the Duo?
-
or is there another way?
Again, thanks for your time time 
If you’re prepared to wait a little longer, you can have real power (not just current) as I wrote earlier:
No, it does not decrease the sample rate, it is the number of samples in the batch that is averaged and reported. You should choose that number so that you have a whole number of mains cycles. When I last tested the sample rate of emonLib, I got calcIrms( ) reading approx. 5588 current samples per second. So the smallest number you should have is 20 ms worth, or 111.76. You can’t have a fraction. Your nearest is 112 samples, covering just over 1 cycle, so your reading will be wrong. How much wrong depends on where on the cycle you start and finish, and you cannot control that in calcIrms( ). 447 will give you almost exactly 4 cycles, so that’s the smallest number I recommend.
Here’s a similar output from IoTaWatt with 20 samples per second. The data varies by less than .5% over the 3 seconds. Each sample is one AC cycle. It is measured at 60Hz, so at 50Hz the sample count would be more like 16+.
When I say one AC cycle, I mean exactly one AC cycle +/- about 0.1% of the cycle. I think the math for jumping in arbitrarily and sampling for a fixed period of time is flawed for two reasons:
-
The frequency varies too much to reliably predict when a whole number of cycles has been sampled. UK frequency for the last hour
-
It matters where you jump in. If you jump in at a peak of the cycle, any short/extra samples due to frequency/sample-rate calibration will result in a much larger error than if the timed sample period begins on or near the zero crossing.
So this data was generated by literally adding one printf statement to the IoTaWatt firmware. Although I output real power and rms voltage, power factor, rms current and of course apparent power are also available to be included in the print statement.
These values are also ms-time-weighted and accumulated, so that the average over any interval, measured in ms, can be easily obtained.
There is a process that already outputs to influxDB - with timestamps.
You may have noticed that I posted this output without explanation. Just to pique your interest. But the moderators insisted on this explanation. Hope it widens the scope of your project.
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.8, vrms 118.1
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.4, vrms 118.0
Seconds: 11, real power: 123.4, vrms 118.0
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.6, vrms 118.0
Seconds: 11, real power: 123.4, vrms 118.0
Seconds: 11, real power: 123.4, vrms 118.0
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.7, vrms 118.0
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.6, vrms 117.9
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.5, vrms 118.0
Seconds: 11, real power: 123.4, vrms 118.0
Seconds: 11, real power: 123.3, vrms 117.9
Seconds: 11, real power: 123.3, vrms 117.9
Seconds: 12, real power: 123.6, vrms 118.0
Seconds: 12, real power: 123.3, vrms 117.9
Seconds: 12, real power: 123.3, vrms 117.7
Seconds: 12, real power: 123.3, vrms 117.9
Seconds: 12, real power: 123.7, vrms 118.1
Seconds: 12, real power: 123.4, vrms 118.0
Seconds: 12, real power: 123.3, vrms 118.0
Seconds: 12, real power: 123.5, vrms 118.0
Seconds: 12, real power: 123.5, vrms 118.0
Seconds: 12, real power: 123.5, vrms 118.0
Seconds: 12, real power: 123.4, vrms 118.0
Seconds: 12, real power: 123.5, vrms 118.0
Seconds: 12, real power: 123.5, vrms 118.0
Seconds: 12, real power: 123.5, vrms 118.0
Seconds: 12, real power: 122.9, vrms 118.2
Seconds: 12, real power: 123.4, vrms 117.9
Seconds: 12, real power: 123.5, vrms 118.1
Seconds: 12, real power: 123.4, vrms 118.0
Seconds: 12, real power: 123.7, vrms 118.1
Seconds: 12, real power: 123.7, vrms 118.1
Seconds: 13, real power: 123.9, vrms 118.0
Seconds: 13, real power: 123.7, vrms 118.1
Seconds: 13, real power: 123.6, vrms 118.1
Seconds: 13, real power: 123.5, vrms 118.0
Seconds: 13, real power: 123.7, vrms 118.1
Seconds: 13, real power: 123.5, vrms 118.0
Seconds: 13, real power: 123.5, vrms 118.0
Seconds: 13, real power: 123.7, vrms 118.1
Seconds: 13, real power: 123.4, vrms 117.9
Seconds: 13, real power: 123.3, vrms 118.0
Seconds: 13, real power: 123.6, vrms 118.0
Seconds: 13, real power: 123.7, vrms 118.1
Seconds: 13, real power: 123.6, vrms 118.1
Seconds: 13, real power: 123.6, vrms 118.1
Seconds: 13, real power: 123.6, vrms 118.1
Seconds: 13, real power: 123.7, vrms 118.2
Seconds: 13, real power: 123.6, vrms 118.1
Seconds: 13, real power: 123.5, vrms 118.0
Seconds: 13, real power: 123.7, vrms 118.1
Seconds: 13, real power: 123.5, vrms 118.0
Seconds: 14, real power: 123.6, vrms 118.1
I am interested in real power but its not required for this particular application. I’ve used the Arduino Due and with the batch size at 1480 (double Irms = emon1.calcIrms(1480); // Calculate Irms only) and we’re now getting 42 readings per a second which is great some sample readings below:
Time in mlsec: 4011, Time in sec: 4, 112.73 0.49
Time in mlsec: 4035, Time in sec: 4, 118.12 0.51
Time in mlsec: 4059, Time in sec: 4, 114.29 0.50
Time in mlsec: 4083, Time in sec: 4, 130.31 0.57
Time in mlsec: 4107, Time in sec: 4, 114.04 0.50
Time in mlsec: 4131, Time in sec: 4, 117.60 0.51
Time in mlsec: 4155, Time in sec: 4, 114.90 0.50
Time in mlsec: 4179, Time in sec: 4, 111.32 0.48
Time in mlsec: 4204, Time in sec: 4, 133.66 0.58
Time in mlsec: 4228, Time in sec: 4, 110.25 0.48
Time in mlsec: 4252, Time in sec: 4, 122.75 0.53
Time in mlsec: 4276, Time in sec: 4, 112.39 0.49
Time in mlsec: 4300, Time in sec: 4, 102.19 0.44
Time in mlsec: 4324, Time in sec: 4, 136.63 0.59
Time in mlsec: 4348, Time in sec: 4, 107.33 0.47
Time in mlsec: 4372, Time in sec: 4, 126.12 0.55
Time in mlsec: 4396, Time in sec: 4, 109.94 0.48
Time in mlsec: 4420, Time in sec: 4, 115.94 0.50
Time in mlsec: 4444, Time in sec: 4, 123.19 0.54
Time in mlsec: 4469, Time in sec: 4, 111.07 0.48
Time in mlsec: 4493, Time in sec: 4, 128.67 0.56
Time in mlsec: 4517, Time in sec: 4, 108.16 0.47
Time in mlsec: 4541, Time in sec: 4, 107.79 0.47
Time in mlsec: 4565, Time in sec: 4, 109.58 0.48
Time in mlsec: 4589, Time in sec: 4, 112.17 0.49
Time in mlsec: 4613, Time in sec: 4, 128.13 0.56
Time in mlsec: 4637, Time in sec: 4, 109.30 0.48
Time in mlsec: 4661, Time in sec: 4, 106.59 0.46
Time in mlsec: 4685, Time in sec: 4, 100.08 0.44
Time in mlsec: 4709, Time in sec: 4, 109.85 0.48
Time in mlsec: 4734, Time in sec: 4, 135.57 0.59
Time in mlsec: 4758, Time in sec: 4, 105.58 0.46
Time in mlsec: 4782, Time in sec: 4, 114.73 0.50
Time in mlsec: 4806, Time in sec: 4, 109.88 0.48
Time in mlsec: 4830, Time in sec: 4, 111.58 0.49
Time in mlsec: 4854, Time in sec: 4, 126.52 0.55
Time in mlsec: 4878, Time in sec: 4, 106.58 0.46
Time in mlsec: 4902, Time in sec: 4, 121.61 0.53
Time in mlsec: 4926, Time in sec: 4, 115.90 0.50
Time in mlsec: 4950, Time in sec: 4, 115.02 0.50
Time in mlsec: 4974, Time in sec: 4, 126.46 0.55
Time in mlsec: 4998, Time in sec: 4, 113.47 0.49
The Arduino code I’ve wrote publishes the data directly over the network to the MQTT server. I know the talk in this thread recommended to use the Arduino read and serial the data to the Pi for the publishing but is that required with the Arduino Due since its multi-core and way more powerful? I would appreciate any thoughts on the subject
I’ve coppied the code for your perusal :
// EmonLibrary examples openenergymonitor.org, Licence GNU GPL V3
/*
take electricity readings from A1 pin and post to mqttTo Do:
- set to get DHCP address
*/
#include <SPI.h> // SPI required for network
#include <Ethernet.h> // eithernet library for
#include <PubSubClient.h> // pubsubclient library
#include “EmonLib.h” // Include Emon Library
EnergyMonitor emon1; // Create an instance// Update these with values suitable for your network.
byte mac = { 0xDE, 0xED, 0xBA, 0xFE, 0xFE, 0xED };
IPAddress ip(172, 16, 24, 99); // sets the ip address of this device
IPAddress server(172, 16, 24, 11); // mqtt server on factory vlanconst char * EquipmentID = “equipment1”; //equipment ID of the machine
const char * FullAsset = “equipment1”; //full name of the machineString temp_str; //Convert the float to the string
char temp[50]; // Temporary variable for charactors//Processes any returned messages
void callback(char* topic, byte* payload, unsigned int length) {
Serial.print(“Message arrived [”);
Serial.print(topic);
Serial.print("] ");
for (int i=0;i<length;i++) {
Serial.print((char)payload[i]);
}
Serial.println();
}EthernetClient ethClient; //create a instance of ethernet client
PubSubClient client(ethClient);unsigned long DueTime; // to store time
void reconnect() {
// Loop until we’re reconnected
while (!client.connected()) {
Serial.print(“Attempting MQTT connection…”);
// Attempt to connect
if (client.connect(“arduinoClient”)) {
Serial.println(“connected”);
// Once connected, publish an announcement…
client.publish(“outTopic”,“hello world”);
// … and resubscribe
client.subscribe(“inTopic”);
} else {
Serial.print(“failed, rc=”);
Serial.print(client.state());
Serial.println(" try again in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}void setup()
{
//network setup
Serial.begin(115200);
client.setServer(server, 1883); //sets connection for mqtt server
client.setCallback(callback); //sets call backEthernet.begin(mac, ip); // Sets mac and ip of the device
delay(1500); // Allow the hardware to sort itself out//emon setup
emon1.current(1, 111.1); // Current: input pin, calibration.
analogReadResolution(ADC_BITS); // sets resultion to 12bits
}
void loop()
{
double Irms = emon1.calcIrms(1480); // Calculate Irms only
//double Irms = emon1.calcIrms(740); // Calculate Irms only
DueTime = millis(); // Returns the number of milliseconds since the Arduino board began running the current program
Serial.print("Time in mlsec: “);Serial.print(DueTime);Serial.print(”, "); // time in millseconds
Serial.print("Time in sec: “);Serial.print(DueTime/1000);Serial.print(”, “); //time in seconds
Serial.print(Irms*230.0); // Apparent power
Serial.print(” “);
Serial.println(Irms); // Irms
Serial.print(” ");if (!client.connected()) {
reconnect();
}//convert irms to string to publish through pubsub
temp_str = String(Irms); //converting ftemp (the float variable above) to a string
temp_str.toCharArray(temp, temp_str.length() + 1); //packaging up the data to publish to mqtt…client.publish(FullAsset,temp); //logs power
client.loop();}
There’s no longer anything magic about the number 1480. It used to be the best approximation for 200 ms monitoring with an ATMega328P, but tweaks to speed up emonLib have made it wrong anyway. Until people say differently, we assume you’re using the emonTx/emonPi because most people here do.
All that I’d suggest is you try to make it sample for a whole number of mains cycles - if you can. You’ll minimise the “end effect” of including a part-cycle - and you don’t know where the cycle starts and ends, so the extra/missing part could be at peak amplitude or it could be around zero. That could well account for part of the variation from reading to reading that you’re seeing - just looking at those numbers, it’s about ±12%. I suggest you do as I wrote in another thread only a few minutes ago: 5 second intervals between EmonLib samples? - #4 by Robert.Wall