Hi.
This is my scenario:
From my EmonPi I receive 2 inputs that are the total Wh exported and imported but are only updated at 0, 15, 30 and 45 minutes. Obviously this input resets when the Pi or the 328p are rebooted.
Presently in Portugal (in recent cases) the energy imported and exported is averaged every 15 minutes that is why I implemented this feature in my EmonPi firmware.
In EmonCMS I receive the two feeds and want to record them to another ones that should be exactly the same until the first Pi/328p reset and then continue to increment normally the new values that now stated from zero.
Example:
Feed Import_Wh_Monitor : logs the value that comes from the Pi as is;
Feed Import_Wh : is to log all even after 328p resets;
At start both are 0 Wh and after a day they are at 10505 Wh
The Pi resets and the the feed Import_Wh_Monitor come back to 0;
Next time Import_Wh_Monitor updates to 110 Wh and at this time the Import_Wh should become 10615 (10505 + 110) Wh.
I’ve tried “Wh Accumulator”, but it seems to only add changes up to 80Wh - That’s strange because I’m pretty sure that I’ve seen values that were not recorded that were minor than others that were recorded.
Then I tried the “Total Pulse Count to Pulse Increment” that I think should have a similar behaviour despite we are not dealing with pulses, but if we consider that 1000 pulses would be a KWh a pulse is a Wh.
I also tried the “Rate of Change” but is misses changes and in one case didn’t even log anything (See Picture).
What is the best way to accomplish what I want to do?
PS: Later I also want to separate this increments with schedules that separate the import/export based on time of the day and I believe that will be another trouble, but that is a problem to deal with when this one is solved.
Cheers.
Picture (it is almost impossible to have a zero balance between import and export, so there must be a spike every 15 minutes - AAA is the Rate Of change of Import and BBB of the Export):
Looking at the code (the file is //var/www/emoncms/Modules/process/process_processlist.php) Wh Accumulator appears to be what you want (and if my memory is correct, they were introduced exactly for the reason you describe: to ensure that the accumulated value in emonCMS does not return to zero when the accumulating source data is reset.
I cannot see a reason for the 80 Wh limit you think is there. If your daily use is 10 kWh, and that is over a 16-hour day, the average is ~600 W, not 60 kW. On a 230 V supply, the “standard” 100 A c.t. will only read 23 kW! So I don’t believe that is causing the 80 Wh problem.
As I read the code, those processes do this:
Wh Accumulator
The input value and the Feed are expected to be in units of watt-hours. There is a limit of 60 kW average during the interval between increments (calculated from energy_change ÷ time_change). Subject to that, the Feed specified is incremented by the input value, the value passed on is the value written into the feed.
kWh Accumulator
The input value and the Feed are expected to be in units of kilowatt-hours, otherwise as above.
Total Pulse Count to Pulse Increment passes onwards the difference between the present value and the last value recorded in the feed, and stores the present value in the feed.
Rate of change passes onwards the difference between the present value and the last value recorded in the feed, and stores this (the difference) in the feed.
Hi.
You are confirming everything I thought, yet it doesn’t seem to work correctly.
I will now reset the destination feeds and configure Wh Accumulators.
Later I will post a picture of the resulting graphs.
Cheers.
When you wrote that, did you mean that the “emon” front end processor (the ATMega 328P that measures the voltage and current and calculates power) is reporting its data to emonHub and so to emonCMS every 15 minutes, and you have set the Feed Interval in emonCMS also to 15 minutes?
The reason I ask is those process steps in emonCMS are executed each time the input data arrives.
Hi.
I didn’t completely understood what you meant with “front end processor”, but just as a clarification, my EmonPi firmware (programmed into the 328p) is completely custom that was mainly based on a firmware from calypso_rae (that has energy diversion with triac) and presently there are so much changes I made that only some base coding may be recognized by him.
I never published my code because he said I wasn’t allowed do share it.
In the near future I intend make my own EmonPi board with some changes I want (some of them already botched on my EmonPi).
Cheers.
This is a board I designed related with the energy monitoring system:
When you wrote “From my EmonPi” that automatically means that you are using an emonPi from The OEM Shop, and until you tell us differently, it runs the standard software. If that is not the case, you must not call your own design an emonPi, you cannot expect us to know anything about it and you cannot expect accurate advice that suits your circumstances.
I think you are defining things differently to what I understand. Only a very small part of the software, that which reads the analogue values, calculates the voltage, currents, powers and energies is located in the '328P. That is what I call the “front end” - the beginning part of the process that finishes with numbers in a database and graphs on a screen. All the remaining processes commonly run in a Raspberry Pi, but it can be almost any computer.
I think your problem might be related to timing differences between when your '328P measures and reports the power and energy, and when the values are processed by emonCMS. You need to look at that area to see why the process steps are not working as you expect.
Hi.
This is the packet that my 328p firmware sends to the EmonHUB, and is subsequently sent to EmonCMS in a linux server:
typedef struct {
int power[NUM_CTS]; // Grid Power, Solar Power, CT3 (Diverted)
int powerTotal; // Grid + Solar Power
int Vrms; // x100 no EmonHUB
unsigned long noSamples;
int lowNoSamples;
int minEnergyState;
int maxEnergyState;
int temp[MaxOnewire];
unsigned long Wh_Import;
unsigned long Wh_Export;
byte triacOn;
unsigned int permanent=20001; // 20000 Temperature Regulation; 20001 Spare energy management only; 10000>19999 On of demand
unsigned int timecontrol_worse=0;
unsigned int timecontrol_best=65535;
int variable[10];
int totalEnergyReport;
} PayloadTX; // create JeeLabs RF packet structure - a neat way of packaging data for RF comms
The variables that are important for this situation are Wh_Import and Wh_Export, this variables keep their values and are only updated in the 328p software every 900 seconds.
This two variables always increase or stay equal based on other accumulators that are reset based on what is explained in the next paragraph.
There is a cron in the Pi that sends a message to the 328p firmware at 0,15,30 and 45 minutes
to synchronize the seconds count, but the 328p firmware also reset after a few seconds if that message is not received before for any reason.
Even if the 328p is reset, only the energy since the last checkpoint (0,15,30 or 45 minutes) is lost, never more than 15 minutes of energy, but this variables reset to zero at that time and I need EmonCMS to recognise that situation and keep adding the new values to the old.
I don’t think this is a problem of timings, the feeds were configured to 5 seconds (as I normally do) despite that value not being updated with less than 15 minutes intervals.
And when I create data by typing values into a web page URL and sending it as JSON, I get this graph, which I think does exactly what you want. Yellow (E3-raw) is the number I typed in, Blue (E3-acc) is the feed written by Wh accumulator:
Ya, it seems to be working correctly, except one small detail, the values are small and the elapsed time is also small.
Here are some pictures of the input configuration the graphs since about 9 hours ago when I said I was going to reset de destination feeds and configure Wh Accumulators:
For Export:
As it is possible to see, only small “steps” are logged to the accumulator.
The only logged “step” in the Export graph was 83Wh.
And the highest logged “step” in the Import graph was the first one, of 86Wh.
Note the last one in the import at 22:00 is 92Wh and was not logged so the limit seems to be between 86 and 92Wh, at least for my time frame of 15 minutes.
When I have time I will also look into process_processlist.php to see if I can understand how it work.
And from the on-screen documentation for the Wh accumulator:
“This processor ensures that when the emontx is reset the watt hour count in emoncms does not reset, it also checks filter’s out spikes in energy use that are larger than a max power threshold set in the processor, assuming these are error’s, the max power threshold is set to 60 kW.”
There is your answer. 83 Wh over 5 s is 60 kW. You are executing the processes every 5 s,
so it sees the same value every 5 s and then when it sees a big step every 15 minutes, it recognises it as an unwanted spike and filters it out.
The only problem here is, I can’t see in the code where the 83 Wh is allowed through into the Feed, but the fact that 83 Wh over 5 s is exactly the power limit is too much of a coincidence for this not to be the problem.
For a possible solution, can you send the 5 s data and the 15 minute data separately, using two separate NodeIDs and only sending the energy data when it changes, so that the input processes only execute every 15 minutes - then you will not hit the 60 kW limit because the power will be calculated from the energy change over 15 minutes rather than over 5 s. That’s not possible with JeeLib but it is with my RFMTxLib.
Hi.
Based on what you said, I created new Feeds that have 15 minutes interval and now I have the same problem I had before and a new one: the timestamps of this feed are not synchronized with the 0, 15, 30 and 45 minutes of the RTC. In fact I think that was the reason I did create the feeds with 5 seconds, that way the maximum sync error would be 5 seconds.
So the 5 second logging is also not the problem.
The first “step” that wasn’t logged is 91Wh.
Your last paragraph seems to imply that even if I record at 15 minutes, if the data is sent in 5 second intervals, the problem will still remain as it is, is that true?
This time I made the graph with the same intervals in the left and right axis so it is easier to compare.
I suspect there is a way of extracting the quarter-hourly data once the raw data is in a feed inside emonCMS, but I think that is not going to work with your design as you have it at present. In any case, I do not know emonCMS well enough to be able to tell you what to do.
What is this RTC? Having two clocks in a system is a well-known problem with the emonTx/emonTH and emonCMS and is the reason why those always send their data is just less than the feed rate in emonCMS.
emonCMS runs on Internet time, so if you were to send a message to your '328P energy monitor to tell it to send it’s accumulated energy values immediately, your synchronisation problem would disappear.
That is how I read the code - you can look for yourself if you wish. I happen to have it in my editor:
public function wh_accumulator($feedid, $time, $value)
{
$max_power = 60000; // in Watt
$totalwh = $value;
global $redis;
if (!$redis) return $value; // return if redis is not available
if ($redis->exists("process:whaccumulator:$feedid")) {
$last_input = $redis->hmget("process:whaccumulator:$feedid",array('time','value'));
$last_feed = $this->feed->get_timevalue($feedid);
$totalwh = $last_feed['value'];
$time_diff = $time - $last_feed['time'];
$val_diff = $value - $last_input['value'];
if ($time_diff>0) {
$power = ($val_diff * 3600) / $time_diff;
if ($val_diff>0 && $power<$max_power) $totalwh += $val_diff;
}
$padding_mode = "join";
$this->feed->insert_data($feedid, $time, $time, $totalwh, $padding_mode);
}
$redis->hMset("process:whaccumulator:$feedid", array('time' => $time, 'value' => $value));
return $totalwh;
}
I consider RTC as “Real Time Clock”, not as in the RTC chip, every device is internet time synced, even the 328p can be considered to be because of the cron that the Pi has to sync it every 15 minutes.
This idea is pretty easy to do, but I don’t like it very much because if a packet is lost there will only be another 15 minutes later…
I was already looking at that code and I’m thinking of changing it for my situation, maybe create a clone of this process and remove the limit that is causing the problems.
