10000 kWh ceiling in emoncms?

Emoncms
1

I have noticed one of the sites I monitor has an issue with reporting the PV generation.

2017-04-17 (1).png1299×1000 56.6 KB

On Sat 15/4/17 around 15:50 the total recorded PV production on 2 separate feeds (phpfina & phptimeseries) reached 10,000 kWh’s and have not shown any increase since, despite plenty of generation. The “solar” power can be seen as expected on all three phases and as a summed total as usual. I also have a “power to kwh/d” feed and that is reading correctly.

As a quick test I added a second “power to kwh” process and feed immediately after the original one, the only difference was this being a new feed it is starting at 0 where as the original is currently at around 10,000 kWh. That started counting immediately and proved the values are being passed into the process ok, and the process is passing that same power value on to the next process ok.

When looking closely at the chart above it is possible to see some slight movement in the “flatline” and when zooming in that becomes more evident.

2017-04-17 (2).png1296×984 55.4 KB

It isn’t until you zoom right in that you can see there is actually a lot of movement but the value remains around 10,000 but changes with every new datapoint. This is the plot since 07:00am this morning til now (around 6hrs)

2017-04-17 (3).png1310×965 93.5 KB

during this time the solar power has been flowing and the kwh/d feed has climbing steady. As has the new “2nd” kwh feed I just added.

That line at full magnification is always smooth on all previous graphs and the values can only increase, there is no provision for a “power to kwh” feed to move backwards, and yet it does here!

I believe this to be a rounding issue somewhere, each time the value goes over 10K’ish it is rounded down or truncated and then the next value does the same, resulting in a value that is almost always the same but not quite, it just bounces around the 10k mark.

What is confusing me further is the fact that MySolar is able to detect the “PV today” correctly from the same feeds (solar & solar_kwh, both phpfina)

2017-04-17 (4).png1224×1010 66.8 KB

So I deduce from this that perhaps there is a correct value being held in Redis or memory and only the persisted values are being rounded or truncated etc.

I’m therefore a little unsure of trying a reboot or redis flush as that may lose data that can currently help locate the issue.

Any idea’s ??? Anyone ???

Running self-hosted v9.8.2 on a Pi+hdd.

2017-04-17 (5).png1319×900 60.3 KB

@TrystanLea or @nchaveiro can you shed any light on this at all? This site was the first of a 20 site pilot so I have 19 more sites a couple to a few months behind this one, but this being a smaller site (generation wise) the other sites will be hitting the 10k mark real soon!

I can provide more charts and data etc.

2

[EDIT:] I now think this is related to the use of floats not doubles in the calculations with emoncms.

5kw generation will translate to 0.006944 kWh’s each 5sec update

Once 9999.9999kWh is passed adding another digit to the value (ie 10,000) the floats accuracy to 7 digits raises the increment needed to register an increase by a factor of 10 and beyond the reach of the production power.

10000.00 + 0.006944 = 10000.006944 gets persisted as 10000.00 and that value is used again in the next calc and so on perhaps??

Does that sound possible?

3

Hi Paul - I’m not a PHP expert by any stretch but I am writing some PHP code (trying to write!) so I thought I’d take a break and toss out some suggestions…

On my emonPi I looked at php.ini file and found this:

; The number of significant digits displayed in floating point numbers.
; http://php.net/precision
precision = 14

Is your precision set to 7?

EDIT: from the Administration page I and running:

PHP Version 5.6.29-0+deb8u1 (Zend Version 2.6.0)

4

Hi Jon, thanks for the suggestion, I checked the php.ini for the precision value and it is set to 14. My reference to 7 is the precision I’m effectively seeing, although I do not know where the bottleneck is.

My PHP version is a tad behind yours at 5.6.27 as seen in the admin page screenshot above. Maybe I’ll try an update & upgrade…

Currently I’m thinking it might be Redis (only a gut feeling at this point) but I do not know how to prove that one way or the other. Perhaps Redis has a Precision setting somewhere too???

5

the problem is probably mysql feeds table the value datatype. I had the same issue in the past and changed datatype to decimal(26,12) instead of float or double (not sure what is being used now). After doing that change update database. And i never had the problem again. (if it’s the same problem but i’m guessing it is). I can’t seem to find the thread i had made about this, maybe it’s on the old forums but can’t seem to find them

found it: here’s the thread i made about it emoncms 9.2 - power to kwh feed stopped incrementing | Archived Forum i kept the decimal(26,12) ever since and problem went away

do note i don’t use redis so i never could test / confirm if it only happend without using redis

6

ignore previous post, i did not read the thread enough. Also don’t use redis, but can tell that i never switched back to any other datatype since i found out about that problem and also never had the problem again afterwards. but i don’t use redis and it was suggested in the old forums thread this did not happen when using redis.

edit also look here seems that’s where i did the initial message about it: EmonCMS 9.31 | 2016.02.13 released | Archived Forum

7

Paul - I know little about power_to_kwh and nothing about redis, but I don’t see a reference to redis mentioned in the power_to_kwh function you had highlighted.

I see it referenced in other sections like the kwh_to_power function. Am I missing the obvious?

8

get_timevalue does that it checks if redis is avaiblable and if so get the value from there if not gets the value from mysql “value” field see emoncms/feed_model.php at master · emoncms/emoncms · GitHub it does something else with virtual feeds but never looked into that as i never used it when i encountered that problem

9

Hello @pb66, I’ve replied to the issue you created on github not really luck in replicating yet, interested in your and anyone else’s thoughts Emoncms calculation errors using redis 3.1.0 and 3.1.2 · Issue #660 · emoncms/emoncms · GitHub

10

Hi Trystan, thanks for taking a look.

I have just posted some real data from the logs that should throw some more light on the issue altough I have to say I am still very much in the dark at this point.

11

Thanks @joyrider3774 your links have been very useful. I could recall all this had been a topic of discussion before but hadn’t found the threads.

Your comment

made me question the casting to float in the get_timevalue function , but I discovered “float”, “double” and “decimal” apparently amount to the same thing in PHP so that would appear to not be the bottleneck. Although the “if not gets the value from mysql “value” field” seems to not be the case, I thought “Last values” always come from mySQL (if not Redis) and that’s why we changed that field to a double in the SQL table. But my “value” fields have not updated since Aug 16’ so that is not where the initial or last values are sourced, perhaps that is the issue??