Does emoncms expects data to be send reguarly?

[joyrider3774]

Soo what i have been doing all the time with the programs i created is send data at a fixed interval for as much as this is possible and set the interval on the feeds to the same time value the data is being send at to emoncms.
Programming the emoncms_history component for home assitant it is possible to only send data when one of the values change. I intialy did this and just set the interval of a phpfina / fiwa feed to 10 seconds not knowing when data would arrive. Otherwise when i looked at realtime graph / widget it would not show a graph for the last X time selected in the graph and only update this (and perhpas other) graphs when new data had arrived.

I’m guessing but not sure that the phptimeseries should be used if not sending data at fixed interval ? is this correct ? also the webapps seem to expect a certain kind of feed type (can’t remember which one fina / fiwa) so if the phptimeseries assumption is correct i have 3 options …

1. always send data at a fixed interval (configurable in home assistant) to emoncms and users can use whatever feed engine they like and realtime graphs would update reguarly webapps would be not a problem
2. only send data if the data actually changed to emoncms and user should only use phptimeseries then and perhaps no webapps ?
3. do i need a combination of both depending on the data to be send ?

assuming this is correct what option would be the best one ? From home assistant’s point of view it’s obviously send only data when data actually changed, but what is it from emoncms points of view / usuage?

[pb66 (Paul)]

Good questions!

The predominant use of fixed interval (PHPfina/PHPfixa) feeds does mean that emoncms indirectly imposes a requirement to send data VERY regularly, the fixed interval is just that and if you are sending data at the same interval as the fixed interval then any network or server loading delays will cause some data to be discarded, this can be reduced by using pre-timestamped data and not using emoncms input stages to timestamp data.

@joyrider3774

I’m guessing but not sure that the phptimeseries should be used if not sending data at fixed interval ?

Correct. and also if accuracy is a priority as every datapoint is recorded with PHPtimeseries and the datapoints retain their exact timestamp they do not get rounded to the time of the feeds fixed interval occurrence.

@joyrider3774

also the webapps seem to expect a certain kind of feed type (can’t remember which one fina / fiwa)

Correct again, they were fairly recently changed to only use PHPfina in line with discussions about the PHPfiwa engine potentially/possibly/probably being phased out.

For sometime now I have been storing all my incoming data in PHPtimeseries feeds for better accuracy and then also using PHPfina feeds where access speeds might be a concern (large or frequent data requests) and also because the webapps excluded the PHPtimeseries engines.

The recent issue (as discussed in your Latest Dev version (display ?) bug with calculated kwh (bar) graphs thread that was also seen in the web apps led me to add PHPtimeseries as an acceptable feed type and I am inclined not to go back, the android app can use either so I think I will continue to use PHPtimeseries and maybe look at stop duplicating data in PHPtimeseries and PHPfina feeds to use the apps, I will still check the server loading and speed implications of not using PHPfina before deleting all my PHPfina feeds, but duplicating feeds just for using the apps is no longer a requirement.

All I did was replace any occurance of "engine":"5", with "engine":"2,5", in these 3 files (16x)

/var/www/html/emoncms/Modules/app/myelectric/myelectric.js

https://github.com/emoncms/app/blob/9.0/myelectric/myelectric.js#L5-L6

/var/www/html/emoncms/Modules/app/mysolarpv/mysolarpv.js

https://github.com/emoncms/app/blob/9.0/mysolarpv/mysolarpv.js#L4-L9

/var/www/html/emoncms/Modules/app/mysolarpvdivert/mysolarpvdivert.js

https://github.com/emoncms/app/blob/9.0/mysolarpvdivert/mysolarpvdivert.js#L4-L13

Whilst it seems to work just fine, I have not tested thoroughly enough to be sure it’s trouble-free, but if it’s good enough for the androidapp it should be fine, it’s the same api’s as far as I know so the web apps should not notice any change.

[joyrider3774]

So i think i can conclude that it is best to send data at a regular interval and not send data when it has changed ? Since it’s possible that a certain sensor would only update its value like every hour while others every 30 seconds etc. So to be “most” compatible it should send the data reguarly even though it did not change value.

@pb66

Correct again, they were fairly recently changed to only use PHPfina in line with discussions about the PHPfiwa engine potentially/possibly/probably being phased out.

Will there be a way to convert the fiwa to fina feeds retaining their original feedid (for not having to change each dashboard that used that feed)? I think not supported engines got hidden using the config and existing feeds using that engine still work but wouldn’t that impose that you need to programmatically still support them ? Unless you can convert them somehow

@pb66

All I did was replace any occurance of "engine":"5", with "engine":"2,5", in these 3 files (16x)

Thats basically what i did also but for fiwa feeds when i noticed i could not select them any more in the webapps

[pb66 (Paul)]

@joyrider3774

So i think i can conclude that it is best to send data at a regular interval and not send data when it has changed ? Since it’s possible that a certain sensor would only update its value like every hour while others every 30 seconds etc. So to be “most” compatible it should send the data reguarly even though it did not change value.

PHPtimeseries is the correct tool for irregular posting so you can choose to use only-on-change with PHPtimeseries or regular sends with either type. The gains made in disc space by not recording a timestamp (ie PHPfina) is lost many times over when you need to store 360 10sec values for a change once every hour.

I was just pointing out the lack of PHPtimeseries support in the apps can be overcome so shouldn’t be a deciding factor, I would like to see PHPtimeseries support return so I will submit a PR, whether that get’s accepted is outside my control though.

Although there is another reason regular sends may be better, for example if a value is 10 at 6am and doesn’t change until 6pm where it becomes 20, when looking at this data on a graph it will show the correct points at 6am and 6pm but the line between them would suggest a gradual change over the 12hrs, ie at midday it would appear the value was 15. This is due to the absence of a value of 10 posted at say, 5:59pm.

@joyrider3774

Will there be a way to convert the fiwa to fina feeds retaining their original feedid (for not having to change each dashboard that used that feed)? I think not supported engines got hidden using the config and existing feeds using that engine still work but wouldn’t that impose that you need to programmatically still support them ? Unless you can convert them somehow

I expect so, you would need to look at the useful scripts repo for a converter and I think PHPfiwa sould be supported in the background even if it is dropped from the current options, but I couldn’t guarantee that will be the case.

[joyrider3774]

@pb66

Although there is another reason regular sends may be better, for example if a value is 10 at 6am and doesn’t change until 6pm where it becomes 20, when looking at this data on a graph it will show the correct points at 6am and 6pm but the line between them would suggest a gradual change over the 12hrs, ie at midday it would appear the value was 15. This is due to the absence of a value of 10 posted at say, 5:59pm.

I had not thought of that, interpollation in the graphs is indeed not always wanted like in this example. I’ll stick to regular sending in worst case scenario i’ll support both but will have to document the consequences then like this example.
Thanks again

[nchaveiro (Chaveiro)]

If you’r talking about local installations, why not use MySQLTimeSeries instead of PHPTimeSeries?

• Having the data on the database allows easy maintenance.

[pb66 (Paul)]

For me the main reason for using PHPtimeseries is to trying to go with the general project direction, as having your own opinion on almost anything, is proving increasingly more difficult. plus I can only offer support on the foum for stuff I have experience with.

Features can get added that only work with officially supported feeds (PHP*) and support for other feed types seems forced at best, as you know we nearly lost mySQL support, the PHPtimeseries seems ok enough to not need to push against the grain, IMO PHPfina is a different kettle of fish though. I wish the project was a little more flexible on this as not everyone needs are the same.

I was actually very interested to see how your suggestion of using influxDB panned out but that got dismissed pretty quickly.