Hi!
I have lots of feeds but with different data types
phptimeseries
phpfiwa
phpfina
Mysql
Which should I use and when?
I log the raw, then convert to kwh and log, then convert to kwh/d and log
finally convert to histogram and log.
Help!
David - This may help with understanding. Hopefully someone smarter will wander along and answer your questions. (this info may not be up to date)
PHPTimeSeries
https://github.com/openenergymonitor/documentation/blob/master/BuildingBlocks/TimeSeries/variableinterval.md
a brief summary of the datatypes
- phpfina
is the default type and it offers faster manipulation in graphs etc due to the fixed intervals allowing for the timestamps to be calculated by number of datapoints rather than queried and handled as another data field. It can also reduce the size on disk of regularly collected data but it’s size will grow relative to time passed and interval size, not in relation to the data captured. eg a 10s fixed interval will record 8640 datapoints a day even if the value is only upadted once a week. - phptimeseries
This is the recommended variable interval feed, it records the time and the value at each update, the timestamp doubles the size and slows it down a little but it’s ideal for irregular updates and it is far more accurate than the fixed interval feeds and IMO it is the best way to faithfully store your data. - phpfiwa
has been or is most likely to be depreciated, it is a multi layer version of the PHPfina - Mysql
was predominantly used for the daily totals in recent years, they are being phased out so mySQL feeds will become scarcer, but mySQL is a very universal database so it’s unlikely support will ever disappear completely especially as all the emoncms settings are also managed in mySQL.
The bottom line is you really only need to choose between fixed or variable interval unless you have a specific need that isn’t covered by the phpfina or phptimeseries. I prefer variable over fixed because once you alter the data to conform to a fixed interval it cannot be undone and I do a lot of debugging/analysis where the individual timestamps are more important than a light and fast dataset, plus I have plenty of harddrive disc space. I do add phpfina feeds for use with the apps and also for any feeds used predominantly for graphing.
2 Likes
Hi Paul,
Thanks for that. At the moment I have a few feeds that suffered the GMT/BST data “reset/corruption” issue so I’m
having to re-calculate the kwh from the power feeds. Lots of my data feeds are logging as phptimeseries
which the power_to_kwh.php script don’t like.
Would you like to create a new feed or overwrite an existing feed? enter 1 for new, 2 for overwrite: 2
Please enter feedid of the accumulating kwh feed you wish to overwrite: 27
ERROR: kWh feed must be a PHPFINA feed
My power feed is phptimeseries as is my kwh feed thats being generated from it.
Looks like I need to update the script to be able to output to a phptimeseries data - or should I make my kwh a PHPFINA data type? ( thats 15 feeds I need to change!)
Are you logging the kwh/d and the histogram’s as well? Or are all calculated on the fly for your dashboards?
I also need to move lots of them from PHPFIWA to PHPFINA as well (about 30 feeds!)
Hello David, Looking through the conversion scripts they really need updating and a script for converting from PHPTimeseries to PHPFina is missing. I will look at adding this script.
In the meantime I have added a quick feed checker script that just lists all the emoncms feeds and their engines and checks the data directory location, I wonder if you could just run this script to start with, check if you get any errors and copy the output here?
The script can be found here: (Run git pull to update the usefulscripts repository)
Then to run:
php check_emoncms_feeds_for_conversion.php
Hi Trystan ,
Here’s the output. I’ve been running OEM since july 2014 so I have a lot of data I don’t want to loose.
Just want to fix the BST?GMT data issues so all the graphs work correctly.
Then I can update the names so that the android app will be happy and the other apps.
Thanks
David
------------------------------------- Emoncms Feed Conversion Check Script ------------------------------------- feed:1 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:2 engine:MYSQL feed:3 engine:MYSQL feed:5 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:6 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:20 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:7 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:8 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:30 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:no datafile:no feed:11 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:12 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:13 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:14 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:31 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:no datafile:no feed:29 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:21 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:22 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:23 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:24 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:25 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:27 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:28 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:32 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:33 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:34 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:35 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:36 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:37 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:38 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:no datafile:no feed:39 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:no datafile:no feed:40 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:41 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:no datafile:no feed:42 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:43 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:45 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:47 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:48 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:49 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:50 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:51 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:52 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:53 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:54 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:55 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:56 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:57 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:58 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:59 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:60 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:61 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:62 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:63 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:64 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:65 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:66 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:67 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:no feed:68 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:69 engine:PHPFINA datadir:/var/lib/phpfina/ metafile:yes datafile:yes feed:70 engine:PHPTIMESERIES datadir:/var/lib/phptimeseries/ datafile:yes feed:71 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:72 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes feed:73 engine:PHPFIWA datadir:/var/lib/phpfiwa/ metafile:yes datafile:yes
FYI
feed 30 is a histogram (dunno why it thinks there is no data file - I can view the data fine
same for 31, 38, 39, 41, and 67
@TrystanLea could the output from this go into the docs somewhere. It is something I need to do to tidy up things. I’m pretty sure I no longer use Timeseries but I noticed the service is still running so I’d like to check that I don’t use it so I can get rid of it.
Perhaps ‘Tidy up old data feeds’ could be a good title 
Hi Paul,
Would it be best to record the raw data as phptimeseries
and then any data manipulation and store that in phpfina ?
thanks
That is what I prefer to do, it’s not ideal if you are short of disk space, trying to reduce sdcard disk writes or minimize server workload, but it does give you both a faithful copy of your data as well as faster data graphing/reporting etc.
OK Cool. I’ll do that then
Disc space isn’t an issue its got a 1TB USB HDD !
Thanks!
I’ve put together 3 new feed conversion scripts that support the following:
phpfiwa to phpfina
phptimeseries to phpfina
phptimestore to phpfina
All available in usefulscripts:
They provide the option to either replace an existing feed or to create a new feed which will allow you to do what @pb66 is suggesting.
I’ve been able to test the phpfiwa and phptimeseries scripts but not the phptimestore one. I’ve also tried to add in quite a few checks so that the script exits at common failure points.
The scripts provide an on screen walk-through for configuration.
Ace thanks for that!
I’m also recalculating the kwh data as it corrupted - but to get the “source” initial log into the correct format would be good.
How can i recalculate the kwh/d data and the histogram data?
Thanks!
Great!
The kWh/d data type has been replaced with cumulative kWh feeds as the recommended approach.
There’s a bit more on that here: Calculating Daily kWh - Guide | OpenEnergyMonitor and Creating kWh per day bar graphs from Accumulating kWh | Archived Forum
For histograms there is now a histogram tool build into the emoncms graph module which avoids the need for histogram feeds in many cases: Histograms - Guide | OpenEnergyMonitor.
@david, I presume you’re looking for a script to build your accumulating kWh feed as per Trystan’s post above, try this one.
Paul
This thread fits great to my question.
I import all data from csv files as mass data to the api. The data contains a timestamp (converted form datetime to unix time) and different values, which are meassured every minute. The import of the csv files happens every 10 or 15 min.
What is the correct data type?
I’d say phptimeseries
but…
… but?
If you read this, you’ll have an insight into how the various databases were designed and how they work. You might then be able to make an informed decision.
1 Like