Feed File Naming

I have 4 emonTx’s each with an RPi running emoncms & each having a different Node number - all connected to one network.
I’m using phptimeseries so a for example data file is … feed_3.MYD.
So it is possible for more than one Node to create a file named feed_3.MYD.

How can I distinguish between them?

It would be great if, for example, feeds from Nodes 13 & 14 were named feed_13_3.MYD and feed_14_3.MYD

Any suggestions on how to achieve this?

Many thx

So are you saying all 4 emonTx’s are logged at all 4 emoncms’s?

No. The feed database in mysql auto-increments from the first feed created (feed id 1) to issue another unique feed id for each feed created regardless of node id or even user id. The feed id must be unique to the whole emoncms instance. There could be multiple users with node 10, so you could end up with multiple feed_10_3.MYD unless you include user id too, plus node names are in common use now so you might end up with something like feed_1_emontx_in_the_garage_3.MYD where the first “1” is the user id.

Is a question that keeps crossing my mind, however, since the same feed can be accessed/updated from more than one input, I do not think the link between node and feed would be always useful.

My current thoughts on this is that phpfina and phptimeseries have different file extns, I see no reason to have 3 separate folders defined in the settings.php. I would like to have a single path defined in settings.php and perhaps get emoncms to create a single folder per user/account and put all that users php* files in one folder. not that this would help your position at all since I assume you are just using the one user.

Any alteration to the auto-incremented part of the feedid would involve big changes to how emoncms works, but I guess it might be possible to make the first part of the feed name more user friendly and make emoncms ignore all but the number after the last underscore, but that probably involves quite a bit of work and may not work as well as you would like.

How often do you need to access the feed files directly?

I recently had to move a load of accounts to another emoncms instance, so I wrote a short script to get a feed list (for the account I wanted to move) and copy all the respective feed files by using the engine value to know where to look and the feedid to make up the filename.

You could do something similar that instead of copying the feed files, it creates symlinks in a separate folder to each feed file using both the feed tag and feed name.

Paul …
Many thx for yr response.

The 4 emontX/Rpi’s are located at the PV inverter, the incoming grid meter, the pool ASHP and the car charging point.

Each emonTx logs to emoncms running on its individual RPi – so 4 emoncms instances are running – all independent of each other.

It’s a 3 phase installation, has one user (pi) and each emonTx/RPi has a different Node number.

So there are several instances where, for example, feed_n.MYD exists on more than one Node.

Yr response convinces me that changing the method of feed file naming would be difficult. However yr mention of symlinks triggers a thought …

Also on the network is a fifth RPi running Raspbian Jessie and called watchman. watchman uses SSHFS over SSH to mount the /home/pi/data directory from each of the 4 emonTx/RPI’s. So on watchman I have 4 directories … /mnt/emonpi-node-n/phptimeseries where n=11,12,13 &14 with each containing the many .MYD files.

The thought you have triggered is …
On watchman create a directory /home/pi/data/phptimeseries
In that directory (for example) have a file named feed_11_3.MYD which is symlinked to /mnt/emonpi-node-11/phptimeseries/feed_3.MYD
Install emoncms to watchman running Raspian Jessie

All this with the objective of using the emoncms graphing & dashboard and csv exporting capability and, in particular, to have My Solar displayed on a RPi screen located on the kitchen counter top.

Does that sound as if it might work?

And it’s all a bit more challenging as the installation is at my son’s place 40 miles away. However, Dataplicity works great to access watchman and from there I can SSH to the 4 individual Node emonTx/RPi’s.

Might this work? Better suggestions?
Many thx

So you are running 4 different emoncms instances, one for each node and then needing to sync these up to a fifth.

I’m not sure why you are running 4 (or 5 ?) separate emoncms instances, I can understand there maybe reason to run 4 or 5 Pi’s if there are (for example) RFM network issues and you have the 4 nodes serially connected, each to a Pi. But in that case I would be inclined to use 3 Pi’s to just forward the data to the 4th and just run a single emoncms instance. The way you have it is making a lot of work for yourself on 2 fronts, firstly you have 5 emoncms’s to maintain rather than just 1, and you also need to do the syncing, which will most likely add latency to when you can view the data centrally.

Is there any reason you couldn’t set emonhub in 3 Pi’s to post to the 4th Pi via http as if it were emoncms.org?

However, if you want to stick with syncing the 4 emoncms’s to a 5th, probably the easiest thing to do is to change the auto-increment start point for the feeds table in the “node” emoncms’s so that the first emoncms starts issuing feeds from say 10000, and the second from 20000, the third from 30000 and the forth from 40000. That way when they all get synced the feed ids will all be unique unless you create more than 9999 feeds on any one node emoncms. This sort of gives you what you were originally asking for as the feed file names will be linked to the source device by the number range, Instead of “feed_11_3.MYD” you could have “feed_110003.MYD” if you use the nodeid x 1000 as the auto-increment start point.

This can be done via phpmyadmin or from the commandline with mysql commands. But it will still be pretty awkward to manage as you still need to pull all the feeds into one place and merge all the folders into one.

Paul …

Thx for yr advice. For further background detail …

The PV inverter (30 kW ground mount array) is 100 metres distant from the house and incoming grid meter so I didn’t consider radio.

Two external wifi antenna have been installed giving site coverage.

I did try with ESP Huzzah modules and gave up.

Finally settled on a robust solution for each of the 4 Nodes – RPi and serial direct with a spinning HDD for both the OS & Data partitions – power bank to be added later (unlike emonTx, RPi does not reboot after a loss of power).

(Cost per Node is in excess of £100 but in the context of a large investment, justified in my view.)

This approach means that data logging will be robust - serial direct and saved at each Node location and not affected by wifi problems – the external antenna do seem to need rebooting every couple of months.

Installation has been progressive as I’ve moved up the learning curve – spent most of the summer on the pool Node and its ASHP and water temperatures. However I now have 4 Nodes running each using a standard OeM approach – hopefully, easier to maintain, get forum help, etc.

My preference is to restrict the bespoke stuff to a fifth RPi. It is hard wired to the network, is my point of contact via Dataplicity, will be a home for a data backup and can periodically ping the 4 Nodes and 2 external antenna sending an email if anything is down.

Yr suggestion for feed file naming will solve the duplicates problem – thx.

Re yr suggestion of posting via http as if it were emoncms.org – if wifi is temporarily down, data will not be posted. When wifi is restored, is the backlog of data then posted? If not then mounting the Nodes data on the fifth RPi (using SSHFS over SSH) will provide a more complete record at the fifth RPi … or am I missing something?

Assuming I can find a way of getting the feed files (phptimeseries at 10 sec intervals) from the 4 Nodes into 1 directory on the fifth RPi that emoncms can use – will it be a problem that the timestamps for each Node will not be exactly aligned?

Remembering that the objective is to use the emoncms graphing & dashboard and csv exporting capability and, in particular, to have My Solar displayed on a RPi screen located on the kitchen counter top.

Thx for yr help.

Paul …

I’ve tried yr suggestion for feed file renaming. Have discovered how to get into the MySQL monitor and to change to the emoncms database.

On one of my 4 Nodes (emonTx & RPi) - If I do … select * from feeds; … I can see 2 feeds ID=1 and ID=2 with their names, etc.

I then tried … ALTER TABLE feeds AUTO_INCREMENT=14000; … and got the response Query OK, 0 Rows affected, Records: 0 Duplicates: 0 Warnings: 0

After a reboot, nothing seems to have changed - the feeds are still ID=1 and ID=2

Then I tried … ALTER TABLE feeds IDENTITY(14000,1); … and got SQL syntax error near IDENTITY(14000,1)

Obviously I’m missing something - any suggestions most welcome
Thx

Changing the auto-increment start point will only effect subsequently created feeds not the existing feeds.

I am not familiar with the “IDENTITY” command but be wary of this approach, the feedids and the feed file names currently match. If you were to change the feed id from 1 to 14000 (via mysql or in phpmyadmin) the files will be orphaned unless you edit the filenames too. This will probably not be easy to do as redis, the stuff in memory and mySQL indexing are all working against you. At the very least you should stop the Apache2 server and flush redis when you do this, but unless you have a lot of data you wish to keep, I would say go through the normal channels and create nes feeds, the raised autoincrement will mean the new feeds are 14001 and 14002 etc.