Reviewing other EmonCMS instance data sources

Emoncms Development
1

Inspired by @Timbones’s nice work over on the dashboard I’ve been looking at ways to explore the data.

I know that at the moment each data source also has it’s own EmonCMS wrapped around it and that’s what we’re using to render the data. However, seeing as the data is openly exposed then we could arguably use another user interface.

ofc I was looking at using my MMSP experimentation app. Partly because of @TrystanLea’s comment about analyzing the space and DHW heating separately which I’ve already coded.

So, I came up with a cunning plan to simply have my app use a different server when asking for data.json. That sorta worked, but it’s based on my feedIds so that’s bad. Next up I ran with my app calling out to the remote feed.json which then allows you to request the right data.json parameters and away you go.

If you imagine doing this with a query string so you can send hyperlinks to it then you end up with code a bit like this to branch on finding the feed.json:

    const params = new Proxy(new URLSearchParams(window.location.search), {
        get: (searchParams, prop) => searchParams.get(prop),
    });
    // See if a custom datasource is specified
    const datasource_value = params.datasource;
    if (datasource_value)
        feed_list_url=datasource_value
    else
        feed_list_url="../feed/list.json"
    console.log("Using feeds from " + feed_list_url)

    // Load the available feeds for this user's system so we can configure the application
    config.feeds = await (await fetch(feed_list_url)).json()

and so on.

Now, ofc the first feed I tried was from this one which I know is much like mine:

http://emon.cryosphere.co.uk/ashp

However, this fell over because I’m hosting my app on an HTTPS server and browsers have become really strict about not loading HTTP content into HTTPS pages.

Next I tried:

https://emoncms.org/heatgeek

That showed another problem, CORS is set and returned from OPTION pre-flight requests to emoncms.org but when you make the subsequent GET request the Access-Control-Allow-Origin header is missing so the browser’s CORS logic blocks you seeing the result.

Then I side-stepped that using the ridiculous cors-anywhere so I have a request like this:

https://example.com/emoncms/app/view?name=MMSP+Heat+Pump&datasource=https://cors-anywhere.herokuapp.com/https://emoncms.org/heatgeek/feed/list.json

This now gives me the other datasources list of feeds and now I need to work out what to render. Clearly the next problem will be that the name of the feeds in their EmonCMS doesn’t match mine. We normally handle this by having it stored as application configuration, but clearly that only covers a single datasource (the "local one). Anyway, I hacked that too so it uses the names from the heatgeek feed.

As a result I can now review other people’s data in my instance of EmonCMS using my custom apps in there. Here’s me looking at the heatgeek feed for example:

image
image2214×886 161 KB

Clearly I’ve only done a little bit of mapping. That’s mainly because the request limit through cors-anywhere is 50 per hour so I very quickly use them up.

Q1. Is this a good idea?

Then, if we decide it is we have some problems to sort out:

Q2. HTTP-based data and HTTPS-based apps don’t get along (but the other way round would be OK)
A2. There are no magic fixes, needs a proxy or some other hack

Q3. CORS setup on emoncms.org needs enhancing
A3. emoncms.org admins would need to decide to do that (for example)

Q4. Feed mapping differs per datasource because of different feed names
A4. I have no idea how to approach this. I can concoct things, but none are very appealing

1 Like
2

That’s some nifty work, David. I had suggested something similar, whereby heatpumpmonitor.org could host it’s own instance of the My Heatpump app, and pull the data it needs from the individual servers. It seems you found some of the reasons that make that difficult to do, so I think I’m glad I skipped that step.

I also wanted to pull data directly from each instance so that I could calculate SCOP. I started by scraping the config from the HTML of the app page, then Trystan added a special page for me: getconfig. This gives you all the feeds that the app defines, but none of the others.

The primary server could be running a proxy, that loads the config, map the feeds and forwards the data to the webserver, bypassing any issues with https and cors.

1 Like
3

Aha! It took me a minute to work out why you sounded relaxed.

You’re side-stepping question 4 by narrowing your focus to just the data that feeds into the existing heat pump app. Saucy. You know people have those configured in the app config on their environments.

Then you can implement the code / app to consume this config and you have those specific fields available to you and it gives you confidence you can identify them.

Of course it would be easy to mention that it restricts you to those common fields, but frankly there is a lot of value to be gained by just getting those.

So it sounds like you should push ahead with that specific use-case.

1 Like
4

Yup, that’s exactly what I did → the code.

There is one caveat: if user doesn’t actually set the feeds but relies on matching the default names, the config doesn’t show them. This is something we can ask emoncms fix when the time comes.

Indeed, but maybe we can request more feeds to be added to the app config…

1 Like
5

Nice to see Tim, good stuff.