Aggregated / Big Data

Hi,

I’m not sure if this is the correct place to post this, but I was looking at my solar production data today, and had a thought…

I think it would be really interesting if we could aggregate data from many users (with their consent of course) and then put it onto a dashboard or graph.
Things like - current PV production (normalised in some way maybe to account for array sizes) totals for the day - or maybe even broken down to geographical areas.
Could also expand to show details of the amount of Electricity produced on a daily / weekly / monthly (etc) basis. I don’t know how many users hook into the EmonCMS website - but it might give an insight into the residential Solar PV production in the UK…

Would this be possible?

Thanks

Tim

Possible? - quite probably.
A good idea? - I’m not sure.
Is there a will? - I’ll wait and see.

If you publish an average per panel (or per square metre, or similar), probably on a regional and national basis (because remember, you don’t have to be in the UK to send your data), then I can see it as being very useful to anyone considering investing in solar generation, or for anyone concerned with the efficiency of their existing installation.

Where I’d have concerns is publishing totals. Being a highly trained, fully experienced, card-carrying cynic, :smiling_imp: I can see a way for an unscrupulous operator to manipulate the data. As emoncms.org is free, it would be comparatively simple to manufacture and send false data, and that would skew the figures. I can’t see what anyone would have to gain from that, but I see the risk. Most users would no doubt send accurate and meaningful data, but if it’s possible, some script-kiddie somewhere is going to do it, sooner or later.

Hi Robert,

I understand your concerns around publishing data, but if it is anonymous, and aggregated, I can’t see how it would be harmful…

What does sound cool to me would be to be able to view a map and see what the current generation rate is across the country / world and also see historical data. So if (as you say) you were interested in installing solar, or monitoring, you could get an idea of the amount of energy you could generate.

I read somewhere recently, that whilst the solar farms are metered for generation, most domestic installations aren’t, so there is no way for the power companies / interested parties to know actual data about domestic generation. This may be a small way to address that, and could maybe be used to promote this site?

Anyway - it is just a thought, so if there is not the interest, then I will leave it be.

FWIW, somehing like that exists on pvoutput.org

2 Likes

that is what we are about to do in small scale.

1,5 kW sunnyboy south 30°
1,5 kW sunnyboy south 45°
1,5 kW sunnyboy north 60° (go figure why anybody would do that)
1,5 kW sunnyboy on at sun tracker system (currently brok)

just for comparison.

our test site location: Hadsten, Denmark
http://79.174.152.130/wordpress/

work has just begun so real data yet naah…

Henrik

Hi All,

Thanks for the examples.

I was thinking of something that shows the aggregated information, rather than the individual outputs.
Maybe a map of an area with regional values in different colours - looking something like this - http://solargis.com/assets/graphic/free-map/GHI/Solargis-United-Kingdom-GHI-solar-resource-map-en.png

With some totals for the region, or country.

I was thinking that the emoncms system must already hold this information, so it would just be a case of displaying it so we can see how much energy is being produced at any one time (or historically).

The data at the link I posted above is aggregated data

e.g. team 16 is the UK team. If you click on the team name, then click on the value immediately following the team name you get individual outputs.

However, clicking on the team name, then clicking on the “map” link, gives you a map showing an aggregated value as well as group links that let you drill down to the region / area / individual level.

Here’s an example:


Hi Bill,

Apologies - I missed that.
I did click around, but didn’t spend enough time to understand it properly.

In that case - yes - that was what I was suggesting. It’s good to know that it is happening somewhere - although I’m not sure it is the most intuitive site.

I’ll look into joining up!

The cost is right. It’s free to join. I’ve used it since 2013. Can’t say I’ve any complaints. :+1:

Cool.
What method do you use to upload your data?

A Python script calls a BASH script which uses curl to do the data transfer.
I do it that way because I use the Python script to read an RS485 Watthour meter at 5 second intervals.

At the time, doing a system call to the BASH script was the quickest/easiest way (for me) to get the job done. I haven’t gotten around to eliminating the BASH part and making it just a single Python script mostly because it works, and the Rpi it runs on doesn’t do anything else other than read the Wh meter.

They have documented the API quite well.
The subsection titled Add Status Service has an example of how to upload data via curl.

Hello Tim,

I think this is an excellent suggestion.
It would be great to group people together by whatever affinity/location and be able to track aggregated data.

It could also give extra value. Imagine that you’re in a group where everyone has a 4kWp PV system. Your output could be monitored and you receive an alert if your production drops below the average of others in the group consistently.

After more searching, I see the Groups Module New Emoncms Groups/Administration module which may do exactly what you’re looking for.
A better link (to the github module with description) is GitHub - emoncms/group: In development: Emoncms groups module

Best regards /Colm

Hello @carbontracking I agree lots of potential for data sharing like this. A fair bit of work curating and presenting the datasets to make them useful I think.

The groups module is more focused around administration of multiple emoncms accounts and is unfortunately a module that has not been maintained for a while, it was developed actively for a particular project a few years back but has not had much attention since.

I’m another pvoutput.org person - my Python script gets the data from emoncms every 5 minutes and pushes it to my pvoutput systems (and also to solcast.com to make my weather forecast / expected output more accurate).

Highbanks combined PV