Struggles with the EmonCMS API

I’m not going to proceed with EmonCMS, but I think it’s an awesome project with a lot of potential, and I’d really like to see it succeed and improve.

I spent 15 years as a software engineer and 10 more years as an energy systems consultant, doing nothing all day but trying to figure out people’s energy systems. I’ve got experience with lots of different commercial EMIS software, much of it with less capability than what you’ve got here. From that perspective, here are a few comments about my experience of trying and failing to use the API to visualize historical and current meter data.

If someone wants to convert these to individual issues on the github issues tracker, be my guest!

Silent failures: I really struggled with bulk upload and API, mostly because of some silent failures – places where the server returned “OK”, but didn’t accept my upload. The fact that this egregious data-loss bug is labelled an “enhancement request” is the main thing that convinced me to walk away from this project.

EmonCMS.org differences: I think EmonCMS.org was behaving differently from the github software in some cases. In particular, failing wherever the API call included a time= or offset= argument. I never did succeed at uploading historical data to EmonCMS.org and then viewing it.

Named-key inputs: if you’re using the json= API with named key inputs, there is no obvious way to then bulk upload to those inputs. No matter what you do, bulk.json creates new numbered inputs like csv=[] instead of using the existing named ones.

API input names: There is no way to use the api to set meaningful names on newly created inputs. If you’re trying to create inputs without a thousand mouse clicks, you’ll have to live without naming them.

API create feeds: I couldn’t find documentation on how to set the various arguments to create.json if I want to create a complete feed over the api. The API help gives a cryptic example, but what’s a tag? What’s engine 5? Is the id a unique identifier for the input?

Null datapoints: For some data sources, a null datapoint simply means that no data was created for that time point. That’s reasonable behaviour. The API seems to reject my entire bulk upload if there is a null datapoint anywhere, which seems both harsh and incorrect.

Re-uploading history: Developing visualizations in the blind is difficult. I tried and tried to create a feed and then bulk upload history data so I could [edit:] see the feed in action. Re-uploading historical data to an input didn’t ever seem to update the dependent feed. Maybe I was doing something wrong, but I never did get it to work the way I’d expect.

2 Likes

As a Software engineer you will apreciate the problems functionality creep causes. I think this is an example. The system AFAIK was designed for taking in real time data. The bulk upload etc is something that has come along later and is not fully developed as yet hence the issues. Until someone uses it in anger, many of these issues remain low priority or hidden completely.

I too have struggled with the API and the format used for json (which is not true json) so I am with you there :smile:.

Indeed, it’s brutally difficult to keep bugs out of new functionality, especially when it’s a tacked-on afterthought as you’ve described. The only way that I’ve ever seen work is to have module-level functional testing against a handful of expected use-cases. Perhaps one day my notes will end up in the hands of someone trying to decide which tests to write!

1 Like

See what it did, perhaps?

Thanks Bill. Corrected to “see the feed in action” for clarity.

I’m sorry too hear you are not going to be using emoncms, but I do appreciate you taking the time to explain why and although I am not a core emoncms developer I would like to respond to the points you raised, hopefully to help future users even if it is too late to tempt you back. It would be of great benefit to the project to have users with your experience on board.

  • Silent failures, agreed this is not a good situation, many softwares including previous versions of our own emonhub have relied on this response for confirmation of a successful post before deleting the local copy. It is an error and should be labelled as a bug. it now has “bug” status.

  • EmonCMS.org differences: Whilst it isn’t clear to many users there are significant differences between the emoncms repo and the code run on emoncms.org. There is a separate repo for emoncmsorg if you are interested in the code, but at the very least the user level differences should be documented or preferably, not exist. This is not a development verses stable version issue, there are very different stratagies applied and these need to be highlighted.

  • Named-key inputs: The issue here is that the “inputname” or the “inputindex” share the same “key” field so you cannot use both, when using CSV bulk input api the “key” is explicitly set to the input’s index number as an “inputname”, the input pages list “key” and “name” fields, behind the scenes they are actually “name” and " description" fields. I have regularly asked for an additional field so that both “inputindex” and “inputname” can be used simultaneously

  • API input names: Although poorly documented, this is actually possible using the input/set api see below. Also take a look at the emoncms device module, I use this a lot for bulk creation of inputs, processlists and feeds. Possibly not in the way it was designed but it works well for the purpose.

Set input fields	http://localhost/emoncms/input/set.json?inputid=0&fields={'description':'Input Description'}
  • API create feeds: I agree. The API’s need an overhaul to be more user friendly or at the very least documented. There are many things that can be done via the api’s that are too complicated or simply not known.

  • Null datapoints: Although I haven’t tested lately, it is/was possible to pass null values to a local emoncms install. This is a feature I did contribute to emoncms some time ago, See the “Can you ‘load’ nulls in BULK json upload format to emoncms.org?” thread for some info.

  • Re-uploading history: Not entirely sure I have the right end of the stick here but… If you are trying to bulk upload back-data to phpfina or phptimeseries, the trick is to create the feed whilst there is no incoming data (or to use the feeds api) to create an un initialised feed. A feed is initialised upon recieving it’s first adta and the timestamp of that first data becomes the “start date” of the feed, once created you cannot post data earlier that the “start date” so making the first datapoint the earliest ever is essential to back-fill historic data.

If I can tempt you to retry emoncms I would happily try and answer any questions you have as I have a fairly good working knowledge of it. The position I (we) and future “power users” have is that it is currently complex, incomplete and undocumented, and because of that, many move on or just accepts its shortfalls. the result is nothing gets improved as there isn’t the demand for the more indepth stuff.

In actual fact it, is in the name of “simplification” that emoncms.org runs very different code to the emoncms repo and that the documentation is moving towards a “quick user guide”, so it good to know there is also a demand for in-depth documentation and advanced features too.

1 Like

Fair enough. The system doesn’t meet your needs. My problem is that you don’t really describe what your needs are. A shopping list of problems doesn’t necessarily mean the system doesn’t work; It just doesn’t work for you.

I was following your previous thread where you were expressing frustration with the bulk update facility. Coincidentally, I was implementing bulk into my software (IotaWatt) at the same time. I posted a question about update not working, along with a log of my transactions. Trystan Lea got right back and told me he found a fixed a bug that caused it. As a result of that bug, my software tried to update the “historical” data that was lost, and I encountered what I think you were talking about in your earlier posts. I suspected that updating history was not possible, mostly because given the architecture I wouldn’t know how to do that efficiently. A few days later, Trystan confirmed my suspicion and posted a statement to that effect on the input API description page.

I agree that the system shouldn’t respond “ok” to such a transaction, but once again, I can see the problems with guaranteeing it will do that without complicating the protocol. I say that because the various data groups in a single message can each specify a relative time that could resolve to any time since Jan 1, 1970. I just take the response to mean “ok, I got the input”, how that transient input data is handled in it’s journey to a permanent home in a feed data stream is subject to the rules in place. Now we know that rules don’t allow historic updates.

Bulk works fine for me to post a lot of new data to bring a feed up to date, but is not designed to backfill historical data.

As another old warhorse in the software industry, I understand your nostalgia for complete and accurate documentation and testing. That’s just not how things work these days. These eMon folks seem dedicated, and the system is not only open, the design is elegant and extensible. It will never be all things to all people, nothing ever is. Good luck with your quest to do whatever it is you are trying to do.

2 Likes

Thanks All.

Agreed that the silent ok’s are an issue. How to respond to errors on the emoncms.org input API needs some work. Part of the original reason for returning ok even on error was that the original API (4-5 years ago) had little error checking and that I was worried that a sudden addition of error responses would crash devices (typical Nanode’s at the time) unable to deal with the error responses. This needs a gradual careful introduction. There is also an additional complication with the way that inputs are passed into input queues that are one way in nature without the possibility of returning the error in the input request. I was thinking that this could be addressed by having input errors listed in the Inputs UI rather than as a response to the input/post or input/bulk API’s. There’s a fair bit of work required to do this, its not an easy fix and needs to be mindful of additional overheads etc on emoncms.org.

Differences between emoncms.org and emoncms, yes as @pb66 mentioned there are significant differences between the emoncms repo and the code run on emoncms.org. In general from a user perspective emoncms.org runs a reduced feature set to that available in the self-install emoncms. Less feed engines, input processors and visualisations. The differences will always be changing over time and emoncms.org in general has a slower moving update rate due to the need to ensure stability and development/testing capacity. If there is a particular difference on this front anyone would like clarification on, let me know.

I would love to get to a point where we have a dedicated team running and developing emoncms.org where we can then really push things forward and get all these details right. I will post again on this soon.

Interesting discussion. I have been figuring out what to use for monitoring for some time. I was using Smart Energy Groups from around the time they started. I stopped, because the free ride was over and value of the site vs. the cost did not seem worth it to me. Having spent small amounts of time over the last year plus looking at alternatives, I appreciate SEG much more now. Part of it, I am sure is that I am more familiar with it, but part of it is that it has a pretty decent design for dealing with different types of data and a good set of visualizations that are pretty easy to use and pretty performant.

The reason I have not gone back is I really want something that works locally, because we have frequent power/Internet outages here and I want a system that will work with that constraint.

I had been hearing about emoncms for some time so I decided to check it out. I first started trying to install it locally, but the instructions for installing it (and keeping it secure) on Windows leave much to be considered easy to use/deploy (even for someone skilled in the art of software development). I then tried using it on emoncms.org. Even this was somewhat complicated, because the graph capability is wholly unsuited to feeds that have a small amount of data. The getting started info does not provide enough information about this. It is vitally important to understand some basic concepts (that I am sure everyone who is familiar with emoncms understands without even thinking about it).

It all starts with Inputs. These get populated automatically when you send data to emoncms, but they really do nothing. This is okay, but it is not very well explained. This is also where I ran into my first problem. The local version allows names for nodes, but emoncms.org requires numbers and there is a limit to the number (probably 32). It took me a long time to figure out that, primarily because the API doesn’t return an error for an obviously wrong call.

Once I figured out that I had to use numbers for the node (but I could use names for next level (tag?) I was able to make some more progress. The next thing I ran into, is all the examples show sending the API key as part of the URI (which is a pretty insecure way of doing it). The help does talk about ways to avoid this, but does not provide any examples. None of the code that I could find in various locations showed how to do it more securely. Since I know how to debug systems I tried debugging it on my local copy to see if I could figure it out. I had a solution, but then someone posted a question (that I can’t find now) about something similar and it turned out that posting to bulk.json instead of post.json with json data instead of data data worked fine. I figured this was easier and changed my code to use the different URI. This took a long time to figure out because the API documentation is sparse and doesn’t really explain all the options that are available and doesn’t have examples of more secure ways of doing things (so, of course, all the code that most people write does things the same insecure way). I am not really that paranoid about the security of the data I am sending, but I have enough training to know that I should at least think about how secure something needs to be.

Finally I had figured out how to get my data traveling to emoncms.org. The next hurdle was figuring out how to visualize it in a meaningful way. It took me a while to figure out my local copy didn’t have everything (dashboards are not included by default). The Getting Started might have said something about this, but I don’t recall for sure. I decided to just use emoncms.org to see if the visualizations would work well or not.

I have used a few sites for this in the past (ThingSpeak, SEG, AdaFruit.IO and probably some others that no longer exist). I do understand this is not an easy thing to do well. I found the dashboard to be incredibly difficult (it looks like it is pretty powerful too) to use. Specifically the Graph. It seems like it works okay if you already have a lot of data in your feed, but it is nearly impossible to use if you only have a little data (because you are just starting out). The UX for graphs is way too geeky. Here is a really good example of the power of SEG: https://smartenergygroups.com/groups/teken_energy

Scroll down to the part that has Real Time Watts and see the multiday realtime data chart that lets you see the big picture and also zoom in to find out the details. I don’t believe there is any way (currently) to anything remotely this good with emoncms. Note also the sliders that you can use to change the amount of data you are seeing and also which part of it. The graph in emoncms can do that, but it is much harder to understand how to do it. Notice how quickly you can scroll through the data. Somehow it manages to do all of this in real time. That is truly amazing and almost magical (as they say magic always comes at a price :slight_smile:) Now I am sure that the person who made that site spent a lot of time on his dashboards, but the thing is SEG provides a really rich set of things that support using it for many types of data. All feeds come with averaging and statistics, so it is really easy to see them on a dashboard. I would love to be able to do these kinds of things with emoncms and I think the other users would love it too. I don’t know what it would take to implement any of these enhancements, but if I continue to use it I will probably attempt to make it do them. (or I might just decide it really is worth the $10-$20/month to get it much easier by using SEG).

I will say that it still feels like early days IOT visualization and things will hopefully get better. Also, there does seem to be a fair amount of engagement of emoncms, which is promising.

@Frogmore42 thanks for the feedback, lots of points, I wont be able to address all in a single reply, so to start with and in response also to @thundersun. I realise that a lot of the using emoncms guides written as part of the OpenEnergyMonitor Guide were not linked to from the emoncms repo, which if you arrive from emoncms.org is the entry point for documentation (a case of things changing over time. The content in the guide is relatively new and replaced older emoncms user guide content) I have added these links in and some further notes alongside. I have also added a terminology section, thanks @thundersun @pb66.