Appologies, I thought you were after the Emoncms API docs rather than IotaWatt directly. @overeasy should be able to help. IotaWatt docs are work in progress.
I notice this is for local feeds, does IotaWatt also update emoncms.org with the same format or is it not available on emoncms.org?
Edit:
I see from the Wiki at Home · boblemaire/IoTaWatt Wiki · GitHub that IotaWatt can upload to EmonCMS.org. Does that mean data can be retrieved using the regular (ms start / stop) API?
This is an adaptation of a similar facility available on Emoncms. It has been modified to better adapt to the way IoTaWatt maintains it’s time series data. So the answer is that you can get the same data from Emoncms (assuming feeds are defined at the interval you need). Without getting into the specific syntax, the main difference is that the Emoncms API is for a single feed, and the IoTaWatt request can return multiple feeds. That means if you are interested in multiple feeds, you would need to make multiple requests of Emoncms. While that may sound burdensome, Emoncms is much faster and can service several requests in the time it takes IoTaWatt to service those requests in a single call - about 4 seconds.
We crossed in the mail on this. Yes, you can upload to Emoncms and it’s exactly the same as data uploaded using any of the OEM products. One signoificant difference though is that IoTaWatt will recover from most WiFi or internet outages and upload the data accumulated during the outage.
1 Like
@overeasy the problem we have is that the servers we use for our Smartphone app are set with a minimum of 15s between each API call. We can switch servers and drop the 15s to 1s but we would prefer to stay with the standard servers.
So this means:
- Only “whole house” monitoring on a single channel would be used.
- To obtain say voltage, realPower, apparentPower and current would take 1 minute and they would be out of “phase” with each other by 15s to 45s.
We wouldn’t want to go through port forwarding to access the local feeds and the extended API.
@glyn.hudson is it likely that the standard API for EmonCMS.org will be upgraded to cover IotaWatt in the near future?
I think we’re talking about two different things here. I understood you to be asking about the ability to query the IotaWatt for data, and then about the existence of a similar capability on Emoncms.
This appears to pertain to the communication between IotaWatt and Emoncms when uploading data.
IotaWatt uploads everything in a single transaction. In fact, it can be set to aggregate several measurement intervals and send them in a single transaction. So for instance, you can monitor all 14 channels and upload three 10 second observations of those 14 channels in a single transaction every 30 seconds.
The data can be sent securely with encryption and it is a synchronous posting so that if an acknowledgement is not received, thedata will be retried until successful.
All of the data that is uploaded to Emoncms is timestamped, so there is never an issue of data being out of phase on sending or retrieving. IotaWatt uploads an exact representation of it’s own internal log with a timestamp.
@overeasy for other OEM hardware we pull the data directly from the hardware to use in our Smartphone app. As the IotaWatt has no external interface i.e. no spare pins, then we are left with pulling the data from EmonCMS.org.
Your upload and aggregation of data to EmonCMS is fine it’s just our ability to pull down the data at 15s intervals from the server that’s the problem.
If EmonCMS used the same API as you use locally then we could pull down simultaneous data feeds, rather than “phase” shifted feeds. Phase has nothing to do with electricity in this context.
I get the context of phase here. The data is time stamped, so whether you get it in one transaction or several, it is the same data from the same time interval. Don’t know if that works for you.
Seems to me that if you could be close enough to the iotawatt to read a serial stream, you should be able to get the data through local WiFi, but you seem to have a specific idea.
Ah yes it would. Bit of extra coding as I was expecting to use “now” but providing I hold the same now value for 3 or 4 API calls it would be OK.
An ESP attached to the OEM hardware interfaces with an online server. The Smartphone app interfaces with the same online server. It appears the app is connected to the ESP but it’s not directly connected, works without all that port forwarding nonsense.
Hi Bob,
I was wondering what plans you have with regards to refactoring the code for IoTaWatt?
You mentioned that you wanted to refactor the output services and I also saw you working on the log format before.
I got a pvoutput output service working for my purposes, though one thing that was missing from the IoTaLog was a “count” of the number of samples in a given log record. There is a logHours member, which is good for obtaining power from the accumulated energy for a power channel, but I also want to report mean Voltage over the period from the voltage channel (reported up to pvoutput).
If my reading of the code is correct, to do this I want to basically divide the accum1 for the voltage channel by the count of samples accumulated in it. Is this a reasonable thing to add to the IoTaWatt log?
For now I am just reporting the current voltage from the input channel directly so it doesn’t work with historic information from the IoTaLog.
Also, in implementing the output service, I couldn’t help but see how similar all the output services are. I wonder if you have plans to make a single more generic HTTP output service that can just be configured for each use case?
Hi Brendon,
Work on the datalog is ongoing, but it is work on the file management class, not the content. The basic datalog record will not change, and the format and organization of the file will not change. There is a limitation of 2^31 filesize imposed by the FAT32 filesystem, which translates to about 1.3 years of continuous logging at 5 second intervals. I have code in test that wraps the file when it is full. I’ve designed and will implement a lower granularity history file to keep the data at 1 min intervals for up to 16 years.
Err, no. I’d recommend you take a closer look at how all that works before redesigning it. If I understand what you are asking, I think the current API does what you want. I’ll explain a few concepts to get you going into a better understanding:
Data point kept in the datalog are always time weighted. That is to say the power channels are logged as the cumulative watt-hours since the log was created, and voltage channels are logged as the cumulative volt-hours. So you’re wondering what the heck is a volt-hour? Simply put, it’s exactly the same as watt-hours, and can be used in the same way.
The key is that each log record also has the cumulative hours of logging since the beginning of time. So if you want to know the average watts of a channel between any two log entries, you simply divide the difference of watt-hours by the difference of hours. Voila! average watts. You can get the average watts used over 5 or 10 seconds, or a minute, or a day, or a year. Any interval for which you can get the bracketing log entries.
Voltage is the same. If you want to know the average voltage for the last hour, just divide the delta volt-hours by the delta hours.
If you look deep in the code, you will see that the same logic is applied at a sub-second level. That is, measurements of a particular channel are occurring multiple times per second. To get an accurate average of the channel, the low level code maintains volt-milliseconds and watt-milliseconds. When it comes time to make a log entry, this cumulative value is divided by the number of milliseconds since the last log entry. If you think about it, I can’t guaranty that the log entries will be made exactly every 5000ms, nor can I sample every channel instantaneously when it’s time to make a log entry. So doing it this way maintains the integrity of the average for the basic log entry, as well as over days, months and years. You will also find that the cumulative hours in each log entry do not evenly increment by 5/3600. It is the fractional part of an hour representing the number of milliseconds since the last log entry. It’s close, but not the same.
In pondering your question I realize that there are some boundary issues that I need to check, but basically, I can’t think of any information in the log that isn’t readily available using the existing API. I think it is a “single generic HTTP output service”. After you digest what I just explained, if you still feel there is a need for something more, tell me what it is and I’ll either explain how to get it using this API, or consider an enhancement.
When you think you understand how it works, ask yourself these questions:
-
What should the API return when you request an entry that does not exist in the log? (Like during a power failure).
-
How would you develop the 1 minute entries for a history log using the datalog?
-
How does an hour long power failure affect the average reading for a complete day?
Hi Jan,
Was wondering the status of your three phase installation. Is it working and do you have any feedback on accuracy?
Currently i am running 2 IotaWatts. I’m quite happy with the measurements. I haven’t actually measured the accuracy but it’s in the ballpark. Now that you mention it: I’ll do do some tests with a heater or lightbulb in the next week or so.
The only 2 things i found strange is
- my water heater (for the sink at my wife’s practice room) seems to have a low pf (only 0.55 with 800W measured while listed at 2000W - but it’s on phase 1 and 1.04 degrees off…)
- the phase error of the breaker with the ups behind it seems to be off (calculated it like you showed) and it seems to be 81.6 degrees off, while the rest is around 0.5 to 1.7 degrees off… . Behind this breaker is a ups (MGE ellipse 750) for my modem/switch/raspberry pi/…
That sounds like a phase problem. Using the wrong voltage/phase reference would be low by cos 120 or .5, PF for a unity device on the wrong phase should also be .5, so it sounds like that’s what might be wrong here. Is the water heater wired to one phase or two phases? if it’s two, that would also be .5 with respect to one or the other if I got the math right. If it is wired to two of the phases, you would need to use two CTs, one on each, and add them together to get total power.
@overeasy : Found the problem with the heater: it was labeled wrong by the electrician who set it up 5 years ago 
it’s not on phase 1 but phase 3… I could not test it yet because the Iota that has it connected is not yet connected to phase 3. (waiting for my splitter).
I did some readings on the other Iota and i have some strange results. I am hoping to have some more fee time to measure some more next weekend.
I was wondering: some of my ct’s are reporting “inverted” (in red). As far as i remember i installed them correctly, like the rest. (but i’ll have to open up the case to check) Could it be that a inverted ct is resulting in a phase shift between the CT and VT? (around 80 degrees - measurement is around 130 degrees with a Pf of approx 0.65)
Another thing i ran into: i was using the outputs page to get the powerusage (W) and Pf, but i saw that the Iota passes it to the webpage when power usage is more than 10W, but the page only shows it when power usage is more than 60W. Is there a particular reason to only show pF above 60W? (i wronly supposed it was pF 1 and that resulted in a calculated phase error )
Hard to say what it is, there are so many variables here, but the red indicator means the power is calculating negative. if you are confident in your orientation (it’s easy to mess that up) then I would be suspicious of the wrong phase again - the cosine of both 120 and 240 is -0.5. Once again, I’m no expert in this but trying to figure it out. IotaWatt is measuring 130 degrees phase difference between the VT and CT signals. The cosine of 130 is .642, so I think that’s where your PF is coming from. I’d try comparing to the other VT, and if you get the same results, wait and try it with the last phase once you get it connected.
Basically it’s an arbitrary threshold based on the idea that as power goes down, the accuracy of the PF calculations diminishes. At 60 watts at my development site, current is 500ma. It’s 250ma in the 240V world. IotaWatt computes it for anything above 10 watts and sends it out in the status response. The js in the app filters anything below 60. Feel free to change it in the app, for what it’s worth. I do think you are relying on the PF too much to sort out your phase assignments. a PF of .65 is a little outside the range for most household stuff. I rarely see below .80.
Ruminating on these setup problems, I had the idea that if you have a ct on each of the three mains, you could us3 the graph app to Plot the power for the all three and then add a questionable load ct. when the load switches on or off, it should be obvious which phase it is associated with.
I have a branch created from recent IoTaWatt master in github in which I have implemented support for a pvoutput (Help) push service.
It is very similar to the existing services you implemented for InfluxDB and EmonCMS.
I am going to be using this with my homebrew IoTaWatt but want to know if you have any feedback on whether you would be interested in integrating this into official IoTaWatt firmware and if so are there any changes you would like me to make to this patch?
I am currently doing some more detailed testing of this patch and any necessary cleanup but it is in a reasonable state at the moment.
I was also planning to document steps required to create a PVOutput account, obtain an API key and configure IotaWatt to push data to that account. But that will happen after testing only if you are interested in this service.
I also won’t be offended if you intend to re-write this service yourself later. I just wanted it to monitor my home system now.
1 Like
[quote=“bcosta, post:98, topic:4932”]
I have
[/quote]Hi Brendon,
I looked at pvoutput after you (I think) posted about it recently. Made a note to add this when I got a chance. Right now I have no internet except for a cellular iPad mini, so I’m getting a little backed up.
I looked briefly at your code, and it is very much modeled after the other handlers. There were a couple of things extra that I’d probably do differently, like using the initialize switch, and keeping three log records worth of heap allocated, but overall, a good framework.
There are some communication problems that I need to resolve, and the results will also be applicable to this. When I get that done, I can maybe PR your branch with those changes and get this ready to come back into the master branch.
In the meantime, sounds like you have what you need.
Regards, Bob
You are doing well to keep up with everything after the release with limited internet access.
Thanks for the feedback. I will improve the memory usage for the log record objects. I only use three of the channels from the old records so I can just store data from those three channels to improve this.
As for heap vs static, I copied the original Influx db code for this which used the heap for larger objects. I believe this is so that you don’t pay as big a cost in “overall” memory usage for a service that is not in use. I.e. Few objects allocated in static scope when service is unused.
Would you still be ok to consider this patch if it differs from the other services and use a single class to store the data for the service which will be allocated on the heap when the service is in use? It seems a better solution than a lot of statics and a few static pointers.
I dont understand what you mean by the “initialize switch”. The service initialization code comes directly from the influx service code. Can you please describe this more to me so I can address it?
In the end this service works for me but I like to contribute where I can if it is helpful.
Thanks,
Brendon.