Looking at ways to make them available. Realistically at least a few months away.[quote=“Simsala, post:18, topic:2989”]
(I try to use the integrated 12bit ADC),
[/quote]
I don’t know much about the onboard ESP8266 ADC. The software requires at least two ADC channels to read power (one voltage reference and one current reference), so the ESP alone can’t do it. If some other ADC were used, the ADC sampling is localized and could be adapted.
Good luck.
@overeasy this is looking really nice! I wanted to check that I understand correctly, do you sample each CT channel sequentially, sampling voltage and current 498 times in one 60Hz cycle 16.6 milliseconds, so 996 samples in 16.6 milliseconds or 59,760 Hz?
Why is it 35.1 AC cycles sampled/second rather than 60 AC cycles sampled/second for 60Hz AC? Is the difference due to time to switch to another CT channel?
Correct.
Basically - yes. To get the maximum samples during a cycle, the samples are just stored. No floating point arithmetic, nothing complicated, just grab samples as fast as possible. After the last zero crossing, a few extra samples are taken to accommodate phase correction. Then the samples are processed, which takes about a millisecond. So it’s too late to start another cycle and the next crossing doesn’t come around for 6 or 7 milliseconds. That’s when the device does everything else. Checks the WiFi server, writes to the SDcard, sends data to a server like eMonCms. There is a scheduler and the communication and logging is done by discrete services that are scheduled and dispatched during that period.
So each sample event takes a minimum of 1.5 cycles. The limit is therefore 40 cycle samplings per second at 60Hz. It would be 33.3 in a 50Hz world. Still, 1.5 - 2 samples each second per channel (20 max).
Thanks for your answer. Actually I would like to use another microcontroller not the ESP. The Adafruit M0 has the same microcontroller as the arduino zero (SAMD21). Will your software only work on the ESP or could it be easily adapted to other microcontroller? Especially the SD-card logging and upload service?
easily… no.
The latest boards have come back and I think at this point I can say that the hardware portion is viable. This “shield” design provides a solid platform and now my attention will move to improving the software, particularly the output side to better interface with the existing services and tools and probably extend the local WiFi interface to support more graphic tools.
The basic IoTaWatt device looks like this:
Add another shield and you get this:
At the risk of revealing my agedness, here’s what I call the Dagwood Sandwich:
That’s about all there is to it. Oh, and installation/use documentation is in the works.
A lot of the web-interface portions are ESP specific. However the metering and calculations should be adaptable to other MCU’s. I have done this with my i2c based version.
Jon would no doubt say it looks “A-OK.” 
Ol’ Bumstead built a mean sandwich, eh? 
All kidding aside, it looks great!
That’s interesting. I’ll stand by my “easily… no” reply. For basic sampling, the eMonTX stuff is pretty much the same calculations - without the gather then process approach.
Is there a post describing your efforts? I can’t imagine that i2c ADCs would be very fast, but then again I’ve pretty much had my head down on this and haven’t taken a really good look at the landscape. No matter how you slice it, you need SPI for an SDcard. I’d be curious what MCU you are using. The ESP is so cheap and capable, I haven’t really taken the time to evaluate alternatives. That said, I have my eye on the ESP32 for the next generation.
Looking Great!
Have you thought about taking this to the next level by parsing the signals with energy disaggregation algorithms and identifying individual devices in the household? There’s at least one product on the market that is doing this now but it only uses two CT sensors so it has to disaggregate an entire house. I would think it would be a lot easier and much more accurate if you were to disaggregate individual breakers. For devices that are always on such as a switch or router, an esp8266 based outlet or individual esp8266 with 1 CT to monitor at the device might be an idea.
The disaggregation concept could also be applied to household water flow. I’ve seen clamp-on ultrasonic flow devices that could be used. In line devices could be used as well provided they can measure low flow accurately. Should be easier to do too, since the number of devices that use water is tiny compared to electrical devices.
Just throwing out some ideas. My brain is full of these ideas, but I just don’t have the software and circuit knowledge to turn them into a product myself.
Not really. I don’t know what’s under the hood of the Sense monitor, but they talk about crazy high sample rates. Judging by the resumes of the developers, I’m guessing they are doing some fancy continuous signal processing to identify the various appliance profiles. Their background is speech recognition so I think it’s like Siri picking out your words from the background noise. Takes a lot more processing power and better sampling than IotaWatt has.
I guess it’s cool to have your phone tell you when your laundry is done or if you left some lights on. Not sure if it can break down usage (as opposed to use) as well as discrete circuit monitoring.
All that said, they will probably be able to do some other worthwhile things like identify problems with devices, possibly even diagnose.
I don’t think the technology is applicable to residential water usage in any meaningful way.
You may be pleasantly surprised at what you can do with visual inspection of your per-breaker power graphs once you get to know them a bit. I’ve had per-breaker monitoring for a while and have become pretty good at recognising the relevant patterns.
One of my breakers (GPOs2) is basically all the outdoor power outlets including the laundry in the basement. It’s very easy to glance at the graph for it before I decide whether to head down to the basement and hang the washing out. It’s got a clear wash cycle followed by 4 rinse/spin cycles, the last one the biggie:
Of course, it’s a much harder problem to automate all that. Rather than have it email me to tell me the laundry is done, I just check the graphs with the human eye/brain. I have automated some fault-detection on other circuits though. For example, I do generate automated emails when the pool pump is rotor-locked, or cavitating due to water starvation.
Yeah, I can certainly pick out the various loads on a circuit at a glance, but that’s been the case with power monitors for years. Sense says they take a “million” samples per second. If that’s true, they are not using a general purpose core to process it. They must be running it through signal processing hardware and harvesting specific information about the signal, rather than the raw signal itself. It’s a completely different approach.
That said, they also have a completely different approach to presenting their data. It has strong points and weak points. With respect to utility and competition in the marketplace, I’ll relate a short story that I was taught as a young teen:
I was working for a guy who sold ice cream out of a few seasonal shacks. One spring, as I was cleaning up one of them for opening, I remarked to him how a big chain ice cream business had built across the street. He said it would increase his business. “Now people driving by won’t be deciding if they want to stop for a cone, they’ll be deciding which place to go to.” He was right, the place flourished.
As these devices become ubiquitous, there is space for many approaches.
Over the last few days Mother Nature provided my household IoTaWatt with a real world test. On Tuesday Mar 14th New England was hit by a major winter storm. What we call a Nor’ easter because the low pressure system travels up the coast over the warm waters of the Gulf Stream, and the CCW circulation dumps that ocean moisture over the coastal landmass in the form of heavy snow with high winds. But I digress - long story short, my cable internet cable went down. For two and a half days.
Good news, electric power in my town stayed up (we have a municipal electric system that is very reliable). As the storm raged, and in the days afterward, the IoTaWatt in my basement continued to monitor and record usage, tucking the data away on the SDcard. It did struggle to sample because the WiFiClient blocks for relatively long periods every time it tried to connect. More on that in a later post.
Last night, at about 2:00am, the internet link came back. It took about 15 minutes for IoTaWatt to bulk upload the backlog of data. When I woke up this morning, I took a look at both the IoTaWatt graph of power over the period and my eMonCms power feeds for the same period. Both were complete.
I have two revenue grade meters in my basement. One measures energy to my minisplit heat-pump, the other my electric hot water heater. The meters are made by EKM. The heat pump is a reactive load with a typical power factor of .89. The hot water is unity power factor. I also keep track of my outdoor mechanical wheel type electric meter vs total-power from two mains CTs (US split-phase).
For the last 11 days, including the internet outage period, the IoTaWatt was within 1% of the heat-pump meter and for the last 5 days the hot water was within 2% (10.4Kwh vs 10.2 - primarily solar thermal, electric just tops off the solar yield). For the 11 days, the IoTa was within 2% of the utility meter.
There were some issues, both with IoTaWatt and eMonCMS, and they will be addressed. But overall, very satisfied with the performance through this event.
@trystanlea you may be interested in this. Although the power feed was 90% complete, the Kwh feed was flatlined for the period that was bulk updated. I don’t have a detailed log of the bulk upload transactions, but have a high degree of confidence that they were presented in ascending chronological order. There is really no difference in the way IoTaWatt processes and send the data from it’s local SD log to the eMonCMS feed except that it happens a lot faster with long bulk transactions.
Apparently the transactions were accepted and parsed by eMonCMS as indicated by the successful update of the power feeds. Somewhere after that things got lost. Most of my input process lists “log to feed” followed by “Power to Kwh”.
Here’s the eMonCMS simultaneous plot of power and Kwh. I turned on “show missing data” to illustrate some holes in the power feed, which you can see is only 90%. These updates are completely synchronous. I always wait for the “ok” before sending more. Is it possible there are race conditions internal to eMonCMS after the transactions are queued?
Here is the same plot directly from IoTaWatt and it’s own SDcard log:
There are no holes, and reviewing the CSV data, there are no nulls which would indicate there was no sampling during a period.
I could probably set up a testing scenario to replicate this kind of transaction barage if you want to take a look at it. Or it will be easy for you to do with your IoTaWatt when you get it going.
Thank @overeasy, I think I have spotted the source of this bug. On the standard install of emoncms $time_now is set by the input in the power_to_kwh processor:
"https://github.com/emoncms/emoncms/blob/master/Modules/process/process_processlist.php#L290
but on emoncms.org its being set using php time(), its therefore setting $kwh_inc = 0; which explains the graph your getting.
"https://github.com/emoncms/emoncmsorg/blob/master/emoncms/Modules/input/process_model.php#L265
I need to double check this and ensure that there will not be any unintended consequences of making the change but I think that may well be the cause, thanks for highlighting it.
How large a bulk upload do you do per request when your catching up after several days say?
I cut it off when the entire URL string is >= 1000. The number of update periods and span of time that includes is a function of the number of input channels being reported and the posting interval.
In this case, I have 15 channels at 10 second intervals. So say about 50 characters per interval, say 60 or 70 for the basics (path, apikey, time) and it comes to roughly 18 or 19 entries covering a period of about 180 seconds.
This is an important issue because while during recovery the URL can by up to 1000 characters, the program has a bulk-send attribute that causes it to aggregate the specified number of postings into a single bulk post during regular operation. Setting that to one = real time updates while I think I’m running at three right now so I only send you a transaction every 30 seconds. Conceivably someone could set the bulk to say 18 and end up with the same scenario as recovery every 3 minutes.
The bulk feature makes real-time widgets lag a bit, but I get my local real-time right from the horses mouth. It has the potential to unload network traffic significantly.
As I see it adafruit have the Adafruit Feather HUZZAH with ESP8266 WiFi but are the only supplier of the Adalogger FeatherWing - RTC + SD Add-on For All Feather Boards. It looks like they are now $25.90 for the pair and $16.40 for international shipping. I have been waiting for ages for the Adalogger to come back into stock which it finally has today.
In the meantime I have been doing some reading and I don’t want to be critical of Bob’s tremendous development effort in any way but a few dumb questions have lodged in my head:
I understand the SD card is a physical connection so any replacement should work. eg This Micro SD card module for 39c.
The real time clock (PCF8523) is no so common but it doesn’t seem so impressive either. A DS3231 RTC Module for $1.28 would probably be more accurate but the programming library would be different. How much effort would it be for the software to support this or a similar module?
I understand that ongoing power issues with the ESP8622 DevBoard which led to the AdaFruit Feather ESP8266 being adopted. What doesn’t the $2.70 ESP-12F Module have that the IotaWatt needs?
One application is a pump shed. Can the NodeMCU ESP8266 devkit still be used with the shields getting it to reset periodically to work with the power issues?
The main thing that I like about the first two options is that they are not tied to a single supplier.
I went with the ADAfruit feather system for the 2.0 version to get the SD and RTC as part of a modular system. Although the ESP is a bit more than the street price for nodeMCU, in the big picture its really still cheap for what it is compared to something like an Arduino. Even the SainSmart duinos (which are fine) are still relatively expensive. [quote=“flywire, post:37, topic:2989”]
$16.40 for international shipping
[/quote]
My biggest disappointment with AdaFruit. They charge me around $13 and I live less than 200 miles from them. I have complained with no response.
Yeah, the SD is just an SPI device. The Adalogger is really just a connector. Just make sure the Clk, Miso, Mosi, and CS (gpio15) are connected along with 3.3v and GND. Be careful because the pinout of the microSD is different from the full size.
The library supports both the DS3231 and the DS1307, so it would be trivial to adapt to them. Accuracy isn’t an issue. The AC synchronization is handled by the ESP millisecond clock. The RTC, while an important part of any datalogger, needs only to be within a few seconds, and IotaWatt checks the NTP server every hour and adjusts.
There are no power issues that I am aware of with the nodeMCU 0.9 or 1.0. My house runs on a 1.0 and has been running uninterrupted for the past two weeks. The ADC boards could easily be modified to accept the nodeMCU, or an adapter board could be made pretty inexpensively (maybe with an RTC and SDsocket).[quote=“flywire, post:37, topic:2989”]
One application is a pump shed. Can the NodeMCU ESP8266 devkit still be used with the shields getting it to reset periodically to work with the power issues?
[/quote]
Once again, there are no powr issues, but the IotaWatt reboots in about 15 seconds, fills in the log gaps, and keeps right on going pretty much uneventfully.[quote=“flywire, post:37, topic:2989”]
The main thing that I like about the first two options is that they are not tied to a single supplier.
[/quote]
I like that too. That’s why it’s open. Hopefully this will catch on when I finish the software and there will be a few hardware options.
