IotaWatt Update - nearly complete

Thanks for your answer. Actually I would like to use another microcontroller not the ESP. The Adafruit M0 has the same microcontroller as the arduino zero (SAMD21). Will your software only work on the ESP or could it be easily adapted to other microcontroller? Especially the SD-card logging and upload service?

easily… no.

1 Like

The latest boards have come back and I think at this point I can say that the hardware portion is viable. This “shield” design provides a solid platform and now my attention will move to improving the software, particularly the output side to better interface with the existing services and tools and probably extend the local WiFi interface to support more graphic tools.

The basic IoTaWatt device looks like this:

On top is the Adafruit ESP8266 feather Huzzah. In the middle is one IoTaWatt ADC shield, and on the bottom is the Adafruit datalogger shield with a real time clock and a 16GB SDcard. The shield has seven channels, typically used as one voltage and six current, so you could monitor six circuits with this basic setup.

Add another shield and you get this:

Seven more channels. Any channel can be configured as either a voltage or current channel, but practically speaking, the first channel on each shield has inputs for both voltage (5.5mm barrel) or current (stereo jack), and related circuitry. So now 14 channels, 1 or 2 voltage and 13 or 12 current.

At the risk of revealing my agedness, here’s what I call the Dagwood Sandwich:

Here the third board is populated with screw terminals for the CTs instead of stereo jacks. Personally, I like the jacks, but for most CTs that requires soldering on a male jack or using these:

Finally, here are the components laid out individually:

The three position header on the back is a jumper to select address (0,1,2 = chan 0-6, 7-13, 14-20). The switch selects 3.3v or 1.2v ADC range.

That’s about all there is to it. Oh, and installation/use documentation is in the works.


A lot of the web-interface portions are ESP specific. However the metering and calculations should be adaptable to other MCU’s. I have done this with my i2c based version.

Jon would no doubt say it looks “A-OK.” :grinning:

Ol’ Bumstead built a mean sandwich, eh? :thumbsup:

All kidding aside, it looks great!

That’s interesting. I’ll stand by my “easily… no” reply. For basic sampling, the eMonTX stuff is pretty much the same calculations - without the gather then process approach.

Is there a post describing your efforts? I can’t imagine that i2c ADCs would be very fast, but then again I’ve pretty much had my head down on this and haven’t taken a really good look at the landscape. No matter how you slice it, you need SPI for an SDcard. I’d be curious what MCU you are using. The ESP is so cheap and capable, I haven’t really taken the time to evaluate alternatives. That said, I have my eye on the ESP32 for the next generation.

Looking Great!

Have you thought about taking this to the next level by parsing the signals with energy disaggregation algorithms and identifying individual devices in the household? There’s at least one product on the market that is doing this now but it only uses two CT sensors so it has to disaggregate an entire house. I would think it would be a lot easier and much more accurate if you were to disaggregate individual breakers. For devices that are always on such as a switch or router, an esp8266 based outlet or individual esp8266 with 1 CT to monitor at the device might be an idea.

The disaggregation concept could also be applied to household water flow. I’ve seen clamp-on ultrasonic flow devices that could be used. In line devices could be used as well provided they can measure low flow accurately. Should be easier to do too, since the number of devices that use water is tiny compared to electrical devices.

Just throwing out some ideas. My brain is full of these ideas, but I just don’t have the software and circuit knowledge to turn them into a product myself.

Not really. I don’t know what’s under the hood of the Sense monitor, but they talk about crazy high sample rates. Judging by the resumes of the developers, I’m guessing they are doing some fancy continuous signal processing to identify the various appliance profiles. Their background is speech recognition so I think it’s like Siri picking out your words from the background noise. Takes a lot more processing power and better sampling than IotaWatt has.

I guess it’s cool to have your phone tell you when your laundry is done or if you left some lights on. Not sure if it can break down usage (as opposed to use) as well as discrete circuit monitoring.

All that said, they will probably be able to do some other worthwhile things like identify problems with devices, possibly even diagnose.

I don’t think the technology is applicable to residential water usage in any meaningful way.

You may be pleasantly surprised at what you can do with visual inspection of your per-breaker power graphs once you get to know them a bit. I’ve had per-breaker monitoring for a while and have become pretty good at recognising the relevant patterns.

One of my breakers (GPOs2) is basically all the outdoor power outlets including the laundry in the basement. It’s very easy to glance at the graph for it before I decide whether to head down to the basement and hang the washing out. It’s got a clear wash cycle followed by 4 rinse/spin cycles, the last one the biggie:

That low-level square wave ticking away in the background is the circulation pump on the solar thermal hotwater heater.

Of course, it’s a much harder problem to automate all that. Rather than have it email me to tell me the laundry is done, I just check the graphs with the human eye/brain. I have automated some fault-detection on other circuits though. For example, I do generate automated emails when the pool pump is rotor-locked, or cavitating due to water starvation.

Yeah, I can certainly pick out the various loads on a circuit at a glance, but that’s been the case with power monitors for years. Sense says they take a “million” samples per second. If that’s true, they are not using a general purpose core to process it. They must be running it through signal processing hardware and harvesting specific information about the signal, rather than the raw signal itself. It’s a completely different approach.

That said, they also have a completely different approach to presenting their data. It has strong points and weak points. With respect to utility and competition in the marketplace, I’ll relate a short story that I was taught as a young teen:

I was working for a guy who sold ice cream out of a few seasonal shacks. One spring, as I was cleaning up one of them for opening, I remarked to him how a big chain ice cream business had built across the street. He said it would increase his business. “Now people driving by won’t be deciding if they want to stop for a cone, they’ll be deciding which place to go to.” He was right, the place flourished.

As these devices become ubiquitous, there is space for many approaches.

Over the last few days Mother Nature provided my household IoTaWatt with a real world test. On Tuesday Mar 14th New England was hit by a major winter storm. What we call a Nor’ easter because the low pressure system travels up the coast over the warm waters of the Gulf Stream, and the CCW circulation dumps that ocean moisture over the coastal landmass in the form of heavy snow with high winds. But I digress - long story short, my cable internet cable went down. For two and a half days.

Good news, electric power in my town stayed up (we have a municipal electric system that is very reliable). As the storm raged, and in the days afterward, the IoTaWatt in my basement continued to monitor and record usage, tucking the data away on the SDcard. It did struggle to sample because the WiFiClient blocks for relatively long periods every time it tried to connect. More on that in a later post.

Last night, at about 2:00am, the internet link came back. It took about 15 minutes for IoTaWatt to bulk upload the backlog of data. When I woke up this morning, I took a look at both the IoTaWatt graph of power over the period and my eMonCms power feeds for the same period. Both were complete.

I have two revenue grade meters in my basement. One measures energy to my minisplit heat-pump, the other my electric hot water heater. The meters are made by EKM. The heat pump is a reactive load with a typical power factor of .89. The hot water is unity power factor. I also keep track of my outdoor mechanical wheel type electric meter vs total-power from two mains CTs (US split-phase).

For the last 11 days, including the internet outage period, the IoTaWatt was within 1% of the heat-pump meter and for the last 5 days the hot water was within 2% (10.4Kwh vs 10.2 - primarily solar thermal, electric just tops off the solar yield). For the 11 days, the IoTa was within 2% of the utility meter.

There were some issues, both with IoTaWatt and eMonCMS, and they will be addressed. But overall, very satisfied with the performance through this event.


@trystanlea you may be interested in this. Although the power feed was 90% complete, the Kwh feed was flatlined for the period that was bulk updated. I don’t have a detailed log of the bulk upload transactions, but have a high degree of confidence that they were presented in ascending chronological order. There is really no difference in the way IoTaWatt processes and send the data from it’s local SD log to the eMonCMS feed except that it happens a lot faster with long bulk transactions.

Apparently the transactions were accepted and parsed by eMonCMS as indicated by the successful update of the power feeds. Somewhere after that things got lost. Most of my input process lists “log to feed” followed by “Power to Kwh”.

Here’s the eMonCMS simultaneous plot of power and Kwh. I turned on “show missing data” to illustrate some holes in the power feed, which you can see is only 90%. These updates are completely synchronous. I always wait for the “ok” before sending more. Is it possible there are race conditions internal to eMonCMS after the transactions are queued?

Here is the same plot directly from IoTaWatt and it’s own SDcard log:

There are no holes, and reviewing the CSV data, there are no nulls which would indicate there was no sampling during a period.

I could probably set up a testing scenario to replicate this kind of transaction barage if you want to take a look at it. Or it will be easy for you to do with your IoTaWatt when you get it going.

Thank @overeasy, I think I have spotted the source of this bug. On the standard install of emoncms $time_now is set by the input in the power_to_kwh processor:


but on its being set using php time(), its therefore setting $kwh_inc = 0; which explains the graph your getting.


I need to double check this and ensure that there will not be any unintended consequences of making the change but I think that may well be the cause, thanks for highlighting it.

How large a bulk upload do you do per request when your catching up after several days say?

I cut it off when the entire URL string is >= 1000. The number of update periods and span of time that includes is a function of the number of input channels being reported and the posting interval.

In this case, I have 15 channels at 10 second intervals. So say about 50 characters per interval, say 60 or 70 for the basics (path, apikey, time) and it comes to roughly 18 or 19 entries covering a period of about 180 seconds.

This is an important issue because while during recovery the URL can by up to 1000 characters, the program has a bulk-send attribute that causes it to aggregate the specified number of postings into a single bulk post during regular operation. Setting that to one = real time updates while I think I’m running at three right now so I only send you a transaction every 30 seconds. Conceivably someone could set the bulk to say 18 and end up with the same scenario as recovery every 3 minutes.

The bulk feature makes real-time widgets lag a bit, but I get my local real-time right from the horses mouth. It has the potential to unload network traffic significantly.

@overeasy great thanks

As I see it adafruit have the Adafruit Feather HUZZAH with ESP8266 WiFi but are the only supplier of the Adalogger FeatherWing - RTC + SD Add-on For All Feather Boards. It looks like they are now $25.90 for the pair and $16.40 for international shipping. I have been waiting for ages for the Adalogger to come back into stock which it finally has today.

In the meantime I have been doing some reading and I don’t want to be critical of Bob’s tremendous development effort in any way but a few dumb questions have lodged in my head:

  1. I understand the SD card is a physical connection so any replacement should work. eg This Micro SD card module for 39c.

  2. The real time clock (PCF8523) is no so common but it doesn’t seem so impressive either. A DS3231 RTC Module for $1.28 would probably be more accurate but the programming library would be different. How much effort would it be for the software to support this or a similar module?

  3. I understand that ongoing power issues with the ESP8622 DevBoard which led to the AdaFruit Feather ESP8266 being adopted. What doesn’t the $2.70 ESP-12F Module have that the IotaWatt needs?

  4. One application is a pump shed. Can the NodeMCU ESP8266 devkit still be used with the shields getting it to reset periodically to work with the power issues?

The main thing that I like about the first two options is that they are not tied to a single supplier.

I went with the ADAfruit feather system for the 2.0 version to get the SD and RTC as part of a modular system. Although the ESP is a bit more than the street price for nodeMCU, in the big picture its really still cheap for what it is compared to something like an Arduino. Even the SainSmart duinos (which are fine) are still relatively expensive. [quote=“flywire, post:37, topic:2989”]
$16.40 for international shipping

My biggest disappointment with AdaFruit. They charge me around $13 and I live less than 200 miles from them. I have complained with no response.

Yeah, the SD is just an SPI device. The Adalogger is really just a connector. Just make sure the Clk, Miso, Mosi, and CS (gpio15) are connected along with 3.3v and GND. Be careful because the pinout of the microSD is different from the full size.

The library supports both the DS3231 and the DS1307, so it would be trivial to adapt to them. Accuracy isn’t an issue. The AC synchronization is handled by the ESP millisecond clock. The RTC, while an important part of any datalogger, needs only to be within a few seconds, and IotaWatt checks the NTP server every hour and adjusts.

There are no power issues that I am aware of with the nodeMCU 0.9 or 1.0. My house runs on a 1.0 and has been running uninterrupted for the past two weeks. The ADC boards could easily be modified to accept the nodeMCU, or an adapter board could be made pretty inexpensively (maybe with an RTC and SDsocket).[quote=“flywire, post:37, topic:2989”]
One application is a pump shed. Can the NodeMCU ESP8266 devkit still be used with the shields getting it to reset periodically to work with the power issues?

Once again, there are no powr issues, but the IotaWatt reboots in about 15 seconds, fills in the log gaps, and keeps right on going pretty much uneventfully.[quote=“flywire, post:37, topic:2989”]
The main thing that I like about the first two options is that they are not tied to a single supplier.

I like that too. That’s why it’s open. Hopefully this will catch on when I finish the software and there will be a few hardware options.

Hi bob - was curious why you never used the the built in ADC. ( or maybe you did and did not like the results)

i built something similar to yours. only based on the emontx form factor- inter changeable software and hardware from arduino to esp ( wemos d1) the only difference I guess Is i used the onboard ADC to do the voltage calculation and it left a full 8 ADC to stack up to 4 units or what ever number of free CS your board supports… 32 cts on a esp and 36 ct if used on adruino and enmontx sheild. ( do not have 36 ct so i can not attest to whether it works well with 36 ct s but it will support it _ I only have 12 cts - but it works fine with those)

but yours looks nicer then my hand built ones

Several reasons:

  • One isn’t enough.
  • It has a 1 volt limitation.
  • Not sure how accurate the 1v reference is
  • Probably nowhere near as fast as the MCPs (though I don’t know that).
  • Wanted to support multiple voltage inputs for three phase and more accurate split phase monitoring.
  • Had to do a lot of experimentation to learn how to use the MCP fast and accurately, another piece of hardware would have doubled that.
  • Doesn’t really do anything significant to the cost but increases the software complexity.

All that said, I’m starting to use the ESP32 and it has 16 ADC channels (if you want to give up some digital pins). I may take advantage of them.

Latest version coming along well. Now supports dynamic configuration so you never have to restart.

Also now has output channels that are functions of input channels, much like eMonCMS inputs. The output channels are available in the local graphing application.

Output channels are configured with a “calculator” like interface:

There’s a demo version at that you can play around with. It has limited functionality but demonstrates the look and feel.

I’ve received quite a few inquiries, from here and GitHub, for hardware. Other efforts have not panned out, so I am moving ahead to make hardware available as soon as possible.