Looking for power outages

I’m having a lot of circuit-breaker trips. I’ve spent a month trying to track it down, but it keeps happening irrespective of which device I disconnect. I’ve asked an electrician to come and do fault-finding.

I want to help them by describing when the outages occur. Maybe I can even describe the power usage, humidity and temperature around those events to provide extra context.

I could do this from my file system cache of the CurrentCost readings, but I feel like playing with EmonCMS.

I’m looking for when the readings are missing or are zero. I’ve tested this script with some recent outages, but now I’m running it from a long time ago to see when things started misbehaving.

Unfortunately I’m only getting about 10 requests a second from my local EmonCMS install which is on a very beefy server, but running in Docker containers. I do know what I’m doing, I have been using Docker in production websites for over five years.

Luckily it’s fine, I’m only doing something like a million requests which will be done in a day. I can wait.

I just wondered if others are using different approaches, perhaps using data.json on the feed API endpoint.

Are you finding the performance benefit is worth the extra complexity of doing the analysis on the client side?

Anyway, here’s a the script so you can have a giggle at my lazy implementation.

import datetime
import tqdm
from typing import Generator
from .emoncms import EmonCMS  # a thing that fetches a single reading

def generate_moment() -> Generator[datetime.datetime, None, None]:
    ''' Generate all the moments we're interested in'''
    start = datetime.datetime.strptime("2021-04-20T00:00:00", "%Y-%m-%dT%H:%M:%S")
    end = datetime.datetime.utcnow()
    moment = start

    while moment < end:
        yield moment
        moment = moment + datetime.timedelta(0, 30)


def is_down_at_moment(moment) -> bool:
    ''' Does the power look like it's gone '''
    try:
        # 385 is my "use"
        reading = EmonCMS.getFeedValue(385, moment)
    except:
        return True
    value = reading["value"]
    if value:
        return False

    return True

def review():
    outage_started = None

    with open("outages.txt", "a") as outages:
        # Convert to list so tqdm can give an ETA
        pbar = tqdm.tqdm(list(generate_moment()))
        for moment in pbar:
            pbar.set_description(moment.isoformat())
            if is_down_at_moment(moment):
                if outage_started == None:
                    outage_started = moment
            else:
                if outage_started:
                    duration = moment - outage_started
                    outages.write(outage_started.isoformat() + "\t" + moment.isoformat() + "\t" + str(int(duration.total_seconds())))
                    outage_started = None

    print("Complete")

# Python-y stuff...
def main():
    review()
if __name__ == "__main__":
    main()

Just as a general comment that I’m afraid may or may not be helpful…

I think most of the “repeated tripping of the circuit breaker” issues I’ve ever had have been down to heating elements of some sort that are dying. Last year it was the heating element (in fact just a 2kW kettle element, I believe) in my honeycomb uncapping tray. Before that it was the grill element in the oven (that’s actually happened a couple of times) and the time before that was the main oven element. I think I once had the hot water cylinder heating element cause it too.

In the case of the capping tray heater, it would actually work for a while and then trip. The grill element only seemed to trip in combination with something else that I could never work out, but replacing it fixed the problem so I stopped looking. Sadly there wasn’t a situation where either reliably tripped the circuit breaker immediately they were switched on.

The only other times I’ve had the circuit breakers trip are sometimes when light bulbs blow, which is fairly obvious, and occasionally when my father-in-law has decided to do something with the electrics live (yeah, I know) and accidentally shorts neutral to earth. That sort of thing is usually pretty clear cut so I’d assume you’d have spotted it already, though hopefully my father-in-law is not in the habit of messing about with your electrics.

As and when you do find out it might be interesting to know, just to file away in the “for future reference” drawer :slight_smile:

James

1 Like

Actually, come to think of it, it could even be a faulty circuit breaker. That shouldn’t be discounted I guess.

James

1 Like

First, establish whether it’s an overcurrent trip or an earth leakage trip.

If it’s an earth leakage trip - it could be a combination of several appliances/circuits, and the last one to take it over the limit causes the trip.

Just to be totally unhelpful, I once came across an earth leakage trip on an oven that hadn’t been used for years - it had just got damp over the winter. Dried out in the summer, switched on and allowed to dry itself thoroughly, it was fine.

And a fire alarm installer once told me he had to dry ‘pyrotenax’ (copper-clad, mineral insulated cable) with a blowlamp, starting from the middle and working outwards to each end, to drive the moisture out. (Presumably not the PVC-sheathed kind :face_with_raised_eyebrow:)

1 Like

Thanks James and Robert.

We’ve had the scenarios you’ve describe happen to us in the past. We’ve changed oven elements and the like. I’ll keep an eye out for your father-in-law.

What’s interesting here is that I think it’s moisture related which is why I’m looking for the outage events to correlate them with recent precipitation (even though I’m confident we’ve seen it when it’s dry).

We went overboard yesterday and checked each socket methodically. We found an un-used one with a bad live connection, you could even hear it arcing. So I fixed that and was crossing my fingers but we still have problems afterwards. I did start to wonder about folks after seeing that someone (over 15 years ago) decided to do a job leaving wires not screwed in because it was hard to get to. So I’ve been taking the other sockets apart to check them and removing the spider’s webs from inside them in case that was causing a trickle to earth. They all look fine.

Last night I was lucky enough that the RCD kept tripping repeatedly when I reset it. So I was able to alter stuff and remove one of the spurs from the back of a socket. We’ve not had any trips since then. Unfortunately, this is the same as last week where eventually I re-connected the spur and it ran fine…well, for a while. Of course this spur is to something interesting; in this case the outdoor freezer so I can’t leave it off. I have to run an extension cable to the outdoor freezer, but from a different circuit which will help with diagnosis. Now of course I’m also wary of all the outdoor stuff anyway, but I had all outdoors disconnected and trips continued to happen.

I think I have a problem in the wiring on that spur, but it’s all under a tiled floor so that’s not amenable to inspection. I know an electrician’s tester will easily spot the problem, but we might struggle if it needs something like rain to make it fail.

It could be a bad RCD, I can’t play with the consumer unit wiring enough to know though. I did move the suspect circuit over to a different overload circuit breaker and the problem still happened. So I believe it’s an earth leakage trip rather than over-current. It could be multiple devices which would cause the problems I’m seeing. I don’t have a way to see this behaviour though with the tools I have so I’d either need to get more or get an electrician, so I’m doing the latter.

A bit like Robert’s oven, for a while it felt like a moisture problem because a few resets would cause it be OK so it felt as though the fault was drying out the problem area for a while. However, it’s been dry a few times when the problem has happened so I’m starting to doubt myself. Hence the script.

I have had things isolated for the last 24 hours and no trips, so the suspect spur is looking more and more suspect.

All this tripping is certainly stress-testing my recovery. Sadly we had Linux doing an update on a machine which went badly (so had to do dnf reinstall *) but the main bit of tediousness is the un-used room thermostat (which is in-circuit for the heat pump) is defaulting to “off” so we have to nudge it after each outage. The heat pump is on it’s own circuit so isn’t under suspicion.

Thanks for your help.

OK, here we go, I made it use data.json and get a whole day at a time. It now runs in about a minute. The old script still reckons it’s got 13 hours to run.

The new logic did cause all sorts of problems. You can’t get all the readings from a year in one call, so I had to iterate over something smaller, in this case days. I also found I was getting no results for some days when my systems were broken so I had to wrap the reading with try...except.

Next thing to do is to look at the outcome. I have a lot more events than I anticipated, even with the 120 second minimum.

import datetime
from typing import Generator
from .emoncms import EmonCMS


def generate_days() -> Generator[datetime.datetime, None, None]:

    moment = datetime.datetime.strptime("2021-04-01T00:00:00", "%Y-%m-%dT%H:%M:%S")
    end = datetime.datetime.utcnow() + datetime.timedelta(1)

    while moment < end:
        yield moment
        moment = moment + datetime.timedelta(1)


def get_readings(start) -> Generator[list, None, None]:
    try:
        for reading in EmonCMS.getFeedValues(feedId=385, moment=start, duration=86400, interval=30):
            yield reading
    except:
        return


def main():
    previous_moment = None

    with open("../review/outages_fast.txt", "w") as outages:
        for day in generate_days():
            for reading in get_readings(day):
                moment = reading["time"]
                if previous_moment:
                    duration = moment - previous_moment
                    if duration.total_seconds() > 120:  # The outage was long enough that it doesn't look like a blip
                        outages.write(previous_moment.isoformat() + "\t" + moment.isoformat() + "\t" + str(int(duration.total_seconds())) + "\n")
                previous_moment = moment

    print("Complete")


# ----------------------------------------
if __name__ == "__main__":
    main()

Here’s the EmonCMS code in case that’s interesting one day:

class EmonCMS:
    @staticmethod
    def getFeedValues(feedId: int, moment: datetime.datetime = datetime.datetime.utcnow(), duration: int = 300, interval: int = 10):

        http = urllib3.PoolManager()

        start = (moment.timestamp() - duration) * 1000
        end = moment.timestamp() * 1000

        # Some very ugly string concatenation using data I provide
        url = (
            "https://some.host/feed/data.json?id="
            + str(feedId)
            + "&apikey=secret_secret_secret&start="
            + str(start)
            + "&end="
            + str(end)
            + "&interval="
            + str(interval)
        )
        r = http.request("GET", url, timeout=2)

        j = json.loads(r.data.decode("utf-8"))
        if j:
            return [{"time": datetime.datetime.fromtimestamp(float(item[0]) / 1000), "value": float(item[1])} for item in j]

        raise Exception("No results")

Hmm, I’m seeing gaps in the API results even though there are values in the source data for that interval which you can see in the graph.

I wonder if it’s related to the interval being 30 seconds and those readings being less often during the window.

Off to investigate.

Yep, with a ten second interval I get more readings.

{'time': datetime.datetime(2022, 1, 9, 20, 47, 30), 'value': 815.0}
{'time': datetime.datetime(2022, 1, 9, 20, 47, 40), 'value': 775.0}
{'time': datetime.datetime(2022, 1, 9, 20, 47, 50), 'value': 787.0}
{'time': datetime.datetime(2022, 1, 9, 20, 48, 10), 'value': 770.0}
{'time': datetime.datetime(2022, 1, 9, 20, 48, 20), 'value': 777.0}
{'time': datetime.datetime(2022, 1, 9, 20, 48, 50), 'value': 804.0}
{'time': datetime.datetime(2022, 1, 9, 20, 49, 10), 'value': 785.0}
{'time': datetime.datetime(2022, 1, 9, 20, 49, 20), 'value': 790.0}
{'time': datetime.datetime(2022, 1, 9, 20, 49, 50), 'value': 804.0}
{'time': datetime.datetime(2022, 1, 9, 20, 50, 10), 'value': 779.0}
{'time': datetime.datetime(2022, 1, 9, 20, 50, 20), 'value': 787.0}
{'time': datetime.datetime(2022, 1, 9, 20, 50, 30), 'value': 787.0}
{'time': datetime.datetime(2022, 1, 9, 20, 50, 40), 'value': 794.0}
{'time': datetime.datetime(2022, 1, 9, 20, 50, 50), 'value': 779.0}

So I’ll need to query with a ten second interval to match the storage definition I set in PHPFINA.

Just for extra excitement, the values come in every 6 seconds from CurrentCost…well in theory anyway.

It’s a good job I made it use the faster approach with the API calls :slight_smile:

Well sadly this really didn’t work out. There are loads of holes in the data which are not related to the power tripping. We haven’t been having them all Summer, but there are holes throughout the year.

I even tried looking at multiple feeds.

Ho hum.

It looks as if you need to do it the old-fashioned way: get an insulation tester, learn how to test safely, and only then start disconnecting and testing circuits and appliances one by one.

Or, as you suggested earlier, get somebody to do it.

1 Like

Yep, I’m lucky that I can afford someone to do it so I’ll go that route this time.

I’d also like to have an RCD on each circuit so that a problem in one area doesn’t pull everything down. I can’t safely fit RCBOs so I’ll see how the electrician gets along and if they are good then will see if they fancy doing that.

If you have a single RCD protecting multiple MCB’s, are the MCB’s or RCD tripping? Presuming it’s the RCD then it’s definitely earth leakage. If you can’t push up the RCD during a trip, turn off the first MCB and try to push up the RCD, and keep doing the same with each MCB. If you can’t push up the RCD with all the MCB’s down, then you likely have an earth to neutral short. If you are suspecting one spur, check that the neutral insulation, it could have a tiny cut on the metal back box of the spur.

Thanks for those thoughts @shaneosullivan

It’s the RCD tripping rather than the MCBs. However, it’s very, very sporadic. Most of the time when it trips it works fine when the RCD is flipped back so we don’t get to diagnose the problem.

On the rare occasion it does trip we found that it’s one MCB driving about 20 sockets.

The electrician did come and confirmed there was a problem on the spur I’d disabled. So that should be good news because we’ve disabled that spur.

However we had a trip after that too. Luckily it was only one and that was a couple of weeks ago. So maybe it’s fine now, or maybe tinkering with things just moved something enough that it’s fine again for a while.

We’ll have to do something when the sun is shining a bit more and we can take the power down for a while and try more things. We’re currently eating everything in the freezer :slight_smile:

BTW we have a TT system here and N-E can get up to 5V which is a little surprising, but the electrician said that was not a problem.

Part of the problem will be that the cupboard with the consumer unit in is too small for a bigger one with surge and RCBO and more spare lines so we’re going to have to figure something out, but there’s no sensible options at the moment so maybe another consumer unit elsewhere will be the way to go.

We had a similar problem, after unplugging everything else it turned out to be an under counter freezer where presumably condensation was accumulating on the top, enough to fill the recess for the electrics.

I think the condensation was because it needed defrosting.

1 Like

Ha! We’re eating all the stuff out of the freezer so we can turn it off :slight_smile:

1 Like