Heat pump experiment review after two years

Ah Tim that worked. I can see HeatingEnergyConsumedRate1 and HeatPumpFrequency changing for example.

Unfortunately those numbers are for the preceeding minute so if I don’t poll each minute I’ll only get a partial picture of what energy was consumed. Of course I could go back to looking at the aggregate number and doing a diff on that but it’s rather tedious.

At least now I’ll know a bit more about what to say to Mitsubishi.

I think I’ll try every two minutes and see what happens.

(if I sound exasperated it’s because I get data from my power usage every six seconds!)

Yep, that seems to be working but it’s clearly wrong. I’ll keep it running at two minutes and see if that runs smoothly.

I could concoct all sorts of theories about how it might be OK if I skip polling exactly on the hour or all sorts of nonsense. Let’s see.

OK, time to get some help. I’ve been chatting to Mitsubishi so this is my post for them to see what’s going on.

Hopefully they’ll take this news in the collaborative spirit we’re all espousing here, but I’ll apologise right here, right now if this ends up with the API being shut down.

2 Likes

As I’ve been able to get per-minute data for almost all of 2022, I wonder if there’s a difference in how we drive it?

This could be a wild herring, but is your script logging in to MELCloud once and using the ContextKey for the following requests?

Also, how often are you sending commands back to MELCloud? I’ve noticed that whenever I update heat pump settings via /Device/SetAtw, the following request for data from /User/ListDevices is sometimes stale. I think this is because SetAtw is returning the same data as ListDevices, tripping over the internal cache. With some clever programming, these could be combined into a single operation to set something and read the new state.

While I have been able to successfully control the flow temperature of the heat pump minute-by-minute with my own scripts, I’ve decided to leave most of the logic to the FTC. I’m only setting SetHeatFlowTemperatureZone1 (for refined weather compensation) and OperationModeZone1 (turns heating on/off based on whole house temperature). This means I’ve sending commands to MELCloud less often, and the heating will still work if my scripts break for some reason. (I’ve moved hot water control to a separate script).

2 Likes

I’ve been looking into this very deeply.

I have disabled the part of my scripts that send commands. The heat pump just ran and ran!

You can see my analysis. Specifically it appears it’s just the polling interval that changes the behaviour. I can’t find anything else that correlates with the troublesome behaviour.

@Timbones I’m going to have to dig into your script in more detail. Can I just confirm you are sure it’s actually getting per-minute data or are you scheduling it to run each minute? I ask because you’ve got that sleep in there which will shuffle it along a bit (but maybe the same bit each time).

This could be a wild herring, but is your script logging in to MELCloud once and using the ContextKey for the following requests?

I grabbed my context key a few years ago and have never changed it.

1 Like

Yes, it runs every minute, and almost always returns fresh data. I can’t remember why I added a 15 second pause before requesting the data. I think it was to allow fresh data to be sent from my heat pump to Mitsubishi. The data I get is a minute later than claimed anyway.

Thanks Tim.

It’s actually sorta fun seeing the 15-second offset in your graphs :slight_smile:

The data I get is a minute later than claimed anyway.

I assume you mean “older” which is what I’m seeing forever too. You can see it in the analysis results.

  • Your polling interval is often very short, often 10s or less. I would recommend at least waiting until 60s after Last Timestamp or after your previous request. Even without this server-side caching issue, you won’t get data any more frequently than 1 minute.
  • My script skips any data where Last Timestamp hasn’t changed. This doesn’t correlate with stale data though, so don’t rely on it for detecting outages.
  • There’s no data only one bit in your analysis where the polling interval was consistently between 60s and 120s: 12:23 to 12:26. It would be interesting to see how well it works with a minimum polling of 1 minute…

I wonder if a freshly acquired context key is needed to allow polling every 1 minute instead of 2 minutes. This might be what my script is doing without me realising, as it will randomly re-authenticate every few days… I’ll test this out next time the server misbehaves again.

Sadly I tried a new context key to no avail. Nice try though.

So many “well, it almost looks OK” :slight_smile:

I still think you’re polling too fast, even with a fresh key - 7 times in a minute in the middle there!

1 Like

I certainly am. It’s part of my chaos-monkey thing where I have to find the edge of the envelope of everything so I know how much I can rely on it. I lost my first two Lear jets that way.

2 Likes

OK @Timbones I’m going to put you on the hook here.

Mitsubishi have come back to me and indicated that the data is only updated every two minutes following the firmware update. They’ve shown me a screenshot from their spreadsheet with it being every two minutes.

Now that I’m polling every two minutes things are peachy.

However, it really genuinely looks to me as though you are getting data every minute:

I’m going to dig into what you are doing differently, but I wanted to check that you believe you really are getting different data each minute from User/ListDevices and that I haven’t just optimistically read your posts with rose-tinted glasses.

I should also purposefully ask which FTC you have.

Thanks

David

Would you ever consider asking them whether we can do anything useful with the MAC-567IF-E HTTP interface and whether they’d enable EchoNetLite like the NZ users get, so we’re not hammering their melcloud as much and we don’t lose access when our internet link hiccups?

Yes. I’ll include that in the list of things I ask them. One step at a time :slight_smile:

1 Like

I am very sure that my heat pump data updates every minute, and always has (except when it doesn’t). Here’s a screenshot of Hourly Temperature report from MELCloud:

Now, even with two-minute update interval I think MELCloud will still plot 60 points per hour, by interpolating ever other point. But looking at the Tank Temperature above, there are definitely distinct values which couldn’t be interpolated.

I note that MMSP requires measurements to be recorded every two minutes (or less).

My internal unit has the model number PAC-IF073B-E, which appears to be supplied with FTC6.

I’ve tried hacking into the local web interface to see if I could get data that way, but to no avail. I even looked at meldec and tried the packet sniffing, but it looks like Mitsubishi are now encrypting the traffic. So yes, would love to be able to read it directly.

1 Like

Here’s a concrete example where I inspected the JSON behind one of those hourly reports.

a            35.5,
b            36.0,
c            37.0,
d            36.0,
e            36.5,

If a and c are real readings then we could imagine b was interpolated. However that would mean c and e are real and d is interpolated and that clearly can’t be true because it’s lower than both.

If b and d are real then c is interpolated, but clearly can’t be true either because it’s higher than both.

Of course we’re assuming linear interpolation here, but speculating there’s a higher-order polynomial would be madness. Surely?

It’s tempting to think there’s a mis-understanding where we are talking about device → MELCloud, or MELCloud → UI and our chums at Mitsubishi are talking about some other link but I can’t for the life of me imagine what it would be.

It may just be that the reporting endpoints and the ListDevice endpoints are different things in MELCloud land and so this analysis of the reports we see in the UI is a red-herring.

I don’t know what to think at the moment.

It would not surprise me if they were showing you screenshots from their MMSP spreadsheets, which are possibly exported from the real data at the reduced rate.

Perhaps, but I’m also seeing variation of data in ListDevices every minute too. Specifically in OutsideTemperature (which is only to nearest degree, and sometimes oscillates) and CurrentEnergyConsumed (which at idle shows 1 Wh being used every 3 minutes or so):

10:36:00, 0
10:37:00, 1
10:38:00, 0
10:39:00, 0
10:40:00, 1
10:41:00, 0
10:42:00, 0
10:43:00, 1
10:44:00, 0
10:45:00, 0
10:46:00, 1
10:47:00, 1
10:48:00, 0
10:49:00, 0

(adds up to 20Wh over an hour, i.e. ~20 watts of power on standby)

Have you tried polling every one minute (and not a second sooner)?

Well, this is a bit sad.

I’m now running your script. It has not magically fixed my problems.

If I run your script every 61 seconds it still comes unstuck. If I run it every two minutes it’s happy.

I tried with / without the sleep. That doesn’t change the actual time between calls to the API.

I tried with / without the LastTimestamp (a.k.a. stale data) check. That occurs after the call to MELCloud so it doesn’t impact the behaviour.

I tried with / without new key generation. This might have helped, but it didn’t. By default it merrily keeps using the same key so I don’t think you are re-generating the key often.

Your poll-backoff overnight would help, but that would not fix my daytime problems.

Your CurrentEnergy* fun is quite interesting. I was just trusting the stats in the other fields.

You do have a bit of jitter caused by fetchLastValue but that comes after the API call. Seeing as you are kicking off the job every minute from an external process (assumably cron) it means you aren’t impacted by that jitter. You would be if your script was going in a loop with a sleep(60) for example.

Your fetch is a single API call, I’m doing three. I was hoping your simpler approach would mean it was less likely to trouble the servers.

Given all this I’m not sure what to conclude. You clearly have per-minute data.

In addition I have a rather confusing situation here. My “sum of bits of energy” is giving me the correct total (it matches MELCloud) even though I’m only seeing half the readings and I don’t think the readings are being “doubled-up”. Still investigating that.

I put this in when I realised that MELCloud was attributing the standby energy to whichever mode it ran last [issues/19] and I really wanted to separate out the idle power to better understand it.

I can only suggest polling every 2 minutes for now, and in a week or two try again with 1 minute interval. This is what I need to do on occasion, when the servers get in a huff with me. A new context key might help at this point, but I have no evidence to support that theory yet.

(brain wave: maybe alternating between two context keys might work???)

This is because we’re converting the Energy into Power (Wh x60 => W), and then getting emoncms to convert that back with Power to kWh process step. Any gaps (nulls) in the power readings are correctly accounted for, so that the overall power consumption/production is mostly correct. The errors will be pretty small, and will probably average out to nothing much.

Had we been accumulating “bits of energy” directly, then we would have been missing half of it.