Python code to read TP-Link HS110

pb66 · 18 January 2019 00:29

As promised in the Wall plug (passthrough) or socket monitors? - #10 by pb66 thread, here’s some code for reading tp-link hs-110 energy monitoring smart sockets and posting to an emonhub socket interfacer.

The script needs to be executable and started at boot via service unit or init script and some settings applied to the emonhub.conf (and possibly UFW). Once in emonhub (emonpi variant) it can/will be published to MQTT so I have included the nodes definitions too.

This isn’t intended as a walk through guide to installing this, it’s just providing some code to get going and example settings for emonhub to get an interfacer up and running.

#!/usr/bin/env python
"""
Script to read TP-Link HS110 Energy monitoring smart sockets and post data to emonhub via a simple socket interfacer. 

It uses "harmonized intervals" so all update timestamps are exact multiple of the interval.

It has very low timeout settings so that it doesn't dwell on a failed/slow connection, it tries and moves on swiftly so other device reads/sends are not blocked. 
"""

__author__ = 'Paul Burnell (@pb66)'

import socket
import json
import time
import datetime

# emonhub address and port as defined in the interfacer set up.
emonhub_host = "192.168.1.78"
emonhub_port = 50013

# Define node ids and addresses for each TP-link HS110 
nodes = {27:{"address":"192.168.1.184"},\
		 28:{"address":"192.168.1.185"},\
		 29:{"address":"192.168.1.164"} }

# Update interval
interval = 10

# Print some info to console, True or False
debug = True
	
# Encryption and Decryption of TP-Link Smart Home Protocol
# XOR Autokey Cipher with starting key = 171
def encrypt(string):
	key = 171
	result = "\0\0\0\0"
	for i in string: 
		a = key ^ ord(i)
		key = a
		result += chr(a)
	return result

def decrypt(string):
	key = 171 
	result = ""
	for i in string: 
		a = key ^ ord(i)
		key = ord(i) 
		result += chr(a)
	return result
	
# Connect to, read and disconnect from device
def read_data(timestamp,node):
	payload = False
	socket_hs110 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
	socket_hs110.settimeout(2)
	socket_hs110.connect((nodes[node]["address"], 9999))
	cmd = '{"emeter":{"get_realtime":{}},\
			"system":{"get_sysinfo":{}},\
			"time"  :{"get_time":{},\
			   		  "get_timezone":{}} }'
	socket_hs110.send(encrypt(cmd))
	read_reply = json.loads(decrypt(socket_hs110.recv(4096)[4:]))
	socket_hs110.close()
	values = [	read_reply['emeter']['get_realtime']['voltage'],\
				read_reply['emeter']['get_realtime']['current'],\
				read_reply['emeter']['get_realtime']['power'],\
				read_reply['emeter']['get_realtime']['total'],\
				read_reply['system']['get_sysinfo']['relay_state'],\
				read_reply['system']['get_sysinfo']['rssi'] ]
	#print(values)
	payload = ' '.join(str(i) for i in [timestamp,node]+values)
	if debug:
		print("Read {}".format(payload))
	return payload

# Send the data to an emonhub socket interfacer
def send_data(payload):
	socket_emonhub = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
	socket_emonhub.settimeout(2)
	socket_emonhub.connect((emonhub_host, emonhub_port))
	reply = socket_emonhub.sendall(payload+"\r\n")
	socket_emonhub.close()
	if debug:
		print("Sent {}".format(payload))
	return reply


# create place to hold each nodes values
for node in nodes:
	nodes[node]["payload"] = False

# Get the current interval time
timestamp = ((time.time()//interval)*interval)

while(True):

	# If the next read is due, read all the devices.
	time_now = time.time()
	if time_now >= timestamp+interval:
		timestamp = ((time_now//interval)*interval)
		for node in nodes:
			try:
				nodes[node]["payload"] = read_data(timestamp,node)
			except Exception as error:
				if debug:
					print("FAIL {} Unable to read node {}: {}".format(timestamp,node,error))
	else:
		# Inbetween reads do not loop too fast
		time.sleep(0.1)


	for node in nodes:
		if nodes[node]["payload"]:
			try:
				send_reply = send_data(nodes[node]["payload"])
				nodes[node]["payload"] = False
			except Exception as error:
				if debug:
					print("FAIL {} Unable to send node {}: {}".format(timestamp,node,error))

Example interfacer entry for emonhub.conf [interfacer] section

      [[TPLinkHS110]]
          Type = EmonHubSocketInterfacer
          [[[init_settings]]]
              port_nb = 50013
          [[[runtimesettings]]]
              timestamped = True
              subchannel = ToEmonCMS,     # Only for emonpi variant of emonhub

and if needed open the port in the firewall (ie if script not running on same device as emonhub and UFW is installed)

sudo ufw allow 50013 && sudo ufw -f enable

Example nodes entries for emonhub.conf [nodes] section

      [[27]]
          nodename = TPLinkHS110a
          [[[rx]]]
              names = Voltage,Current,Power,Energy,Status,RSSI
              units = V,A,W,kWh,state,dB
      [[28]]
          nodename = TPLinkHS110b
          [[[rx]]]
              names = Voltage,Current,Power,Energy,Status,RSSI
              units = V,A,W,kWh,state,dB
      [[29]]
          nodename = TPLinkHS110c
          [[[rx]]]
              names = Voltage,Current,Power,Energy,Status,RSSI
              units = V,A,W,kWh,state,dB
              #datacodes = 0,0,0,0,0,0
              #scales = 1,1,1,1,1,1

strictly speaking the datacodes and scales shouldn’t be needed, but I have included them in just one entry above just to indicate what would/could be used.

Example console output if debug is set to True

Read 1547767370.0 27 247.095985 0.023666 0 0 0 -53
Read 1547767370.0 28 248.32088 0.019906 0 1222.02 0 -57
Read 1547767370.0 29 246.823264 3.75433 926.64569 27.302 1 -76
Sent 1547767370.0 27 247.095985 0.023666 0 0 0 -53
Sent 1547767370.0 28 248.32088 0.019906 0 1222.02 0 -57
Sent 1547767370.0 29 246.823264 3.75433 926.64569 27.302 1 -76
Read 1547767380.0 27 247.131161 0.023279 0 0 0 -52
Read 1547767380.0 28 248.293119 0.019755 0 1222.02 0 -56
Read 1547767380.0 29 246.80449 3.738486 922.668525 27.305 1 -75
Sent 1547767380.0 27 247.131161 0.023279 0 0 0 -52
Sent 1547767380.0 28 248.293119 0.019755 0 1222.02 0 -56
Sent 1547767380.0 29 246.80449 3.738486 922.668525 27.305 1 -75
Read 1547767390.0 27 247.256333 0.023778 0 0 0 -52
Read 1547767390.0 28 248.442524 0.020359 0 1222.02 0 -57
Read 1547767390.0 29 246.940376 3.737328 922.895971 27.308 1 -74
Sent 1547767390.0 27 247.256333 0.023778 0 0 0 -52
Sent 1547767390.0 28 248.442524 0.020359 0 1222.02 0 -57
Sent 1547767390.0 29 246.940376 3.737328 922.895971 27.308 1 -74

In time I would like to perhaps keep the sockets open to reduce the workload of keep opening and closing, plus then the script could listen for a command to switch on and off, I have the code to do that (somewhere) but the emonhub socket interfacer would also need some work to accomplish this.

I do not run an emonSD and I have not had a chance to test the settings on an emonPi, so please let me know if I’ve got the settings wrong. This is not the script I use myself as I have other stuff going on in mine. I have pulled code from my script to make this more generic forum friendly version.

[edit - I’ve been running this overnight on a Pi and found the occasional timeout, I have changed the timeout settings from 1s to 2s and that seems to have fixed it, no timeouts since. That is still a tight timeout, the idea is to just allow enough time for a “usual” connection and move on as there will be another attempt within 10s, no point sitting there for 60s.]

bdhughes · 6 October 2020 19:53

Hi Paul

I recently got some of these plugs and have just tried your script. It looks like I don’t get anything back.

This is the error i’m getting:
FAIL 1602013830.0 Unable to read node 28: No JSON object could be decoded

I can see my devices on the network and port 9999 is open.

Do your devices still work? Perhaps the protocol on the later ones is different or the encryption has been changed?

Any ideas?

Brian

pb66 · 6 October 2020 23:19

Hi Brian, I have 3 of these, 2 are still in service with no issues, the other hasn’t been used in a while.

I just dug it out to try it and whilst I had some issues getting it connected to my network for a string of annoying reasons, device wouldn’t connect to unchanged local network despite reserved IP address, change of mobile means I no longer had Kasa installed, for some reason Kasa no longer recognised my email as a valid account and yet once I created a new account with another email it told me the device was already linked to another account. However! Once it was finally connected to the network it started reporting data straight away, so for me, the script part is still working fine.

A couple of things to check

I assume you’ve edited the script for your own IP addresses?
Are you using Python 2 or 3? The device I have this set up on is still Python 2, but note the shebang at the top of the script is just “python” so the default for your OS will be selected. Edit that line to “python2” to force it to use python 2 over python 3 if it is installed. It maybe that the script needs some tinkering for use on python 3, I haven’t looked at it for sometime.

Other than that, I have now noticed that there is a “v2” tp-link hs-110, how that differs to a “v1” I have no idea, but mine are indeed v1s. Whilst wrestling with the 3rd device earlier the Kasa app tried to corner me into updating the firmware, based on your post I ducked that update until I can be sure it will continue to work as (aside from todays network issue) I’ve had no issues with these devices as is so there’s no rush to update the FW if there is a risk of breaking things.

Maybe try increasing the timeout in case the timing has changed? Do you have debug enabled? Does the reply payload make sense? ie has the structure or labelling changed? You could try adding some print lines to help debug, I also just noticed the command could be shortened (simpler might more successful?) as there is a call for sometime and timezone data unused in this script. Change the

cmd = '{"emeter":{"get_realtime":{}},\
			"system":{"get_sysinfo":{}},\
			"time"  :{"get_time":{},\
			   		  "get_timezone":{}} }'

to

cmd = '{"emeter":{"get_realtime":{}},\
			"system":{"get_sysinfo":{}} }'

If I think of anything else I’ll add it here. Good luck

bdhughes · 7 October 2020 09:59

Hi Paul

Thanks for your reply and experiments to investigate. My plugs say v4.1 on the lable on the back. So, I’ve investigated a bit further and found a python program that does work with my devices - the comments say:
# by Lubomir Stroetmann
# Copyright 2016 softScheck GmbH

Although I actually found it on freebasic.net - I think it came from github originally.

This program is able to send various commands to the switch. I noticed that the encrypt function is slightly different to yours. The result variable is initialised differently so I commented out your line and put the modified version in:

#result = “\0\0\0\0”
result = pack(’>I’, len(string))

That’s a ‘Greater Than’ and a capital letter i. It looks like a pipe on this font but it’s not.

I also had to put the following import in too:
from struct import pack

Then I started to get some data, however it looks like the json keys have changed since your version to include the units so I had to modify the key names as follows:
voltage → voltage_mv
current → current_ma
power → power_mw
total → total_wh

I’ve got it all working and I’m very happy. I don’t really know what ‘pack’ does but it made all the difference.

Thanks for your code and pointers.
Brian

Bill.Thomson · 7 October 2020 11:42

Python pack performs conversions between Python values and C structs represented as Python strings.

pb66 · 9 October 2020 00:46

Ahhh yes! Most of my code above originates from his work too. He did this blog on reverse engineering the tp-link devices.

which led to this code

which I shamelessly picked at for the bits I needed for the script above.

Your findings with the encryption changes inspired me to delve in a little closer and I found that 3yrs ago the code for that encryption was changed in his scripts “make it work with hs105”

so I suspect at some point since, the hs100 and hs110 FWs have changed to use a similar protocol to the HS105 and that has slipped under the radar as the changes needed for the later FW were already there for the HS105 perhaps.

I also noticed that the code was refactored for python3 8mths ago so the info is all there to update the script, maybe I’ll try the “hs-105 changes” here with my devices to see if it’s backwards compatible with older FW if I get a chance.

The python pack module is also used in emonhub form decoding/encoding packets of data used by OEM’s RFM devices such as emonTX/TH/Pi etc etc.

Thanks for sharing your fix

sxw · 9 October 2020 19:15

Hijacking a little here, as this just popped up new again. I ended up doing this for a hs110 energy monitor, and didn’t find this post so I came up with the following script. Call it with the IP address of your plug, and the MQTT service you want to publish to. It assumes you still have your mqtt password set to the default.

You’ll need paho-mqtt and python-kasa installed. I run this from cron with something like:

/home/pi/kasa-upload.py 192.168.1.244 drier >/dev/null 2>&1

Here’s the script:

#!/usr/bin/python3

import asyncio;
from kasa import SmartPlug
import paho.mqtt.publish as publish
import sys

plug = SmartPlug(sys.argv[1])
asyncio.run(plug.update())
power = plug.emeter_realtime.get('power_mw')
power = power / 1000

publish.single("emon/kasa/%s" % sys.argv[2], payload=power,
               auth = {'username':"emonpi", 'password':"emonpimqtt2016"})

pb66 · 4 December 2020 11:18

9 posts were split to a new topic: Problem installing the python-kasa package for use with TP-Link HS110

pb66 · 4 December 2020 17:20

Just a quick note to confirm this simple edit, plus adding

from struct import pack

to the imports at the top of the script works with both the latest and the earlier protocols. The “pack” line is just flexible in length rather than fixed as it was originally.

Again, I have confirmed this change. The structure seems the same but the labels have changed so a simple test to determine which labels to use should make the script compatible with both old and new devices.