Note: the encrypted input is now live on emoncms.org only. Not yet local Emoncms e.g emonPi
This post describes a protocol for securely posting input data to Emoncms. It has been developed and tested, and is in the process of being integrated into the Emoncms Git by @TrystanLea. It’s currently running on Emoncms.org.
BACKGROUND: passing input data to Emoncms at regular intervals can pose a variety of security risks including exposure of the read/write key which can be used in many other API calls for malicious activity and of course the exposure of the data which can compromise privacy and security.
HTTPS: When available, secure posting is easily accomplished. But some of the processors that are being used to collect input data do not have a practical HTTPS capability, or the memory required to do the authentication (digital signature), handshaking (asymmetric cryptography) or buffering required by some servers.
In addition to securing the transmitted data, HTTPS provides two other benefits:
- It assures that you are connected to the domain that you thought you connected to (man in the middle attacks).
- It insures against data loss or corruption.
IOTAWATT: This protocol was developed to respond to the need to make an ESP8266 based input device secure. While the ESP8266 does have some TLS capabilities, there were several issues that made using it less attractive or impossible with IotaWatt.
Unlike browsers which can establish anonymous TLS connections using public keys of common CAs to authenticate, the common ESP8266 approach requires that a trusted SHA1 hash of the server’s certificate be available for authentication. These certificates typically have a lifespan measured in months, thus the firmware needs to be updated to match the ever changing server certificate.
While the ESP8266 has enough heap to do all the crypto work (a small sketch has about 60K), IotaWatt is not a small sketch. There’s a lot going on and it has less than 20K available on a good day. The TLS routines seem to require more than that to work reliably.
While its possible to do asynchronous WiFi with the ESP8266, the conventional classes and methods that do a lot of the work for you run synchronously. They block for the duration of a HTTP transaction. With the handshake, TLS transactions can take up to twice as long (500-600ms). IotaWatt relies on constant sampling for accuracy and this is a significant interruption.
NEW AES ENCRYPTION PROTOCOL:
This protocol was worked out to try to solve all of the above problems in a practical way. There are theoretical flaws, but practically speaking, the protection is believed to exceed the value of the data. In today’s cyber environment, its always a risk/benefit calculation.
In establishing a HTTPS connection, the purpose of the upfront handshaking is to:
- Authenticate the server
- Agree on a symmetric encryption algorithm
- Establish a shared secret key upon which to encrypt
We already have a shared secret key (the read/write key), and we have already agreed on a symmetric encryption algorithm (AES_128_CBC).
We authenticate the server by requiring a response that only a holder of the secret key could produce (SHA256 hash of the unencrypted data).
THE ALGORITHM:
OEM has implemented a /input/encrypted transaction. It will be formally described in the input-api-helper, and this post is subject to changes made there.
The basic transaction is a HTTP POST to the /input/encrypted URI using a
Content-Type: application/x-www-form-urlencoded
header. That tells the Emoncms server that the POST data is in URL encoded form, which is what you normally see after the ? in a GET request. Specifically, the POST consists of:
username=USERNAME&data=DATA
-
USERNAME is the Emoncms username for the account.
-
DATA is the Json array of input data in slightly modified “bulk” input format, encrypted with AES_128_CBC, and then base64 encoded (URL friendly).
So to break that down, the plain-text Json data array looks like this:
[[unixtime,node,input1,input2,etc],[unixtime,node,input1,input2,etc]]
Note that there is no time= or sentime= as in the bulk protocol. The time in each entry is the absolute unixtime that pertains to that data array. If the input contains multiple data packets, they should be ordered by ascending unixtime.
The node entry can be an integer node, or a quoted string with an alphanumeric node name.
The inputn entries are the actual data values for key 1,2,3 etc.
example:
[[1503071510,“iotawatt”,120.4,204,63,0,…],[1503071520,“iotawatt”,120.3,204,62,0,…],…]
The data array is encrypted using the AES_128_CBC algorithm, the apikey, and a random initialization vector which is simply a 16 byte random string.
The resultant encrypted data (same length as the unencrypted data) is appended to the initialization vector, and the combined string is base64 encoded. There are a few variants of base 64 encoding. We use the so-called “URL friendly” ( ‘-’ as 62 and ‘_’ as 63) because they are unambiguous in a URL (Content-Type: application/x-www-form-urlencoded).
That’s it for the POST packet. It’s sent to Emoncms where the read/write key associated with the username is looked up and used to decode the data array and it is processed.
Upon successful decode and processing, Emoncms produces a SHA256 hash of the decrypted data and sends that back. The sender independently produces a SHA256 hash of the original data and compares it to the response. If it matches, the recipient was able to decrypt the data, thus acknowledging receipt and verifying authenticity of the responder.