New secure input to Emoncms without HTTPS

Note: the encrypted input is now live on emoncms.org only. Not yet local Emoncms e.g emonPi

This post describes a protocol for securely posting input data to Emoncms. It has been developed and tested, and is in the process of being integrated into the Emoncms Git by @TrystanLea. It’s currently running on Emoncms.org.

BACKGROUND: passing input data to Emoncms at regular intervals can pose a variety of security risks including exposure of the read/write key which can be used in many other API calls for malicious activity and of course the exposure of the data which can compromise privacy and security.

HTTPS: When available, secure posting is easily accomplished. But some of the processors that are being used to collect input data do not have a practical HTTPS capability, or the memory required to do the authentication (digital signature), handshaking (asymmetric cryptography) or buffering required by some servers.

In addition to securing the transmitted data, HTTPS provides two other benefits:

  • It assures that you are connected to the domain that you thought you connected to (man in the middle attacks).
  • It insures against data loss or corruption.

IOTAWATT: This protocol was developed to respond to the need to make an ESP8266 based input device secure. While the ESP8266 does have some TLS capabilities, there were several issues that made using it less attractive or impossible with IotaWatt.

Unlike browsers which can establish anonymous TLS connections using public keys of common CAs to authenticate, the common ESP8266 approach requires that a trusted SHA1 hash of the server’s certificate be available for authentication. These certificates typically have a lifespan measured in months, thus the firmware needs to be updated to match the ever changing server certificate.

While the ESP8266 has enough heap to do all the crypto work (a small sketch has about 60K), IotaWatt is not a small sketch. There’s a lot going on and it has less than 20K available on a good day. The TLS routines seem to require more than that to work reliably.

While its possible to do asynchronous WiFi with the ESP8266, the conventional classes and methods that do a lot of the work for you run synchronously. They block for the duration of a HTTP transaction. With the handshake, TLS transactions can take up to twice as long (500-600ms). IotaWatt relies on constant sampling for accuracy and this is a significant interruption.

NEW AES ENCRYPTION PROTOCOL:
This protocol was worked out to try to solve all of the above problems in a practical way. There are theoretical flaws, but practically speaking, the protection is believed to exceed the value of the data. In today’s cyber environment, its always a risk/benefit calculation.

In establishing a HTTPS connection, the purpose of the upfront handshaking is to:

  • Authenticate the server
  • Agree on a symmetric encryption algorithm
  • Establish a shared secret key upon which to encrypt

We already have a shared secret key (the read/write key), and we have already agreed on a symmetric encryption algorithm (AES_128_CBC).

We authenticate the server by requiring a response that only a holder of the secret key could produce (SHA256 hash of the unencrypted data).

THE ALGORITHM:

OEM has implemented a /input/encrypted transaction. It will be formally described in the input-api-helper, and this post is subject to changes made there.

The basic transaction is a HTTP POST to the /input/encrypted URI using a

Content-Type: application/x-www-form-urlencoded

header. That tells the Emoncms server that the POST data is in URL encoded form, which is what you normally see after the ? in a GET request. Specifically, the POST consists of:

username=USERNAME&data=DATA

  • USERNAME is the Emoncms username for the account.

  • DATA is the Json array of input data in slightly modified “bulk” input format, encrypted with AES_128_CBC, and then base64 encoded (URL friendly).

So to break that down, the plain-text Json data array looks like this:

[[unixtime,node,input1,input2,etc],[unixtime,node,input1,input2,etc]]

Note that there is no time= or sentime= as in the bulk protocol. The time in each entry is the absolute unixtime that pertains to that data array. If the input contains multiple data packets, they should be ordered by ascending unixtime.

The node entry can be an integer node, or a quoted string with an alphanumeric node name.

The inputn entries are the actual data values for key 1,2,3 etc.

example:

[[1503071510,“iotawatt”,120.4,204,63,0,…],[1503071520,“iotawatt”,120.3,204,62,0,…],…]

The data array is encrypted using the AES_128_CBC algorithm, the apikey, and a random initialization vector which is simply a 16 byte random string.

The resultant encrypted data (same length as the unencrypted data) is appended to the initialization vector, and the combined string is base64 encoded. There are a few variants of base 64 encoding. We use the so-called “URL friendly” ( ‘-’ as 62 and ‘_’ as 63) because they are unambiguous in a URL (Content-Type: application/x-www-form-urlencoded).

That’s it for the POST packet. It’s sent to Emoncms where the read/write key associated with the username is looked up and used to decode the data array and it is processed.

Upon successful decode and processing, Emoncms produces a SHA256 hash of the decrypted data and sends that back. The sender independently produces a SHA256 hash of the original data and compares it to the response. If it matches, the recipient was able to decrypt the data, thus acknowledging receipt and verifying authenticity of the responder.

3 Likes

Nice write up Bob, thanks.

Paul

For anyone interested in looking at the emoncms code, here’s the emoncms branch with these changes
https://github.com/emoncms/emoncms/tree/input-improvements

In order to make the increasing input_controller code easier to work with I have moved the input/post, input/bulk and input/encrypted code into a new input_methods class and improved the reusability of parts of the code.

The specific input/encrypted implementation can be found in the input_methods class starting on line 36.
https://github.com/emoncms/emoncms/blob/input-improvements/Modules/input/input_methods.php#L36

I’ve updated the input API helper page in the input-improvements branch to reflect the changes (but not yet on emoncms.org)

Im going to do a bit more testing here before merging into the master branch. At which point it would be great to have some help testing before we merge it into the stable branch.

1 Like

There’s also an example here of how to post data to the new input/encrypted api using a PHP scipt:
https://github.com/emoncms/emoncms/blob/input-improvements/docs/input_encrypted.md

1 Like

There is now an updated implementation of this secure input method, which has now been adapted to be a more generic aes128cbc encryption transport layer, following coversation here: Input improvements by TrystanLea · Pull Request #710 · emoncms/emoncms · GitHub.

It is available in the emoncms aes128cbc-encryption branch for testing.

I found this when looking for HTTPS threads. @TrystanLea, is there a timeline to merge this?

[edit]
As an aside, I opened an issue a while back that time as a string is not handled correctly. If you are changing the structure of the files it might be a good time to look at this issue. I think the constant casting to an int of time variables is the key issue.

It would be good to see the back of that practice for other reasons too.

Hello @borpin, Yes I have been holding off merging this on the main emoncms repo as in our extended discussion on the topic on a previous closed pull request here Input improvements by TrystanLea · Pull Request #710 · emoncms/emoncms · GitHub there where differing opinions and I ran out of time to work on it. We needed the feature specifically on emoncms.org to support IotaWatt and so I implemented it there and left the pull request open on the emoncms repo so that I could potentially come back to it.

I was quite happy with the solution and if its something people would like to see included in the main emoncms repo Id be happy to merge. Is it a feature that you would find useful?

1 Like

I’ve just replied on the issue re time question, not quite sure that I follow just yet.
https://github.com/emoncms/emoncms/issues/777

Hi @TrystanLea,
There are a number of IoTaWatt users that post to a local Emonpi, so I think it would be useful to merge it, given there have been no issues on your end.

@borpin @overeasy I’ve merged it in, documentation is on the input API helper page

2 Likes