Update firmware using emonUpload tool

Hardware
1

I have been working on a python based command line tool to grab the latest firmware for all our hardware units (emonTx V3, emonTH, emonTH V2, emonPi) via GitHub releases API and make it easy for the user to burn an Arduino bootloader via ISP (if needed) then upload the latest firmware via serial USB to UART. The tool is called emonUpload. We use it internally in the factory and in our office to upload firmware to all our units before shipping.

emonUpload also supports viewing serial debug output via the internal serial monitor and unit testing of hardware functions via PlatformIO.

See emonUpload GitHub page for full feature list, install instructions and example operation:

GitHub - openenergymonitor/emonupload: Upload / update latest OpenEnergyMonitor firmware

emonUpload can be used to update OpenEnergyMonitor units to latest firmware versions.

Latest version of our pre-built RaspberryPi (emonPi & emonBase) image (emonSD-01Nov16 DUE FOR RELEASE SOON) will include emonUpload pre-installed and configured.

1 Like
3

Trying and getting this error:

pi@emonpi(rw):emonupload$ pip install -r requirements.txt
Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 1))
Downloading/unpacking gitpython (from -r requirements.txt (line 2))
  Downloading GitPython-2.1.0-py2.py3-none-any.whl (441kB): 441kB downloaded
Requirement already satisfied (use --upgrade to upgrade): pyserial in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 3))
Downloading/unpacking gitdb2>=2.0.0 (from gitpython->-r requirements.txt (line 2))
  Downloading gitdb2-2.0.0-py2.py3-none-any.whl (63kB): 63kB downloaded
Downloading/unpacking smmap2>=2.0.0 (from gitdb2>=2.0.0->gitpython->-r requirements.txt (line 2))
  Downloading smmap2-2.0.1-py2.py3-none-any.whl
Installing collected packages: gitpython, gitdb2, smmap2
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 295, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1436, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 672, in install
    self.move_wheel_files(self.source_dir, root=root)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 902, in move_wheel_files
    pycompile=self.pycompile,
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 214, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 204, in clobber
    os.makedirs(destdir)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/GitPython-2.1.0.dist-info'

Storing debug log for failure in /home/pi/.pip/pip.log

pi@emonpi(rw):emonupload$ ./emonupload.py
Traceback (most recent call last):
  File "./emonupload.py", line 12, in <module>
    import serial, sys, string, commands, time, subprocess, os, urllib2, requests, urllib, json, git
ImportError: No module named git


pi@emonpi(rw):emonupload$ sudo apt-get install git python-pip make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
build-essential is already the newest version.
git is already the newest version.
git set to manually installed.
libssl-dev is already the newest version.
libssl-dev set to manually installed.
make is already the newest version.
make set to manually installed.
python-pip is already the newest version.
zlib1g-dev is already the newest version.
zlib1g-dev set to manually installed.
The following extra packages will be installed:
  bzip2-doc libreadline6-dev libtinfo-dev
Suggested packages:
  readline-doc sqlite3-doc
The following NEW packages will be installed:
  bzip2-doc libbz2-dev libreadline-dev libreadline6-dev libsqlite3-dev libtinfo-dev
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 488 kB/993 kB of archives.
After this operation, 2,514 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Err http://mirrordirector.raspbian.org/raspbian/ jessie/main libsqlite3-dev armhf 3.8.7.1-1+deb8u1
  404  Not Found [IP: 5.153.225.207 80]
E: Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/s/sqlite3/libsqlite3-dev_3.8.7.1-1+deb8u1_armhf.deb  404  Not Found [IP: 5.153.225.207 80]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
4

What version pip are you running?

$ pip --version
pip 8.1.2

Try running as sudo: sudo pip install -r requirements.txt

To fix the failed the fetch archives

Run update before install:

sudo apt-get update
sudo apt-get install git python-pip make build-essential libssl-dev zlib1g-dev

Thanks , I will add this to the install notes :thumbsup:

5

pi@emonpi(rw):~$ pip --version
pip 1.5.6 from /usr/lib/python2.7/dist-packages (python 2.7)

6

I think this has to be fixed:

pi@emonpi(rw):~$ sudo apt get install python avrdude git-core -y
E: Invalid operation get

ā€¦

maybe working now

pi@emonpi(rw):emonupload$ ./emonupload.py

RFM69Pi detected
Testing internet connection...
Internet connection detected
Updating e9cd0d4..b8bb0d0
Fast-forward
 README.md | 3 +++
 1 file changed, 3 insertions(+)
UPDATE FOUND....emonUpload RESTART REQUIRED
EXITING...
pi@emonpi(rw):emonupload$ sudo reboot now

Broadcast message from pi@emonpi on pts/0 (Fri 2016-11-04 16:46:41 UTC):

The system is going down for reboot NOW!
7

The error was caused by a missing dash character. (between apt and get)

The command should be: sudo apt-get install python avrdude git-core -y

8

SOLVEDL donā€™t know what happened but after RFM69Pi update, one of my two EmonTH data stopped to be received :scream: restarting the emonth data flown again

testing or serial visualization not working - error telling me PlatformIO not installed, but it was successull

9

Great, nice work. Yes @Bill.Thomson is correct. There should be a - in between apt and get e.g. apt-get install. I have corrected the install instructions. Thanks for your help:

PlatofmIO must be installed to view serial monitor and unit test. To install platformio on Linux:

$ sudo python -c "$(curl -fsSL https://raw.githubusercontent.com/platformio/platformio/master/scripts/get-platformio.py)"

See: http://docs.platformio.org/en/latest/installation.html#installer-script

10

I do confirm PlatformIO was successfull. Retrying:

pi@emonpi(rw):~$ sudo python -c "$(curl -fsSL https://raw.githubusercontent.com/                                                                                                                     platformio/platformio/master/scripts/get-platformio.py)"

==> Installing Python Package Manager ...
Requirement already up-to-date: pip in /usr/local/lib/python2.7/dist-packages

[SUCCESS]

==> Installing PlatformIO and dependencies ...
Requirement already up-to-date: platformio in /usr/local/lib/python2.7/dist-packages
Requirement already up-to-date: pyserial<4 in /usr/local/lib/python2.7/dist-packages (from platformio)
Requirement already up-to-date: requests<3,>=2.4.0 in /usr/local/lib/python2.7/dist-packages (from platformio)
Requirement already up-to-date: semantic-version>=2.5.0 in /usr/local/lib/python2.7/dist-packages (from platformio)
Requirement already up-to-date: lockfile<0.13,>=0.9.1 in /usr/local/lib/python2.7/dist-packages (from platformio)
Requirement already up-to-date: colorama in /usr/local/lib/python2.7/dist-packages (from platformio)
Requirement already up-to-date: click<6,>=5 in /usr/local/lib/python2.7/dist-packages (from platformio)
Requirement already up-to-date: bottle<0.13 in /usr/local/lib/python2.7/dist-packages (from platformio)

[SUCCESS]

 ==> Installation process has been successfully FINISHED! <==

Usage: platformio [OPTIONS] COMMAND [ARGS]...

Options:
  --version          Show the version and exit.
  -f, --force        Force to accept any confirmation prompts.
  -c, --caller TEXT  Caller ID (service).
  -h, --help         Show this message and exit.

Commands:
  boards    Pre-configured Embedded Boards
  ci        Continuous Integration
  device    Monitor device or list existing
  init      Initialize PlatformIO project or update existing
  lib       Library Manager
  platform  Platform Manager
  run       Process project environments
  settings  Manage PlatformIO settings
  test      Unit Testing
  update    Update installed Platforms, Packages and Libraries
  upgrade   Upgrade PlatformIO to the latest version



----------------------------------------
Please RESTART your Terminal Application
----------------------------------------

Then run `platformio --help` command.

========================================================================

I still get error, for instance when I choose:

(s) to view Serial

(0) for older units serial @9600

Error PlatformIO avrdude is NOT installed

11

Ah yes, PlatformIO is installed but the atmelavr platform is not installed.

$ pio platform install atmelavr --with-package tool-avrdude

I have added this to the install guide.

The platforms usually get automatically installed when Pio is ran for the first time. However, since we are calling the functions directly autoinstall will not happen. If you are still having issues with emonupload detecting platformIO try and compile some code e.g.

$ cd emonupload/repos/openenergymonitor-emontx
$ pio run
12

Now it works ! Thanks.

1 Like
13

Great! Thanks for beta testing :slight_smile:

The tool is very basic (interface could certainly be improved) but it has made a big improvement for our deployment of code and testing in the factory and in our office.

14

Glyn, I like what you have done with this simple script.

Which boot loader are you burning on the chips from the factory now ?

Is the purpose of this script to enable home users to update by plugging in an emonTX (for instance) over a serial cable or a ISP cable ?

15

Thanks, We use the ā€˜unoā€™ optiboot bootloader.

The main purpose of the script was for us to use in the factory and the lab, however, itā€™s easy enough for anyone to use. It;s much easier to upload the precompiled firmware than have to deal with compiling from source. However, the main reason for a user to update the firmware is to implement a custom modification which requires compiling which is now much easier if using PlatformIO: Firmware Modification - Guide | OpenEnergyMonitor

16

Hey Glyn,

Thanks for putting this into the open source ā€¦ it helped me greatly today.

In case anyone else gets into the state I did. Hereā€™s the chain of events in brief:

Friday (Today is Monday) I ā€œbrickedā€ my emontx as it got reset during an update via UART; donā€™t ask how!!
It seems this interruption ate the bootloader. I spent the weekend figuring out how to get it back:

  1. Breadboarded a programmer using an Arduino Nano and the ā€œArduino as ISPā€ sketch (multiple hours)
  2. Discovered that the baud rate must be 19200 for the Arduino as ISP to work -b 19200
  3. Discovered the capacitor on the reset to ground requirement (on the programmer Arduino). Added that but still got flaky connectivity which resulted in the fuses (just set) being read in a wrong state after the bootloader hex was uploaded using:

avrdude -v -p atmega328p -c avrispmkII -P com5 -b 19200 -e -Ulock:w:0x3F:m -Uefuse:w:0x05:m -Uhfuse:w:0xDE:m -Ulfuse:w:0xFF:m -U flash:w:emonTxV3_RFM69CW_latest_433_bootloader.hex:i -Ulock:w:0x0F:m:

  1. Moved to using an Arduino Mega instead of a Nano as the programmer (suggested on several sites as better than the Nano). BTW: The default is to use an Arduino Uno, but I only had the Mega and nanos lying around.
  2. Finally jumped into emonupload on github and found the naked optiboot_atmega328.hex.
  3. Found that in the emonupload command for avrdude there is a slight difference in the fuse. I have no idea if this is important ā€¦ but itā€™s there: hfuse:w:0xD6: rather than hfuse:w:0xDE: from the wiki!

avrdude -p atmega328p -c avrispmkII -P com5 -b 19200 -e -U efuse:w:0x05:m -U hfuse:w:0xD6:m -U lfuse:w:0xFF:m -U flash:w:optiboot_atmega328.hex:i -Ulock:w:0x0f:m

In the end I used this command from emonupload with the optiboot_atmega328.hex via Arduino Mega programmer into the ISP header on the emontx board. NOTE: I put the 5V from the programmer into the 5V pin on the UART NOT the ISP header. This results in the chip and the RF module getting 3.3V rather than 5V which I read somewhere was a good way to avoid killing the RF module!
Once that was done I disconnected the ISP and went to regular connection using a USB to UART device and uploaded (without interruption) the firmware using PlatformIO in the normal fashion.

After all that Iā€™m now back to where I was 3 days ago! Hopefully this helps someone ā€¦

2 Likes
17

D6 will preserve eeprom content when flashing via serial where as DE will not, although we do not use the eeprom as much as Iā€™d like so, no itā€™s not important, but IMO D6 is the better option because you can do version upgrades without losing eeprom settings (eg custom node id on a emonTx). With DE you would need to upload the latest firmware and then remember to change your node id again each time. However DE is the standard for the Arduino Uno etc probably because they are prototyping boards so a re-flash may well mean a different sketch/project altogether.

3 Likes