I downloaded the latest emonSD-07Nov16 image, mounted the data partition inside a limux virtual machine, and made some changes to emonHub and EmonCMS. Changes are non-breaking, 100% sure.
Now when I launched the image on the rasberry-pi, I was unable to receive inputs from the RFM device to emoncms.
With some debugging, I found that the mqtt.service file was missing. How come this happened? Does the latest image not have all the required files, or did it get deleted on start-up?
I see in some other threads people are having some issues using the mqtt.service aswell, but surely booting the latest image should not have caused this?
Can you download and attach the update.log for us to take a peek at?
Is there any chance these changes are blocking the emoncms repo updating via a git pull?
The latest image doesnāt have the mqtt_input.service file, it was added since the original image was built. The file is included in the updated emoncms repo and copied across to the /etc/systemd/system folder by the updated emoncms update script, If the script has updated to be aware of the file but cannot find the file it would suggest the emoncms git pull has failed or is pointing at a different branch.
Very likely. Since the changes are not ācommittedā, its likely the pull would fail. Guess I would rather need to update the updating script to apply my changes after the pull-request.
I2C LCD NOT DETECTED
Starting emonPi Update >
EUID: 1000
Tue 17 Jan 07:28:18 UTC 2017
emonSD-07Nov16
git pull /home/pi/emonpi
From https://github.com/openenergymonitor/emonpi
008c2b6..a4263e5 master -> origin/master
* [new branch] emonhub-restart -> origin/emonhub-restart
* [new tag] 2.8.0 -> 2.8.0
* [new tag] 2.8.1 -> 2.8.1
* [new tag] 2.8.2 -> 2.8.2
Updating 008c2b6..a4263e5
Fast-forward
.travis.yml | 26 +-
README.md | 2 +-
bash-rw-indicator | 7 +-
emoncmsupdate | 30 +
emonhub-sudoers | 1 +
emonpiupdate | 1 +
firmware/compiled/archive/emonPi_V2.7.hex | 1195 ++++++++++++
firmware/compiled/archive/emonPi_V2.8.1.hex | 1102 +++++++++++
firmware/compiled/archive/emonPi_V2.8.2.hex | 1120 ++++++++++++
firmware/compiled/archive/emonPi_V2.8.hex | 1194 ++++++++++++
firmware/compiled/latest.hex | 2340 +++++++++++-------------
firmware/compiled/readme.md | 4 +-
firmware/libraries/Readme.md | 4 +-
firmware/platformio.ini | 19 +-
firmware/readme.md | 61 +-
firmware/src/lcd_serial.ino | 38 +-
firmware/src/rf.ino | 75 +-
firmware/src/src.ino | 97 +-
firmware/src/startup.ino | 6 +-
firmware/test_sketches/i2c_scan/.gitignore | 2 +
firmware/test_sketches/i2c_scan/.travis.yml | 65 +
firmware/test_sketches/i2c_scan/lib/readme.txt | 36 +
firmware/test_sketches/i2c_scan/platformio.ini | 22 +
firmware/test_sketches/i2c_scan/src/src.ino | 66 +
firstbootupdate | 18 +-
lcd/emonPiLCD.py | 43 +-
lcd/emonPiLCD_update.py | 21 +-
lcd/lcd_test.py | 10 +-
lcd/lcddriver.py | 7 +-
logrotate.conf | 2 +-
rc.local_jessieminimal | 5 +-
rfm69piupdate.sh | 28 +
update | 2 +
33 files changed, 6265 insertions(+), 1384 deletions(-)
create mode 100644 emonhub-sudoers
create mode 100644 firmware/compiled/archive/emonPi_V2.7.hex
create mode 100644 firmware/compiled/archive/emonPi_V2.8.1.hex
create mode 100644 firmware/compiled/archive/emonPi_V2.8.2.hex
create mode 100644 firmware/compiled/archive/emonPi_V2.8.hex
create mode 100644 firmware/test_sketches/i2c_scan/.gitignore
create mode 100644 firmware/test_sketches/i2c_scan/.travis.yml
create mode 100644 firmware/test_sketches/i2c_scan/lib/readme.txt
create mode 100644 firmware/test_sketches/i2c_scan/platformio.ini
create mode 100644 firmware/test_sketches/i2c_scan/src/src.ino
create mode 100755 rfm69piupdate.sh
git pull /home/pi/RFM2Pi
error: insufficient permission for adding an object to repository database .git/objects
fatal: failed to write object
fatal: unpack-objects failed
git pull /home/pi/emonhub
From https://github.com/openenergymonitor/emonhub
afdb53e..9ab0f49 emon-pi -> origin/emon-pi
error: Your local changes to the following files would be overwritten by merge:
src/interfacers/EmonHubEmoncmsHTTPInterfacer.py
Please, commit your changes or stash them before you can merge.
Aborting
Updating afdb53e..9ab0f49
git pull /home/pi/oem_openHab
From https://github.com/openenergymonitor/oem_openHab
04c7caa..125ea33 master -> origin/master
Updating 04c7caa..125ea33
Fast-forward
Readme.md | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
git pull /home/pi/usefulscripts
Already up-to-date.
git pull /home/pi/huawei-hilink-status
error: cannot open .git/FETCH_HEAD: Permission denied
Start emonPi Atmega328 firmware update:
=================================
EmonPi update started
=================================
EUID: 1000
Collecting paho-mqtt
Downloading paho-mqtt-1.2.tar.gz (49kB)
Building wheels for collected packages: paho-mqtt
Running setup.py bdist_wheel for paho-mqtt: started
Running setup.py bdist_wheel for paho-mqtt: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/fa/db/fb/b495e37057e2f40534726b3c00ab26a58fc80fb8d17223df07
Successfully built paho-mqtt
Installing collected packages: paho-mqtt
Found existing installation: paho-mqtt 1.1
Uninstalling paho-mqtt-1.1:
Successfully uninstalled paho-mqtt-1.1
Successfully installed paho-mqtt-1.2
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Stopping OpenEnergyMonitor emonHub: emonhub has been stopped ok.
Start ATmega328 serial upload using avrdude with latest.hex
Discrete Sampling
avrdude -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex
avrdude-original: Using autoreset DTR on GPIO Pin 7
avrdude-original: stk500_getsync() attempt 1 of 10: not in sync: resp=0xfc
avrdude-original: stk500_getsync() attempt 2 of 10: not in sync: resp=0xe0
avrdude-original: stk500_getsync() attempt 3 of 10: not in sync: resp=0x8e
avrdude-original: stk500_getsync() attempt 4 of 10: not in sync: resp=0x1c
avrdude-original: stk500_getsync() attempt 5 of 10: not in sync: resp=0x8e
avrdude-original: stk500_getsync() attempt 6 of 10: not in sync: resp=0xe0
avrdude-original: stk500_getsync() attempt 7 of 10: not in sync: resp=0x0e
avrdude-original: stk500_getsync() attempt 8 of 10: not in sync: resp=0xf8
avrdude-original: stk500_getsync() attempt 9 of 10: not in sync: resp=0xe0
avrdude-original: stk500_getsync() attempt 10 of 10: not in sync: resp=0xe0
avrdude-original done. Thank you.
strace: |autoreset: Broken pipe
Starting OpenEnergyMonitor emonHub: emonhub has been started ok.
Start emonhub update script:
=================================
EmonPi update started
=================================
Running emonhub automatic node addition script
EUID: 1000
EUID: 1000
[[5]]
Node 5 already present
[[6]]
Node 6 already present
[[7]]
Node 7 already present
[[8]]
Node 8 already present
[[9]]
Node 9 already present
[[10]]
Node 10 already present
[[11]]
Node 11 already present
[[19]]
Node 19 already present
[[20]]
Node 20 already present
[[21]]
Node 21 already present
[[22]]
Node 22 already present
[[23]]
Node 23 already present
[[24]]
Node 24 already present
[[25]]
Node 25 already present
[[26]]
Node 26 already present
Start emoncms update:
=================================
Emoncms update started
Tue 17 Jan 07:28:37 UTC 2017
EUID: 1000
Checking cron tab for service runner entry...
service runner crontab entry already installed
/home/pi/emonpi/emonhub-sudoers: parsed OK
Install emonhub reboot sudoers entry
git pull /var/www/emoncms
dev-mosquitto-php
master
* stable
symlinked_modules
From https://github.com/emoncms/emoncms
dc70ba7..5aace47 stable -> origin/stable
0c52196..70192d8 master -> origin/master
* [new branch] revert-604-master_contribution -> origin/revert-604-master_contribution
* [new tag] 9.3.0 -> 9.3.0
* [new tag] 9.5.0 -> 9.5.0
* [new tag] 9.5.1 -> 9.5.1
* [new tag] 9.6.0 -> 9.6.0
* [new tag] 9.7.0 -> 9.7.0
* [new tag] 9.7.1 -> 9.7.1
* [new tag] 9.7.9 -> 9.7.9
error: Your local changes to the following files would be overwritten by merge:
Theme/basic/theme.php
Please, commit your changes or stash them before you can merge.
Aborting
Updating dc70ba7..5aace47
git pull /var/www/emoncms/Modules/app
* 9.0
Already up-to-date.
git pull /var/www/emoncms/Modules/config
* 9.0
From https://github.com/emoncms/config
8438a27..faabc47 9.0 -> origin/9.0
Updating 8438a27..faabc47
Fast-forward
README.md | 18 ++++++---
config.png | Bin 0 -> 471961 bytes
config_controller.php | 51 +++++++++++++++++--------
edit.php | 103 +++++++++++++++++++++-----------------------------
emonnhub-sudoers | 1 +
5 files changed, 91 insertions(+), 82 deletions(-)
create mode 100644 config.png
create mode 100644 emonnhub-sudoers
git pull /var/www/emoncms/Modules/wifi
* 9.0
From https://github.com/emoncms/wifi
8598aec..036e8ab 9.0 -> origin/9.0
Updating 8598aec..036e8ab
Fast-forward
view.html | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
git pull /var/www/emoncms/Modules/dashboard
environ-master
* master
From https://github.com/emoncms/dashboard
d47bcc0..0687caa master -> origin/master
Updating d47bcc0..0687caa
Fast-forward
Views/dashboard_edit_view.php | 37 ++-
Views/dashboard_view.php | 6 +-
Views/dashboardeditor.css | 62 +++++
Views/icons/Containers.png | Bin 0 -> 729 bytes
Views/icons/Text.png | Bin 0 -> 756 bytes
Views/icons/Visualisations.png | Bin 0 -> 926 bytes
Views/icons/Widgets.png | Bin 0 -> 958 bytes
Views/icons/emon-icon-back.png | Bin 3110 -> 519 bytes
Views/icons/emon-icon-delete.png | Bin 3404 -> 519 bytes
Views/icons/emon-icon-dial.png | Bin 5214 -> 0 bytes
Views/icons/emon-icon-frame.png | Bin 3616 -> 0 bytes
Views/icons/emon-icon-front.png | Bin 2605 -> 510 bytes
Views/icons/emon-icon-gear.png | Bin 5168 -> 493 bytes
Views/icons/emon-icon-plot.png | Bin 4814 -> 0 bytes
Views/icons/emon-icon-redo.png | Bin 2664 -> 494 bytes
Views/icons/emon-icon-save.png | Bin 1387 -> 0 bytes
Views/icons/emon-icon-text.png | Bin 1695 -> 0 bytes
Views/icons/emon-icon-tool.png | Bin 1764 -> 420 bytes
Views/icons/emon-icon-undo.png | Bin 2751 -> 483 bytes
Views/icons/emon-icon-view.png | Bin 4986 -> 552 bytes
Views/icons/gear-icon-outlined.png | Bin 17273 -> 2229 bytes
Views/js/designer.js | 4 +-
dashboard_langjs.php | 6 +
locale/es_ES/LC_MESSAGES/messages.mo | Bin 8208 -> 8583 bytes
locale/es_ES/LC_MESSAGES/messages.po | 214 +++++++++-------
widget/bar/bar_render.js | 484 ++++++++++++++++++-----------------
widget/curl/curl_render.js | 22 +-
widget/cylinder/cylinder_render.js | 31 +--
widget/dial/dial_render.js | 6 +-
widget/feedvalue/feedvalue_render.js | 12 +-
widget/jgauge/jgauge_render.js | 4 +-
widget/jgauge2/jgauge2_render.js | 8 +-
widget/windrose/windrose_render.js | 8 +-
33 files changed, 514 insertions(+), 390 deletions(-)
create mode 100644 Views/dashboardeditor.css
create mode 100644 Views/icons/Containers.png
create mode 100644 Views/icons/Text.png
create mode 100644 Views/icons/Visualisations.png
create mode 100644 Views/icons/Widgets.png
delete mode 100644 Views/icons/emon-icon-dial.png
delete mode 100644 Views/icons/emon-icon-frame.png
delete mode 100644 Views/icons/emon-icon-plot.png
delete mode 100644 Views/icons/emon-icon-save.png
delete mode 100644 Views/icons/emon-icon-text.png
git pull /var/www/emoncms/Modules/graph
* master
From https://github.com/emoncms/graph
84beb45..3d18ee5 master -> origin/master
Updating 84beb45..3d18ee5
Fast-forward
graph.js | 67 ++++++++++++++++++++++++++++++++++++++++++++++++----------------
view.php | 19 +++++++++---------
2 files changed, 60 insertions(+), 26 deletions(-)
git pull /home/pi/postprocess
Already on 'emonpi'
Your branch is up-to-date with 'remotes/origin/emonpi'.
Already up-to-date.
git pull /home/pi/backup
* master
From https://github.com/emoncms/backup
00617c2..f86ed05 master -> origin/master
Updating 00617c2..f86ed05
Fast-forward
backup/backup_view.php | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
replacing initd mqtt_input with systemd mqtt input
Stopping Daemon for the emoncms MQTT script: mqtt_input.
cp: cannot stat ā/var/www/emoncms/scripts/mqtt_input.serviceā: No such file or directory
Failed to execute operation: No such file or directory
Failed to start mqtt_input.service: Unit mqtt_input.service failed to load: No such file or directory.
Update Emoncms database
["ALTER TABLE dashboard MODIFY `main` tinyint(1) Default '0'","ALTER TABLE dashboard MODIFY `public` tinyint(1) Default '0'","ALTER TABLE dashboard MODIFY `published` tinyint(1) Default '0'","ALTER TABLE dashboard MODIFY `showdescription` tinyint(1) Default '0'"]
Restarting Services...
Restarting OpenEnergyMonitor emonHub: emonhub has been restarted ok.
Log is turned off
Restarting feedwriter
Restarting openhab (via systemctl): openhab.service.
Failed to restart mqtt_input.service: Unit mqtt_input.service failed to load: No such file or directory.
set log rotate config owner to root
Restarting Services...
Starting emonPi LCD service..
mount: / is busy
Filesystem is locked - Read Only access
type ' rpi-rw ' to unlock
Tue 17 Jan 07:29:19 UTC 2017
...................
emonPi update done
I have bumped your user level up a notch so you can attach the logs in-line. We try to avoid the use of links in place of log files as if the source disappears the link is broken and the thread incomplete, so I have copied the log to your post.
Any suggestions how to go about this? I want to make the image pre-compiled, so that I dont need to do any extra work after booting.
Maybe I can edit where those pull requests are coming from, and just point them to my own repo, and I then need to manually keep the 2 synced?
Where is this update script located, and from where is it called on start-up?
That should work but you need to keep the āotherā repo bang up to date, using this same example if your repo didnāt contain the mqtt_input.service file then it would have still fallen over, so you need to ensure any upstream changes are merged before any incomplete update routines can be run, unless you duplicate all the repos to retain control.
Even with the āotherā repo you would still be vulnerable to changes in the main emonpi update script over ruling your changes, for example the backup module pull has a checkout command, if a checkout gets added to the emoncms update command then the update would also switch branches and your local changes discarded until you manually checkout your āotherā branch again.
The update routine is quite involved but specifically at first boot the firstbootupdate script is run by rc.local at boot and that triggers an update if there is no pre-exising update.log.
Then the main update is done by service-runner-update.sh and that in turn uses the emoncmsupdate emonhubupdate and emonpiupdate scripts
PR submitted to avoid the mqtt_input daemon update assuming the git pull to the emoncms repo was successful when it may not have been and therefore the file is missing.
Another question, does the filesystem expand function run during first-boot or something? I mounted a original unmodified emonpi image to a mem-card, and the saved it back to pc, but now its almost 8gigs.
Anyway to preserve the āsmallā sizing?
The actual content of the image is about 3.8 gigs.
Problem is getting a saved image back to pc once its been burned to a memory card. The image then becomes the size of memory card, regardless if it booted or not.
Perhaps @TrystanLea or @glyn.hudson could comment how they build or modify prepared images?
The usual method is to use dd to copy a set size as, yes, a disk or image copy will be dictated by the size of the card. A quick and dirty method might be to use a 4gb sdcard to carry out your personalisations and then save that 4gb image for use on 8gb SDcards.
There are a couple of tricks to really squash the image for delivery;
Before you DD the image off the card;
Update the system and clean up
Donāt expand the SD card
apt-get clean all
Do all your mods
Use DD on the booted Pi, write a file full of zeros and then delete it - do this on every partition.
dd if=/dev/zero of=/file.out bs=1M count=4096
rm -rf /file.out
Shutdown cleanly and DD the card to an image file - this will be the same size as the whole card - this is normalā¦ the reason you DONT expand the SD image on the Pi - is so that once we are done with the next step it will fit on 4G cards stillā¦
Thanks for the solid reply Andy!
Whats the purpose of āwrite a file full of zeros and then delete itā? Can I do this from through the ssh on the booted device? How would I access the other partitions then?
I have not used DD before, please forgive if questions donāt make sense!
The reason for the dd to write zeros all over the CF card is quite simple and quite complex all at once.
Most filesystems on a computer, donāt actually delete a file when you tell them to, they remove the pointers so that the system doesnāt know that the file is there any more, they mark the blocks as available and thats it - the actual data is still there (this is why file recovery software worksā¦ well so long as the blocks have not been over writtenā¦).
So when you use dd to take an image of the card - it does so at the block level, including all the crap you have written before and deleted again (or changed jn the case of anything that ever changed everā¦).
When you compress the image - you also retain and compress all of the garbage with it, but its toast - we donāt need that, and this is where writing zeros all over the card comes inā¦
so now we make a giant file, full of zeros to deliberately fill up the CF card (with zeros) and then delete it (leaving those zeros behind rememberā¦).
now when you take the image of the card, those zeroās are still copied (because its a block level copy) BUT, when you come to zip (compress) the image ready for deployment, zeros compress to nothing - where miscellaneous binary crap - doesnāt.
I hope thats a good enough explination - if not please say so and Iāll see if I can help you understand whatever you didnāt quite get.
