Newbie - probably non-standard installation - where to go from here

hpeyerl · 29 October 2019 12:35

Hi,

I have an older Pi2 and bought an EmonTXv3. I created an SD card from emonSD-17Oct19 which boots fine. Because I didn’t buy an EmonPi and my Pi2 doesn’t have Wifi, I decided to wire the serial port on the pi directly to the EmonTXv3.

One thing of note is that the firstbootupdate process failed. There were lots of corrupt git repositories on the SD card image. Example:

Updating /opt/emoncms/modules/backup module

git branch: stable
git tags: 2.0.0-34-g993f3ac
no local changes
running: git pull origin stable

error: object file .git/objects/c8/94afc436443773ed8a9b3707e45406b10aac71 is empty
error: object file .git/objects/c8/94afc436443773ed8a9b3707e45406b10aac71 is empty
fatal: loose object c894afc436443773ed8a9b3707e45406b10aac71 (stored in .git/objects/c8/94afc436443773ed8a9b3707e45406b10aac71) is corrupt
fatal: the remote end hung up unexpectedly
Already on ‘stable’
Your branch is up to date with ‘origin/stable’.

Updating /opt/emoncms/modules/demandshaper module

git branch: stable
error: object file .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3 is empty
error: object file .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3 is empty
fatal: loose object 1630622c688469b6af2d8ea13605310e9a37ece3 (stored in .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3) is corrupt
git tags:
error: object file .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3 is empty
error: object file .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3 is empty
fatal: loose object 1630622c688469b6af2d8ea13605310e9a37ece3 (stored in .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3) is corrupt
no local changes
running: git pull origin stable

error: object file .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3 is empty
fatal: loose object 1630622c688469b6af2d8ea13605310e9a37ece3 (stored in .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3) is corrupt
error: object file .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3 is empty
error: object file .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3 is empty
fatal: loose object 1630622c688469b6af2d8ea13605310e9a37ece3 (stored in .git/objects/16/30622c688469b6af2d8ea13605310e9a37ece3) is corrupt

I had to go through a few iterations of removing modules and so forth, cloning and checking out stable before I could get the firstbootupdate process to finish successfully.

After that, I had trouble because there was no settings.ini and so forth.

Anyway, after a bit of googling and experimentation, I believe I have the system up and running. I was able to create a user on the web frontend.

I wired up the EmonTXv3 to the serial ports (looks like the silk-screen on the EmonTxv3 has the Tx and Rx labels backwards. Thankfully I buzzed the pins with my meter first to confirm they’re 3v3). I was able to confirm via minicom that the EmonTx was sending records:

ct1:0,ct2:0,ct3:0,ct4:0,vrms:11231,pulse:0

and I was able to get Emonhub configured to read those records:

2019-10-29 12:16:11,635 DEBUG Serial 122 NEW FRAME : ct1:0,ct2:0,ct3:0,ct4:0,vrms:11234,pulse:0
2019-10-29 12:16:11,636 DEBUG Serial 122 Timestamp : 1572351371.63
2019-10-29 12:16:11,638 DEBUG Serial 122 From Node : 0
2019-10-29 12:16:11,639 DEBUG Serial 122 Values : [0, 0, 0, 0, 11234, 0]
2019-10-29 12:16:11,640 DEBUG Serial 122 Sent to channel(start)’ : ToEmonCMS
2019-10-29 12:16:11,641 DEBUG Serial 122 Sent to channel(end)’ : ToEmonCMS
2019-10-29 12:16:11,688 DEBUG MQTT Publishing: emon/0/ct1 0
2019-10-29 12:16:11,691 DEBUG MQTT Publishing: emon/0/ct2 0
2019-10-29 12:16:11,694 DEBUG MQTT Publishing: emon/0/ct3 0
2019-10-29 12:16:11,697 DEBUG MQTT Publishing: emon/0/ct4 0
2019-10-29 12:16:11,700 DEBUG MQTT Publishing: emon/0/vrms 11234
2019-10-29 12:16:11,703 DEBUG MQTT Publishing: emon/0/pulse 0
2019-10-29 12:16:11,706 INFO MQTT Publishing: emonhub/rx/0/values 0,0,0,0,11234,0

(my CT’s aren’t connected to anything yet).

So I find myself asking “now what?”. Questions:

In the web frontend it looks like I need to create an input but I don’t know how to bind the serial ‘node’ from EmonHub to EmonCMS?
I have the temperature sensor plugged into the RJ-45 on the EmonTX but I don’t see the temperature reading in the serial frame. How does that information get transmitted?

My eventual goal is to get this information into my HomeAssistant instance which I think I’ll probably do via MQTT so that’s no problem. But using EmonCMS would be nice as well. I hope to eventually buy a couple more EmonTX’s because my acreage has 3 panels in 2 different places.

borpin · 29 October 2019 12:57

It shouldn’t make a difference really.

What size is the SD card?

hpeyerl · 29 October 2019 13:05

32GB.

Looks like it didn’t expand after boot but nothing is at capacity.

179 0 31342592 mmcblk0
179 1 262144 mmcblk0p1
179 2 4213760 mmcblk0p2
179 3 10634240 mmcblk0p3

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  1.9G  2.0G  49% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           488M     0  488M   0% /dev/shm
tmpfs           488M   56M  433M  12% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           488M     0  488M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            30M     0   30M   0% /tmp
/dev/mmcblk0p3   10G  3.8M  9.5G   1% /var/opt/emoncms
/dev/mmcblk0p1  253M   53M  200M  21% /boot
log2ram          50M  4.0M   47M   8% /var/log
tmpfs            98M     0   98M   0% /run/user/1000

borpin · 29 October 2019 13:10

No it won’t. There is another thread about that. (emonSD-17Oct19 Image partition sizes - #6 by borpin) The only solution is to include the changes and install from the Scripts directly.

It looks like you had a bad flash.

I’m not an expert on emonhub so cannot really help there. Have you searched the forum?

Posting the current emonhub.config will help.

Please put 3 ‘backticks’ ``` (top left of keyboard) on a line before and a line after to format code.

hpeyerl · 29 October 2019 13:40

ok. Well, I don’t need the full available space anyway. That’s less important than just getting things into EmonCMS for me.

Here’s my emonhub.conf:

    Type = EmonHubTx3eInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        com_baud = 115200                        #
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,

I guess my issue is I don’t know how this is all supposed to fit together. It looks like “ToEmonCMS” should just cause my samples to end up in EmonCMS somehow…

Anyway, I’ll leave that up there for whoever else might have some input.

borpin · 29 October 2019 13:45

Is that the whole file? Should be more than that.

Here are the configuration docs emonhub/configuration.md at emon-pi · openenergymonitor/emonhub · GitHub

Robert.Wall · 29 October 2019 16:29

You are correct. They are labelled opposite to convention. i.e., Tx is looking for Tx and Rx is looking for Rx. The crossover actually happens on the emonTx PCB. I can’t explain why it was done like that.

You don’t create inputs, they are created automatically when the first data arrives. To set up emonHub for serial input, you need to look here:
https://wiki.openenergymonitor.org/index.php/EmonTx_V3.4#Direct_connection_emonTx_V3_.3E_Raspberry_Pi_GPIO
(And the Serial terminal labelling is also mentioned there.)

hpeyerl · 29 October 2019 19:02

sorry no. I just pasted the part that is different from the distribution. I can paste the entire file when I get home later.

borpin · 29 October 2019 19:34

Ok, that is fine. Have a look at the docs and see if that solves the issue.

hpeyerl · 30 October 2019 11:05

Ok. So it looks like I figured out how to do the direct serial the hard way instead of finding that wiki page.

I just connected a CT and clipped it around the hot line of a table lamp and emonhub tells me it’s seeing data:

2019-10-30 10:59:53,506 DEBUG    Serial     8382 NEW FRAME : ct1:91,ct2:0,ct3:0,ct4:0,vrms:11222,pulse:0
2019-10-30 10:59:53,508 DEBUG    Serial     8382 Timestamp : 1572433193.51
2019-10-30 10:59:53,509 DEBUG    Serial     8382 From Node : 0
2019-10-30 10:59:53,510 DEBUG    Serial     8382    Values : [91, 0, 0, 0, 11222, 0]
2019-10-30 10:59:53,511 DEBUG    Serial     8382 Sent to channel(start)' : ToEmonCMS
2019-10-30 10:59:53,512 DEBUG    Serial     8382 Sent to channel(end)' : ToEmonCMS
2019-10-30 10:59:53,542 DEBUG    MQTT       Publishing: emon/0/ct1 91
2019-10-30 10:59:53,544 DEBUG    MQTT       Publishing: emon/0/ct2 0
2019-10-30 10:59:53,547 DEBUG    MQTT       Publishing: emon/0/ct3 0
2019-10-30 10:59:53,549 DEBUG    MQTT       Publishing: emon/0/ct4 0
2019-10-30 10:59:53,552 DEBUG    MQTT       Publishing: emon/0/vrms 11222
2019-10-30 10:59:53,554 DEBUG    MQTT       Publishing: emon/0/pulse 0
2019-10-30 10:59:53,556 INFO     MQTT       Publishing: emonhub/rx/0/values 91,0,0,0,11222,0

and it did appear to create an input but there doesn’t seem to be any data being logged to that input.

When I try to create a feed I get an error:

ERROR: Feed could not be created, could not write meta data file fopen(/var/lib/phpfina/5.meta): failed to open stream: No such file or directory

Maybe I need to tear this down and try to create a new flash card. This one was clearly corrupt from the beginning.

borpin · 30 October 2019 11:38

It did seem not to be overly happy.

I’d get a Raspbian Lite image andtailor the install to suit you so leave out the WiFi stuff.

It will also mean Raspbian sets itself up correctly for the Pi you have.

Keep your config file for emonhub .

hpeyerl · 31 October 2019 11:27

Alright, I’m back to where I was before. There was no corruption. I did have to:

mkdir /var/lib/phpfina
chown www-data:www-data !$

emonhub is getting samples and sending them to emoncms as above. emoncms found an input. I created a feed from said input but nothing is showing up on the graph as though the data is not being kept.

The last 3 samples from my emonhub.log:

2019-10-31 11:25:36,978 DEBUG    Serial     157 NEW FRAME : ct1:96,ct2:0,ct3:0,ct4:0,vrms:11374,pulse:0
2019-10-31 11:25:36,980 DEBUG    Serial     157 Timestamp : 1572521136.98
2019-10-31 11:25:36,982 DEBUG    Serial     157 From Node : 0
2019-10-31 11:25:36,983 DEBUG    Serial     157    Values : [96, 0, 0, 0, 11374, 0]
2019-10-31 11:25:36,984 DEBUG    Serial     157 Sent to channel(start)' : ToEmonCMS
2019-10-31 11:25:36,985 DEBUG    Serial     157 Sent to channel(end)' : ToEmonCMS
2019-10-31 11:25:37,082 DEBUG    MQTT       Publishing: emon/0/ct1 96
2019-10-31 11:25:37,084 DEBUG    MQTT       Publishing: emon/0/ct2 0
2019-10-31 11:25:37,086 DEBUG    MQTT       Publishing: emon/0/ct3 0
2019-10-31 11:25:37,089 DEBUG    MQTT       Publishing: emon/0/ct4 0
2019-10-31 11:25:37,091 DEBUG    MQTT       Publishing: emon/0/vrms 11374
2019-10-31 11:25:37,093 DEBUG    MQTT       Publishing: emon/0/pulse 0
2019-10-31 11:25:37,095 INFO     MQTT       Publishing: emonhub/rx/0/values 96,0,0,0,11374,0
2019-10-31 11:25:46,894 DEBUG    Serial     158 NEW FRAME : ct1:96,ct2:0,ct3:0,ct4:0,vrms:11378,pulse:0
2019-10-31 11:25:46,895 DEBUG    Serial     158 Timestamp : 1572521146.89
2019-10-31 11:25:46,896 DEBUG    Serial     158 From Node : 0
2019-10-31 11:25:46,896 DEBUG    Serial     158    Values : [96, 0, 0, 0, 11378, 0]
2019-10-31 11:25:46,897 DEBUG    Serial     158 Sent to channel(start)' : ToEmonCMS
2019-10-31 11:25:46,898 DEBUG    Serial     158 Sent to channel(end)' : ToEmonCMS
2019-10-31 11:25:47,193 DEBUG    MQTT       Publishing: emon/0/ct1 96
2019-10-31 11:25:47,195 DEBUG    MQTT       Publishing: emon/0/ct2 0
2019-10-31 11:25:47,197 DEBUG    MQTT       Publishing: emon/0/ct3 0
2019-10-31 11:25:47,199 DEBUG    MQTT       Publishing: emon/0/ct4 0
2019-10-31 11:25:47,201 DEBUG    MQTT       Publishing: emon/0/vrms 11378
2019-10-31 11:25:47,203 DEBUG    MQTT       Publishing: emon/0/pulse 0
2019-10-31 11:25:47,205 INFO     MQTT       Publishing: emonhub/rx/0/values 96,0,0,0,11378,0
2019-10-31 11:25:54,587 DEBUG    emoncmsorg Buffer size: 3
2019-10-31 11:25:56,788 DEBUG    Serial     159 NEW FRAME : ct1:97,ct2:0,ct3:0,ct4:0,vrms:11374,pulse:0
2019-10-31 11:25:56,789 DEBUG    Serial     159 Timestamp : 1572521156.79
2019-10-31 11:25:56,790 DEBUG    Serial     159 From Node : 0
2019-10-31 11:25:56,791 DEBUG    Serial     159    Values : [97, 0, 0, 0, 11374, 0]
2019-10-31 11:25:56,792 DEBUG    Serial     159 Sent to channel(start)' : ToEmonCMS
2019-10-31 11:25:56,793 DEBUG    Serial     159 Sent to channel(end)' : ToEmonCMS
2019-10-31 11:25:56,884 DEBUG    MQTT       Publishing: emon/0/ct1 97
2019-10-31 11:25:56,887 DEBUG    MQTT       Publishing: emon/0/ct2 0
2019-10-31 11:25:56,889 DEBUG    MQTT       Publishing: emon/0/ct3 0
2019-10-31 11:25:56,891 DEBUG    MQTT       Publishing: emon/0/ct4 0
2019-10-31 11:25:56,894 DEBUG    MQTT       Publishing: emon/0/vrms 11374
2019-10-31 11:25:56,896 DEBUG    MQTT       Publishing: emon/0/pulse 0
2019-10-31 11:25:56,899 INFO     MQTT       Publishing: emonhub/rx/0/values 97,0,0,0,11374,0

borpin · 31 October 2019 12:05

Did you use the import process?

I think I know why. Need to investigate.

hpeyerl · 31 October 2019 12:13

I think I know why. The “default-settings.ini” file has “/var/lib/phpfina” the “example.settings.ini” has “/var/opt/emoncms/phpfina”. The latter is what actually exists. So I copied “default-settings.ini” to “settings.ini” which caused the problem.

Since I posted that, I’ve changed it to /var/opt/emoncms/phpfina and recreated my feeds/processes.

Still not getting anything logged though.

borpin · 31 October 2019 12:24

Ah, no that is not the right thing to do.

The default-settings.ini is just that. You then have a more personalised settings.ini file that overrides some parts of the default.

You should not need to edit either on a standard install. The fact you have, may prevent any further updates as git will complain.

The problem actually is that there is probably a settings.php in the folder and that takes precedence.

So what you want is to

cp example.settings.ini settings.ini
mv settings.php settings.php.old

You need to use the /var/opt/ folder as this is optimised for multiple writes of data.

This may also solve other problems as the proper settings.ini will have other changes required as well.

hpeyerl · 31 October 2019 12:29

Question about inputs/feeds.

In emonhub.conf, I have:

[[Serial]]
    Type = EmonHubTx3eInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        com_baud = 115200                        #
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,
.....
[nodes]
[[99]]
    nodename = emontx3-direct-serial
    [[[rx]]]
       names = power1, power2, power3, power4, vrms
       datacode = 0      # not essential as "0" is default datacode for serial interfacer
       scale = 1         # not essential as "1" is default scale for serial interfacer
       units =W,W,W,W,V

and in http://emonpi.local/input/view, I have:

Node: 99
Name: emontx3-direct-serial
Location: Main_Panel

If I click ‘initialize’ it seems to create 3 inputs under 99, that being a “use”, a “use_kwh”, and a “Lamp” (I named it “Lamp” at some point and now it seems to have stuck)… So in my simplistic view, it looks like it should link “power1” (which is Lamp) to node 99 and the data should be recorded. But I don’t know if this is how it’s supposed to work?

My graph(s) are still empty. Feeling kind of dumb here.

borpin · 31 October 2019 12:32

~~I think the maximum node number you can use is 32.~~ Ignore that, you are using MQTT to pass the data.

What does your ‘Inputs’ page look like?

hpeyerl · 31 October 2019 12:42

I have MQTT because it was already there by default but also because I figure that’s how I will get the data into my HomeAssistant but that’s for later. I don’t care about that right now.

I only have ‘99’ because that’s what’s in my emonhub.conf which I got from the bottom of this README.md:

borpin · 31 October 2019 12:45

Because this says null, no data is getting to the input

So not going to the Feed.

Have you fixed your settings.ini (as per above)? It might be the MQTT credentials are wrong for emoncms to use.

hpeyerl · 31 October 2019 12:58

There was no ‘settings.php’. I guess that’s good.
I copied example.settings.ini to settings.ini but:
2a) mqtt was “enabled=false” so I set it to true. I systemctl started emoncms_mqtt.service
2b) sql password was “password” instead of “emonpiemoncmsmysql2016” from my old settings.ini and thus the web frontend couldn’t connect to the database.

No change in behavior however. Still have a red “n/a " and NULL” in inputs.