Thanks @bwduncan the install.sh script did the trick. All working now! great 
3 Likes
Did you need to pass it a parameter?
Just for completeness, this was what I did to get it running on my emonpi;
cd /opt/openenergymonitor/emonhub
git remote add bwduncan https://github.com/bwduncan/emonhub.git
git fetch bwduncan
git checkout bwduncan/python3
sudo apt-get update
sudo apt-get install -y python3-serial python3-configobj python3-pip
sudo pip3 install paho-mqtt requests
sudo systemctl restart emonhub.service
To go back you simply need to
git checkout origin/emon-pi
sudo systemctl restart emonhub.service
1 Like
no, no parameter required, just ran ./install.sh
1 Like
@bwduncan, this all seems to be running OK but the logs are rotated too quickly to be sure.
I’m wondering if we could add a configuration element to disable the log rotation (it could then be handled by a logrotate configuration if necessary).
This option would default to rotating the logs (if not present).
You could also possibly provide config elements that would allow a user to configure the size and number of rotated logs.
The advantage of using logrotate to rotate the logs, is that it can move them to another folder and also compress the log files.
I know I can do this manually by not specifying a log file then parsing the syslog output, but this would be cleaner and anyone could use it.
If at a later date there was a requirement to debug emonhub, getting complete logs would be easier.
HI @borpin,
There are system tools which will do this much better and easier than doing it in python.
- journald is probably the easiest. Just write the logs to stderr and let systemd store and rotate them.
- rsyslog can be made to take the output and put it in a specific file and do the log rotation.
Both are already installed and would need minimal config to get it to do what we have now, but better.
Thoughts?
Bruce
Emonhub already does the rotation itself, it works well and does not need any other interventions to ensure the logs don’t get out of hand. we had some issues with this a while back when it wasn’t working and the logs were filling up so no, I’m not suggesting getting rid of the mechanism.
What I was looking for was a means to disable this internal rotation for specific purposes such as creating a fuller log for this testing or debugging. It would then be the responsibility of the person disabling the log rotation to manage the logs.
Just digging into this (and I am by no means a python expert), but I think there is a logic error in the code wrt logging - 2 syslog handlers could be created depending on the logfile argument being present or not and the use syslog setting.
Does the second syslog setup simply overwrite the first so there is only one syslog handler?
I was thinking of adding a setting ROTATE_LOGFILE = yes (default) to the conf file
if log file argument
if rotate log setting exists AND is no
Create non-rotating log file handler
else
create rotating log file handler
else
create syslog
end if
That’s 404. I think it should be
https://github.com/bwduncan/emonhub/blob/emon-pi/src/emonhub.py#L281-L298
1 Like
Odd, it is different to the one on my system…
Ah, I’d picked up the wrong branch with my link. You have changed the log file logic then?
Argh I’m losing track of the edits here lol.
The setup is somewhat confusing, with configuration coming from the arguments, the config file and static bits of code. Hence my proposal to tidy it up a bit.
Currently the code will log to (FILE or STDERR) and SYSLOG depending on which options are set. Since stderr will go to journald then yes it’s possible to have syslogging twice.
1 Like
I’m confused as well. The directory listing for the file says the last change was last year! So …
So I think, check for a config file first, then if a logfile is to be created, look for a config item to see if the logfile should be set up as rotated or not.
Personally, I’m with Bruce. We should be moving towards removing all this logging code and let journald handle it all. I don’t think there’s anything we need that it can’t do.
1 Like
I agree. I’ll add that to my pull request soon.
The only potential advantage to having logging set up before reading the config file is that you get an error if parsing the config file fails, but if journald is logging everything anyway then the exception should make the problem clear!
NO. It is there so it is fire and forget on any system. It has been discussed before and there was a firm decision not to remove it, so it will stay.
It will not be accepted. The internal log rotation will stay.
Search on the forum for var log full and you will see the problems caused when it was (due to a configuration error) not being used.
What I can support is the option to enable a non-rotating log (ideal to get a user to set temporarily for debugging purposes) and a proper use_syslog setting that does only use syslog.
Still not clear if the current code can create 2 syslog handlers or not.
It’s attitudes like that that put me off getting involved with OEM.
No, it is the fact this has been discussed ad-infinitum, search the forum if you do not believe me, and there has been a firm decision for emonhub to rotate the logs by default and not rely on external systems to do so. Those external systems change which can then result in an unstable system, as has happened in the not too distant past.
It is a deliberate design decision that has been made and there is no intention in spending time discussing it again.
What Bruce (@bwduncan) has done in converting the script to Python 3 is fantastic, way outside my skill set, but how the logs are rotated is not up for discussion (again).
TBF I was originally in favour of letting rsyslog do the work, but understood and accepted the rationale behind using the Python rotation mechanism.
3 Likes
OK if we can limit the flexibility (and scope, therefore combinatoric arrangements) of the python I’ll be happy. Given that it can do file (with built-in rotation), stderr (to journald as it’s configured right now) and syslog (at the same time as the other two) I’m going to leave it alone for now in the interests of getting the python3 changes merged.