I understood this to have already been implemented and tested
You must disable auto-restart at least in a separate “debugging” branch at the very least so that faults are easily noticed and not swept under the carpet, based on the traceback content we can (hopefully?) quickly add minor fixes to improve/implement sanitation checks so that the released version is not dependent on the restarts for normal running.
Another option in lieu of disabling the auto-restart for debugging is to add an email trigger so that rather than (re)logging the last 100 log entries elsewhere, it emails the last 100 entries to the user, this means we can return to a single logfile and there is also an alert to the user so that faults get reported and dealt with at the time.
It is not helpful to repeatedly send all this data silently to a 2nd logfile and then try and debug some months later when things eventually grind to halt, especially as you will now undoubtedly focus on preventing a condition arising that requires a reboot. Plus there is no better time to tackle an issue than when it happens as the user can give further info and context.
Sending an email in python is relatively straight forward and once implemented can be used for other things, I would not want to depend on nodered or any other external services.
As for the tracebacks, it appears the @log_exceptions_from_class_method decorator wraps the run function with this code
https://github.com/openenergymonitor/emonhub/blob/emon-pi/src/interfacers/emonhub_interfacer.py#L34-L42
Personally I would have move the content of run to another function eg _run and called that new function from within a try/except in run eg something along the lines of
def run(self):
try:
self._run(self)
except:
if self.stop:
self._log.debug("Stopping "+self.name+" interfacer)
else:
self._log.warning("Exception caught in "+self.name+" thread. "+traceback.format_exc())
using the self.stop in the exception would hopefully filter the crashes from the controlled stops. I haven’t tested this, it is just a thought at this point. I am no longer familiar enough with the emonhub code to know if this would definitely work.