Hmm… none of those timestamps align particularly with the fsck ones in your previous post. Maybe check the syslog around the time of those fsck entries - you may have to scan through it using less (or more if you’re so inclined) to see if there’s anything obvious related to apache failing to start. There may have been a dependency that failed too, which wouldn’t show up as “apache2” in your search.
@djh
I tried yr suggestion …
pi@emonpi-node-13:~ $ systemd-analyze blame
3min 1.530s [email protected]
9.393s dhcpcd.service
5.529s mariadb.service
3.664s dev-sda1.device
2.887s apt-daily.service
2.180s dphys-swapfile.service
1.824s rc-local.service
1.730s networking.service
1.463s redis-server.service
1.417s lwrfd.service
1.411s raspi-config.service
1.365s rsyslog.service
1.265s emonPiLCD.service
1.100s avahi-daemon.service
1.058s dnsmasq.service
928ms plymouth-start.service
798ms keyboard-setup.service
797ms ssh.service
668ms phpsessionclean.service
636ms alsa-restore.service
629ms systemd-logind.service
595ms mosquitto.service
533ms [email protected]
438ms apache2.service
428ms systemd-timesyncd.service
412ms rng-tools.service
386ms systemd-fsck-root.service
377ms home-pi-data.mount
375ms fake-hwclock.service
368ms systemd-tmpfiles-setup-dev.service
359ms triggerhappy.service
337ms systemd-udevd.service
334ms systemd-udev-trigger.service
334ms systemd-tmpfiles-setup.service
331ms wifi-country.service
305ms dev-mqueue.mount
292ms run-rpc_pipefs.mount
288ms systemd-journald.service
228ms systemd-modules-load.service
201ms systemd-rfkill.service
186ms ufw.service
183ms console-setup.service
179ms nfs-config.service
173ms systemd-tmpfiles-clean.service
150ms sys-kernel-debug.mount
131ms systemd-sysctl.service
103ms systemd-journal-flush.service
102ms systemd-update-utmp.service
93ms plymouth-read-write.service
80ms [email protected]
77ms systemd-remount-fs.service
70ms systemd-user-sessions.service
68ms var-tmp.mount
65ms systemd-random-seed.service
58ms kmod-static-nodes.service
50ms tmp.mount
37ms plymouth-quit.service
29ms systemd-update-utmp-runlevel.service
26ms sys-kernel-config.mount
25ms plymouth-quit-wait.service
24ms boot.mount
lines 39-61/61 (END)
But don’t know how to interpret the resuts.
Anything obvious to you?
I interpret that as 3 minutes and 1.530 seconds to fsck /dev/sda2 and 533 milliseconds to do /dev/mmcblk0p1
That is a cracking command - not found that before.
It looks from the blame output that apache did start (438ms).
Note that strictly speaking you should use systemctl not service to restart a unit now.
If it happens again, do a systemctl status apache2 (before you restart) and you should see the last few lines of the log which may tell you why it failed.
I had an issue with running the http interface to a package called monit. Although the unit itself started, the http interface always failed. Tried loads of different wants and requires but in the end, after some digging I worked out how to put a timer unit in so it delayed the startup. Note that you no longer enable the service file (as in sudo systemctl enable unit.service) but enable the timer (as in sudo systemctl enable unit.timer).
Although the referenced error isn’t a show stopper, adding
ServerName localhost
to your apache.conf file will make that error go away.
Actually, that was the command I remembered but systemd-analyze critical-chain might be more useful. The man page is at systemd-analyze
2 Likes
It did happen again - entirely my doing.
After I rebooted, I could puTTy in but not browse to the web interface.
And remembering yr suggestion, I did …
pi@emonpi-node-13:~ $ systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; disabled; vendor preset: enabled)
Active: inactive (dead)
lines 1-3/3 (END)
This gave me no clues …
pi@emonpi-node-13:/ $ cat /lib/systemd/system/apache2.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
Environment=APACHE_STARTED_BY_SYSTEMD=true
ExecStart=/usr/sbin/apachectl start
ExecStop=/usr/sbin/apachectl stop
ExecReload=/usr/sbin/apachectl graceful
PrivateTmp=true
Restart=on-abort
[Install]
WantedBy=multi-user.target
I then did …
pi@emonpi-node-13:~ $ systemd-analyze blame
3min 1.665s [email protected]
9.410s dhcpcd.service
5.157s mariadb.service
3.774s dev-sda1.device
2.733s dphys-swapfile.service
1.656s rc-local.service
1.600s redis-server.service
1.406s raspi-config.service
1.275s lwrfd.service
1.090s plymouth-start.service
1.018s triggerhappy.service
959ms keyboard-setup.service
947ms rsyslog.service
902ms dnsmasq.service
845ms networking.service
721ms avahi-daemon.service
671ms ssh.service
600ms phpsessionclean.service
556ms systemd-logind.service
556ms mosquitto.service
519ms emonPiLCD.service
475ms systemd-timesyncd.service
458ms alsa-restore.service
457ms systemd-tmpfiles-setup.service
456ms fake-hwclock.service
443ms [email protected]
369ms rng-tools.service
360ms systemd-udevd.service
353ms systemd-fsck-root.service
343ms systemd-tmpfiles-setup-dev.service
324ms wifi-country.service
314ms sys-kernel-debug.mount
313ms systemd-udev-trigger.service
310ms run-rpc_pipefs.mount
308ms systemd-journald.service
306ms dev-mqueue.mount
300ms home-pi-data.mount
263ms kmod-static-nodes.service
252ms systemd-modules-load.service
231ms console-setup.service
216ms nfs-config.service
189ms systemd-rfkill.service
162ms systemd-tmpfiles-clean.service
161ms systemd-sysctl.service
154ms ufw.service
117ms systemd-update-utmp.service
81ms systemd-remount-fs.service
80ms [email protected]
77ms systemd-random-seed.service
75ms systemd-user-sessions.service
75ms systemd-journal-flush.service
75ms plymouth-read-write.service
48ms tmp.mount
34ms var-tmp.mount
33ms plymouth-quit.service
31ms plymouth-quit-wait.service
30ms systemd-update-utmp-runlevel.service
24ms boot.mount
15ms sys-kernel-config.mount
lines 37-59/59 (END)
Expectedly, the fsck on sda2 (the ext2 /home/pi/data partition on USB HDD) takes a long time.
As an aside perhaps that would better be ext4 - it’s an HDD not a ‘sensitive’ SDHC ??? And ext4 fsck’s seem much faster.
But I don’t see apache2 in the list.
So I checked the logs around the relevant time … apart from a note that apache2 was starting, I could see no other relevant references.
The ‘inelegant’ but pragmatic fix seems to be - at the end of rc.local add a few seconds sleep and then sudo systemctl restart apache2
I’ll give that a try.
All I can think is that it is not enabled to start at boot.
systemctl list-unit-files | grep enabled
lists all enabled units.
If it is not there
sudo systemctl enable apache2.service
You need then to start it, if it is not started.
It’s not there …
pi@emonpi-node-13:/ $ systemctl list-unit-files | grep enabled
[email protected] enabled
avahi-daemon.service enabled
bluetooth.service enabled
console-setup.service enabled
cron.service enabled
dbus-org.bluez.service enabled
dbus-org.freedesktop.Avahi.service enabled
dhcpcd.service enabled
dhcpcd5.service enabled
dnsmasq.service enabled
emoncms_mqtt.service enabled
emonhub.service enabled
fake-hwclock.service enabled
feedwriter.service enabled
[email protected] enabled
keyboard-setup.service enabled
mariadb.service enabled
mysql.service enabled
mysqld.service enabled
networking.service enabled
raspberrypi-net-mods.service enabled
rpi-display-backlight.service enabled
rsync.service enabled
rsyslog.service enabled
service-runner.service enabled
ssh.service enabled
sshswitch.service enabled
supervisor.service enabled
syslog.service enabled
systemd-timesyncd.service enabled
triggerhappy.service enabled
ufw.service enabled
wifi-country.service enabled
avahi-daemon.socket enabled
triggerhappy.socket enabled
nfs-client.target enabled
remote-fs.target enabled
apt-daily-upgrade.timer enabled
apt-daily.timer enabled
phpsessionclean.timer enabled
pi@emonpi-node-13:/ $
Why not … ?
The ‘inelegant’ but pragmatic fix - at the end of rc.local add a few seconds sleep and then sudo systemctl restart apache2
@johnbanks are you still running the emonSD version of the rc.local or have you opted for your own version?
The boot up times and whether a service is enabled to start at boot is pretty much irrelevant when running the emonSD image as it doesn’t matter what services start initially as many of them are restarted in rc.local.
It will not be due to fsck running that causes apache to fail to start since it is expected to fail to start first time, that’s why it is restarted in rc.local and fsck will be totally done and dusted way before apache2 is restarted the second time round.
This is also possibly why the systemd-analyze blame shows no issues, because it may be just reporting the time taken to start the currently running services the last time they were started, expecting that to have been the initial systemd controlled boot up, not the mish mash in the rc.local.
As @borpin says, you should enable the apache2 service as it’s not on that list, but unless you have abandoned the emonSD rc.local file, it’s unlikely to resolve your issue. In that instance, you are closer to fixing it by adding the edit you suggest to the rc.local (as horrendous as that sounds) but perhaps you could just reshuffle the restart order or tweak the existing delays rather than adding another delay and restart.
If you have moved your logfiles out of tmpfs/RAM to disk, then your best approach is to ditch much of what’s in the rc.local, then enabling the apache2 service should fix your issue, and without all that going on in rc.local, you will have a faster boot up and the starting of the services put back under the control of systemd as intended.
No idea, have you run or could you have run sudo systemctl disable apache2 at any point, accidentally instead of sudo systemctl stop apache2 for example?
Who knows why…
Just use the enable command above and it will go in and you shouldn’t need to worry again nor add anything to rc.local.
Which is a really bad bit of legacy work from the days long before systemd.
Personally I’d ditch the EmonSD as a base and use DietPi and Lighttpd…
I’ll get my hat 
2 Likes
Many thx for yr time & inputs on this.
I recognise that I’m a minority user - rootfs & data on a USB HDD and temporarily needing logs to persist to help diagnose my problems.
I now conclude that a personalised rc.local is probably the best way for me with my minority need.
However - this Forum exchange may result in benefirts fot the majority emonSD image user …
- removing fastboot from cmdline.txt
- updating rc.local to the recent jessieminimal changes
Again - many thx for yr time