Forbidden access Raspberry Pi web server

(Greebo) #41

Hmm… none of those timestamps align particularly with the fsck ones in your previous post. Maybe check the syslog around the time of those fsck entries - you may have to scan through it using less (or more if you’re so inclined) to see if there’s anything obvious related to apache failing to start. There may have been a dependency that failed too, which wouldn’t show up as “apache2” in your search.

(John Banks) #42

I tried yr suggestion …

[email protected]:~ $ systemd-analyze blame
     3min 1.530s [email protected]
          9.393s dhcpcd.service
          5.529s mariadb.service
          3.664s dev-sda1.device
          2.887s apt-daily.service
          2.180s dphys-swapfile.service
          1.824s rc-local.service
          1.730s networking.service
          1.463s redis-server.service
          1.417s lwrfd.service
          1.411s raspi-config.service
          1.365s rsyslog.service
          1.265s emonPiLCD.service
          1.100s avahi-daemon.service
          1.058s dnsmasq.service
           928ms plymouth-start.service
           798ms keyboard-setup.service
           797ms ssh.service
           668ms phpsessionclean.service
           636ms alsa-restore.service
           629ms systemd-logind.service
           595ms mosquitto.service
           533ms [email protected]
           438ms apache2.service
           428ms systemd-timesyncd.service
           412ms rng-tools.service
           386ms systemd-fsck-root.service
           377ms home-pi-data.mount
           375ms fake-hwclock.service
           368ms systemd-tmpfiles-setup-dev.service
           359ms triggerhappy.service
           337ms systemd-udevd.service
           334ms systemd-udev-trigger.service
           334ms systemd-tmpfiles-setup.service
           331ms wifi-country.service
           305ms dev-mqueue.mount
           292ms run-rpc_pipefs.mount
           288ms systemd-journald.service
           228ms systemd-modules-load.service
           201ms systemd-rfkill.service
           186ms ufw.service
           183ms console-setup.service
           179ms nfs-config.service
           173ms systemd-tmpfiles-clean.service
           150ms sys-kernel-debug.mount
           131ms systemd-sysctl.service
           103ms systemd-journal-flush.service
           102ms systemd-update-utmp.service
            93ms plymouth-read-write.service
            80ms [email protected]
            77ms systemd-remount-fs.service
            70ms systemd-user-sessions.service
            68ms var-tmp.mount
            65ms systemd-random-seed.service
            58ms kmod-static-nodes.service
            50ms tmp.mount
            37ms plymouth-quit.service
            29ms systemd-update-utmp-runlevel.service
            26ms sys-kernel-config.mount
            25ms plymouth-quit-wait.service
            24ms boot.mount
lines 39-61/61 (END)

But don’t know how to interpret the resuts.

Anything obvious to you?

(Greebo) #43

I interpret that as 3 minutes and 1.530 seconds to fsck /dev/sda2 and 533 milliseconds to do /dev/mmcblk0p1

(Brian Orpin) #44

That is a cracking command - not found that before.

It looks from the blame output that apache did start (438ms).

Note that strictly speaking you should use systemctl not service to restart a unit now.

If it happens again, do a systemctl status apache2 (before you restart) and you should see the last few lines of the log which may tell you why it failed.

I had an issue with running the http interface to a package called monit. Although the unit itself started, the http interface always failed. Tried loads of different wants and requires but in the end, after some digging I worked out how to put a timer unit in so it delayed the startup. Note that you no longer enable the service file (as in sudo systemctl enable unit.service) but enable the timer (as in sudo systemctl enable unit.timer).

(Bill Thomson) #45

Although the referenced error isn’t a show stopper, adding

ServerName localhost

to your apache.conf file will make that error go away.

(Dave Howorth) #46

Actually, that was the command I remembered but systemd-analyze critical-chain might be more useful. The man page is at

(John Banks) #47


It did happen again - entirely my doing.
After I rebooted, I could puTTy in but not browse to the web interface.
And remembering yr suggestion, I did …

[email protected]:~ $ systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
lines 1-3/3 (END)

This gave me no clues …

[email protected]:/ $ cat /lib/systemd/system/apache2.service
Description=The Apache HTTP Server

ExecStart=/usr/sbin/apachectl start
ExecStop=/usr/sbin/apachectl stop
ExecReload=/usr/sbin/apachectl graceful


I then did …

[email protected]:~ $ systemd-analyze blame
     3min 1.665s [email protected]
          9.410s dhcpcd.service
          5.157s mariadb.service
          3.774s dev-sda1.device
          2.733s dphys-swapfile.service
          1.656s rc-local.service
          1.600s redis-server.service
          1.406s raspi-config.service
          1.275s lwrfd.service
          1.090s plymouth-start.service
          1.018s triggerhappy.service
           959ms keyboard-setup.service
           947ms rsyslog.service
           902ms dnsmasq.service
           845ms networking.service
           721ms avahi-daemon.service
           671ms ssh.service
           600ms phpsessionclean.service
           556ms systemd-logind.service
           556ms mosquitto.service
           519ms emonPiLCD.service
           475ms systemd-timesyncd.service
           458ms alsa-restore.service
           457ms systemd-tmpfiles-setup.service
           456ms fake-hwclock.service
           443ms [email protected]
           369ms rng-tools.service
           360ms systemd-udevd.service
           353ms systemd-fsck-root.service
           343ms systemd-tmpfiles-setup-dev.service
           324ms wifi-country.service
           314ms sys-kernel-debug.mount
           313ms systemd-udev-trigger.service
           310ms run-rpc_pipefs.mount
           308ms systemd-journald.service
           306ms dev-mqueue.mount
           300ms home-pi-data.mount
           263ms kmod-static-nodes.service
           252ms systemd-modules-load.service
           231ms console-setup.service
           216ms nfs-config.service
           189ms systemd-rfkill.service
           162ms systemd-tmpfiles-clean.service
           161ms systemd-sysctl.service
           154ms ufw.service
           117ms systemd-update-utmp.service
            81ms systemd-remount-fs.service
            80ms [email protected]
            77ms systemd-random-seed.service
            75ms systemd-user-sessions.service
            75ms systemd-journal-flush.service
            75ms plymouth-read-write.service
            48ms tmp.mount
            34ms var-tmp.mount
            33ms plymouth-quit.service
            31ms plymouth-quit-wait.service
            30ms systemd-update-utmp-runlevel.service
            24ms boot.mount
            15ms sys-kernel-config.mount
lines 37-59/59 (END)

Expectedly, the fsck on sda2 (the ext2 /home/pi/data partition on USB HDD) takes a long time.
As an aside perhaps that would better be ext4 - it’s an HDD not a ‘sensitive’ SDHC ??? And ext4 fsck’s seem much faster.

But I don’t see apache2 in the list.

So I checked the logs around the relevant time … apart from a note that apache2 was starting, I could see no other relevant references.

The ‘inelegant’ but pragmatic fix seems to be - at the end of rc.local add a few seconds sleep and then sudo systemctl restart apache2
I’ll give that a try.

(Brian Orpin) #48

All I can think is that it is not enabled to start at boot.

systemctl list-unit-files | grep enabled

lists all enabled units.

If it is not there

sudo systemctl enable apache2.service

You need then to start it, if it is not started.

(John Banks) #49


It’s not there …

[email protected]:/ $ systemctl list-unit-files | grep enabled
[email protected]                        enabled
avahi-daemon.service                   enabled
bluetooth.service                      enabled
console-setup.service                  enabled
cron.service                           enabled
dbus-org.bluez.service                 enabled
dbus-org.freedesktop.Avahi.service     enabled
dhcpcd.service                         enabled
dhcpcd5.service                        enabled
dnsmasq.service                        enabled
emoncms_mqtt.service                   enabled
emonhub.service                        enabled
fake-hwclock.service                   enabled
feedwriter.service                     enabled
[email protected]                         enabled
keyboard-setup.service                 enabled
mariadb.service                        enabled
mysql.service                          enabled
mysqld.service                         enabled
networking.service                     enabled
raspberrypi-net-mods.service           enabled
rpi-display-backlight.service          enabled
rsync.service                          enabled
rsyslog.service                        enabled
service-runner.service                 enabled
ssh.service                            enabled
sshswitch.service                      enabled
supervisor.service                     enabled
syslog.service                         enabled
systemd-timesyncd.service              enabled
triggerhappy.service                   enabled
ufw.service                            enabled
wifi-country.service                   enabled
avahi-daemon.socket                    enabled
triggerhappy.socket                    enabled                      enabled                       enabled
apt-daily-upgrade.timer                enabled
apt-daily.timer                        enabled
phpsessionclean.timer                  enabled
[email protected]:/ $

Why not … ?
The ‘inelegant’ but pragmatic fix - at the end of rc.local add a few seconds sleep and then sudo systemctl restart apache2

(Paul) #50

@johnbanks are you still running the emonSD version of the rc.local or have you opted for your own version?

The boot up times and whether a service is enabled to start at boot is pretty much irrelevant when running the emonSD image as it doesn’t matter what services start initially as many of them are restarted in rc.local.

It will not be due to fsck running that causes apache to fail to start since it is expected to fail to start first time, that’s why it is restarted in rc.local and fsck will be totally done and dusted way before apache2 is restarted the second time round.

This is also possibly why the systemd-analyze blame shows no issues, because it may be just reporting the time taken to start the currently running services the last time they were started, expecting that to have been the initial systemd controlled boot up, not the mish mash in the rc.local.

As @borpin says, you should enable the apache2 service as it’s not on that list, but unless you have abandoned the emonSD rc.local file, it’s unlikely to resolve your issue. In that instance, you are closer to fixing it by adding the edit you suggest to the rc.local (as horrendous as that sounds) but perhaps you could just reshuffle the restart order or tweak the existing delays rather than adding another delay and restart.

If you have moved your logfiles out of tmpfs/RAM to disk, then your best approach is to ditch much of what’s in the rc.local, then enabling the apache2 service should fix your issue, and without all that going on in rc.local, you will have a faster boot up and the starting of the services put back under the control of systemd as intended.

No idea, have you run or could you have run sudo systemctl disable apache2 at any point, accidentally instead of sudo systemctl stop apache2 for example?

(Brian Orpin) #51

Who knows why…

Just use the enable command above and it will go in and you shouldn’t need to worry again nor add anything to rc.local.

(Brian Orpin) #52

Which is a really bad bit of legacy work from the days long before systemd.

Personally I’d ditch the EmonSD as a base and use DietPi and Lighttpd…

I’ll get my hat :smile:

(John Banks) #53

@borpin and @pb66

Many thx for yr time & inputs on this.

I recognise that I’m a minority user - rootfs & data on a USB HDD and temporarily needing logs to persist to help diagnose my problems.

I now conclude that a personalised rc.local is probably the best way for me with my minority need.

However - this Forum exchange may result in benefirts fot the majority emonSD image user …

  • removing fastboot from cmdline.txt
  • updating rc.local to the recent jessieminimal changes

Again - many thx for yr time