Forbidden access Raspberry Pi web server

The two times I have done this (used a HDD) have both had separate power supplies for the HDD. I do remember having issues early on that I traced to the HDD not getting enough power. The second one was one of these brilliant PiDrives which come with a cable for separate data and power.

It is a shame the PiDrive is discontinued.

I’m making slow progress …

Using an SDHC with Raspbian Lite, I could fsck the 2 partitions on my USB HDD.
And that fixed the journal errors, etc.

Then back running the SDHC with emonpi - emonpi was up again - I could SSH to it.
But I still could not browse to it - 192.168.0.6 (in my case) was not reachable.

I spent hours looking thru’ logs & googling and then did … sudo service apache2 restart
Voila - all OK - back in business!

My questions are …
Why did the apache2 service stop running in the first place?
But much more importantly - why did it not start on the many reboots that I tried? - including (very importantly) the reboot after fixing journal errors, etc on the USB HDD.

Services are started by /etc/rc.local which has a sleep 3 but commented out. My thinking is that a USB HDD perhaps would be happier with that delay or even perhaps a little more.

Any comments/suggestions - most welcome

Correct.
Add rootdelay=5 to your /boot/cmdline.txt file. (details at the link below)

Ref: How to Boot Up Raspberry Pi 3 from External Hard Disk - Make Tech Easier

1 Like

That bit of log will have gone now but doing a 'systemctl status apache2.service` would have given you the last few entries. Could easily be that the service was being started before the HDD was ready.

Now I’ve had time to check, you are indeed correct. It seems the warning might actually be trying to tell you to use fsck.mode=skip to replace what might be intetended by the fastboot command in cmdline.txt (like for like) as fastboot is not supported in Raspbian.

I had thought it was telling you to use fsck to repair the disk (hence my suggestion) and that the “pass” was perhaps a non-interactive mode, but no that is not the case.

Whilst the fastboot is indeed part of the emonSD, it serves no purpose and if the cmdline.txt contained something like fsck.repair=yes as does the standard Raspbian image, your issue would probably not of occurred as it would have self-healed.

While it may be necessary for some hdd’s to require the bootup to be slowed down, I’ve never needed to use a delay for any of my hdd’s. I would imagine if the hdd needed more time then other things way before just the apache2 service would fail first. besides the apache2 service is started a second time in rc.local so even if it failed once it would start second time around.

It is far more likely that your issue arises from a change in ongoing access speed of using the hdd instead of the sdcard. In the rc.local file there are some delays (as you’ve noticed) that have been altered more than once over the years with new or changes to the images. This alters the speed that the listed services (which includes apache2) are started for the last time, so it is here that your service is failing to start, not in the bootup. Not because of the initial starting on hdd, but because restarting of the services in this manual manner is not tested with your HW setup, so the times probably need tweaking.

Ordinarily systemctl manages all this but the emonSD undermines systemctl, which is rather ironic really considering how apparently important it is to change the whole project to use systemctl and drop perfectly good systemV stuff.

Hopefully all logs since boot are retained whilst running, so if you reboot and the service isn’t running, the error should still be there.

[edit] Actually, I just walked away from the pc and recalled you have moved the log folder back to disk (out of RAM) IIRC. That might just be the reason. Creating all those folders and files, then changing permissions and ownerships in rc.local may take a tad longer when written to disk rather than held in RAM, and that might mean the timings (delays in rc.local) need tweaking.

Paul …
Many thx for that.
I’m a bit mystified tho’ …
In the past few days, I’ve had 2 instances of emonTx/RPi running emonSD Oct 2018 image from a USB HDD that have failed to boot after a crash.
In both cases, when I subsequently connect these USB HDD’s to a RPi running the latest Raspbian Lite and do fsck, errors are found in the ext4 emon rootfs partition and the ext2 emon data partition. I say yes and these errors are corrected.
Then, when I connect these ‘corrected’ USB HDD’s again to emon, everything runs fine.

The cmdline.txt in the emonSD image Oct 2018 contains fsck.repair=yes but later in the line includes fastboot.

We know that fastboot is now outdated.

Is it possible that outdated fastboot stops fsck.repair=yes from running?

If so, should the ‘emon powers that be’ be aware of this?

Thx

All my reading of fastboot indicates it skips fsck, but does it override the fsck.repair=yes option? I’d say your experience would suggest the answer is yes.

I’ve just checked my (old) EmonBase Pi using the EmonSD image from 07Nov16 and it doesn’t contain the fastboot option in cmdline.txt, are you sure fastboot came from the 30Oct18 image and wasn’t something that accidentally got put there during your migration to USB HDD?

Is there anyone else with the new 30Oct18 image that can SSH in and post the output of:
cat /boot/cmdline.txt

@Greebo
Immediately before my latest post and to be sure of my facts, I burned a fresh emonSD image Oct 2018.
Here’s the cmdline.txt

dwc_otg.lpm_enable=0 console=tty1 elevator=noop root=/dev/mmcblk0p2 rootfstype=ext4 fsck.repair=yes rootwait fastboot noswap

The cmdline.txt file has a date/time of 16/08/2018 17:04

My using a USB HDD had no influence - I submit

1 Like

Fair enough!
I only asked the question because moving to boot off USB HDD does usually involve stuffing about with cmdline.txt, and it seems very odd that someone would deliberately add fastboot to a cmdline that already included fsck.repair=yes

when I get a chance, I’ll download the latest Raspbian stretch-Lite image to see what the base image started with…
@glyn.hudson/@TrystanLea any ideas where fastboot came from?

@Greebo
This might save you some time …
From a Raspian Lite downloaded a couple of days ago …

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=e254d8e4-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

No fastboot

Ah thanks, that will save me another download :slight_smile:

I’d already checked a Raspbian stretch-lite from about 6 months ago which also had no fastboot, so we’ll probably just need to wait for Glyn or Trystan to see their mentions above for any answers on how it got there.

Just thinking off the top of my head. If there’s a power cut that takes down the emonBase/pi as well as one or more emonTx, would an fsck take long enough that the emonwhatever would miss the startup of the emonTx and thus handle optical pulse sensors incorrectly? And would a fastboot reverse the order so the sensors were handled correctly?

@djh
I can’t answer your question but only comment …
If you want fastboot functionality then use fsck.mode=skip in cmdline.txt - per per my syslog …

Mar 10 14:57:45 emonpi-node-13 systemd-fsck[131]: Please pass 'fsck.mode=skip' rather than 'fastboot' on the kernel command line

The RPi based emonPi/emonBase will always take longer to cold boot that the emonTx and emonPi add-on board. You could try and minimise the difference by trying to streamline the RPi boot up, the first place to start would be to get rid of the restarting in rc.local mentioned earlier. all these services are being started a second time plus there are the sleep delays too.

If implemented correctly, the pulsecounts should be unaffected bar potentially missing a couple of pulses during the reboot. ie the first low (near zero) count will be assumed to be the reset by emoncms when it comes online and the next count (5-10s later) will be the first count recorded and added to the total recorded in emoncms prior to the outage.

I doubt you would ever get the RPi to boot faster than an emonTx unless you add a long delay to the emonTx, in which case you still miss those initial pulses so nothing is gained.

indeed, we have no build guide for the current image, but the previous build guide and the emoncms/emonhub install docs both have fsck.mode=yes but not fastboot so I’m not sure when that got included.

Indeed, emoncms seems to have a strange mix of sysVinit and systemd commands :frowning: On distros that are exclusively systemd, that along with the widespread inclusion of absolute paths are problematic.

There’s a bigger problem here. fastboot appears to stop fsck from running at boot - ever. The default is fsck.mode=auto which lets the kernel decide if fsck is required or not as the filesystems try and mount. If it finds a corrupt filesystem, it’ll run fsck using the options specified in fsck.repair=, in this case yes, which means just answer yes to all prompts from fsck.

If the fsck doesn’t run at all and the file system needs fsck in order to mount, then you’re going to miss more than just a few pulse counts as @johnbanks found out. His system was essentially down until he was able to work out why it wasn’t booting and force a fsck on the filesystem from a different linux device many hours later.

Either fsck needs to run or it doesn’t… I don’t understand the decision of forcing it not to run with fastboot and then specifying that when it should run, always answer yes to any prompts. I’d suggest instead that the ramifications of fastboot on fsck were not understood when someone added fastboot to the cmdline. It may have made sense with an r/o root partition, but we’ve moved away from that in the latest image.

Specifying fsck.mode=skip instead will likewise achieve a potentially unbootable Pi after a hard power off.

@Greebo

Thx
My bottom line take on this is …
Delete fastboot from cmdline.txt

As proof - I did that yesterday and today was ‘messing around’ and caused a crash.
I pulled then reconnected power.
It took a couple of mins? to come up - an anxious time - but it did.

Selected lines from the log showed that file systems were being checked and repaired on the SDHC (just acting as boot) and on both partitions on my USB HDD (rootfs and data) …

Mar 12 17:31:49 emonpi-node-13 systemd[1]: Starting File System Check on Root Device...
Mar 12 17:31:49 emonpi-node-13 systemd[1]: Started File System Check Daemon to report status.
Mar 12 17:31:49 emonpi-node-13 systemd[1]: Started File System Check on Root Device.
Mar 12 17:31:49 emonpi-node-13 systemd[1]: Starting File System Check on /dev/mmcblk0p1...
Mar 12 17:31:49 emonpi-node-13 systemd[1]: Started File System Check on /dev/mmcblk0p1.
Mar 12 17:31:49 emonpi-node-13 systemd[1]: Starting File System Check on /dev/sda2...
Mar 12 17:31:49 emonpi-node-13 systemd[1]: Started File System Check on /dev/sda2.

It was not a perfect recovery - I could puTTY in but not browse to the web interface …
sudo service apache2 restart fixed that.

My pragmatic/inelegant further solution will be to add at the end of /etc/rc.local …
A few secs of sleep and a call to restart apache2 for yet another time.

Was there any indication why apache didn’t start cleanly?

How long did the fsck's take to complete?

@Greebo

fsck’s took a min or 2 but maybe less -seemed an age because I was anxious.

My very latest winSCP log download is quite large - eg: daemon.log has 1,062,341 lines!

Doing a search for apache2 gave the following results from all files in the log download …

[Tue Mar 12 13:17:23.767494 2019] [core:notice] [pid 8293] AH00094: Command line: '/usr/sbin/apache2'
[Tue Mar 12 23:06:10.781812 2019] [core:notice] [pid 3044] AH00094: Command line: '/usr/sbin/apache2'
[Mon Mar 11 06:25:02.794952 2019] [core:notice] [pid 890] AH00094: Command line: '/usr/sbin/apache2'
Mar 10 15:01:39 emonpi-node-13 sudo:       pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/usr/sbin/service apache2 restart
Mar 12 23:06:09 emonpi-node-13 sudo:       pi : TTY=pts/1 ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/service apache2 restart
Mar 10 14:50:55 emonpi-node-13 apachectl[1158]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Mar 10 15:01:40 emonpi-node-13 apachectl[879]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Mar 11 06:25:02 emonpi-node-13 apachectl[24065]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Mar 12 13:17:23 emonpi-node-13 apachectl[8276]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Mar 12 23:06:10 emonpi-node-13 apachectl[3033]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Mar 12 13:17:23 emonpi-node-13 apachectl[8276]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Mar 12 23:06:10 emonpi-node-13 apachectl[3033]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message

I’m happy to do a seach on any other phrases that you may suggest.

You could try systemd-analyze blame to see how long fsck took.

1 Like