Hi all,
I have replaced the SD card on my emonbase with a USB flash drive. Everything works great except that every 5 or 10 minutes the systems freezes for up to 2mn.
I have tried all the troubleshooting tools I could get my hands on and the only one that reports something suspicious is atop, which shows the flash disk being over 100% “Busy”. “iotop -o -P” running at the same time show Actual disk read/writes at 0 or a few bytes/s:
pi@emonpi:~ $ iostat -d -x /dev/sda
Linux 4.14.71-v7+ (emonpi) 14/07/19 _armv7l_ (4 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.06 0.18 0.17 0.59 3.26 1.40 12.20 5.22 6834.12 307.19 8697.44 329.19 25.15
When the system is unresponsive the queue depth on /dev/sda is around 40, and the latency (avio) goes up to 10s. According to iotop the two processes doing most of the writing are feedwriter.php and jdb2 (journaling).
The system is
Linux 4.14.71-v7+ with emoncms low-write 10.1.1
Here is the output of fdisk on the flash drive:
Disk /dev/sda: 28.9 GiB, 31029460992 bytes, 60604416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x50c2568e
Device Boot Start End Sectors Size Id Type
/dev/sda1 8192 96663 88472 43.2M c W95 FAT32 (LBA)
/dev/sda2 98304 8290303 8192000 3.9G 83 Linux
/dev/sda3 8290304 60604415 52314112 25G 83 Linux
I am running out of ideas on how to troubleshoot this, any suggestion welcome, thanks!
Franck
Hello Frank, could there be a hardware issue with the flash drive, that its for some reason unable to write to a sector or something like that?
Alternatively if the disk is fine, it might be worth trying an ext2 formatted partition EmonScripts/readme.md at master · openenergymonitor/EmonScripts · GitHub but I’ve had no problem running on ext4 in the past either…
The filesystem is already ext2 - I used the emonSD-30Oct18 image and it must be what it had:
Filesystem Type Size Used Avail Use% Mounted on
/dev/root ext4 3.9G 1.7G 2.0G 46% /
/dev/sda1 vfat 43M 22M 21M 52% /boot
/dev/sda3 ext2 25G 243M 24G 2% /home/pi/data
Running e2fsck on the mounted /dev/sda3 partition reported errors, so I unmounted, fixed the errors, and rebooted.
As soon as I plug-in the devices that feed in inputs to emoncms, the problem appears:
DSK | sda | busy 102% | read 0 | write 1 | KiB/r 0 | KiB/w 1 | MBr/s 0.0 | MBw/s 0.0 | avio 10000 ms
feedwriter.php is the only process writing to the disk, but it wrote about 700k over 5mn, so I suspect there is a problem with the driver or with the drive.
The problem is, I would rather not go through a complete re-install on another flash drive without understand what is happening first…
Franck
Here is the iostat output for the disk: the number of transactions seems ok, but the wait times are definitely not:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda
2.30 0.21 4.60 0.48 102.10 1.38 40.67 13.61 2624.42 159.55 26095.77 141.79 72.15
sda2
1.85 0.10 4.20 0.09 97.15 0.84 45.69 1.94 446.39 139.79 14762.51 152.67 65.48
sda3
0.13 0.11 0.37 0.39 4.24 0.54 12.60 11.66 15058.36 402.72 28706.41 788.52 59.87
sda1
0.32 0.00 0.03 0.00 0.45 0.00 28.77 0.01 185.94 5.24 11570.00 184.06 0.58
Franck
I am going back to the SD card for now, I can’t get the two USB sticks I have (Datatraveler G3 and DataTraveler G4) to work reliably.
If I plug-in a keyboard and a DVI monitor to the Pi:
- the SD card boots fine and the boot sequence appears on the screen
- with either of the USB drives nothing shows up on the monitor and the unit won’t boot
- if I unplug keyboard and monitor the Pi boots from my original USB stick; it won’t boot from the copy I created using dd
I suspect something is going on with power consumption, so until I get a powered hub I’ll revert to the SD card.
Franck
Which model Pi are you trying this on?
Starting with the B+ and the Pi2 you can double the amount of current available to a
USB port by setting max_usb_current=1 in /boot/config.txt
Keep in mind you’ll need a PSU capable of supplying said current…
Thanks Bill,
That is good to know, although a USB drive should consume at most around 100mA max, well under the 600mA the USB port is supposed to support.
I have a 2.5A power supply that looks good on paper, but my next round of experiments will probably involve a scope to try and detect any voltage drop cause by the drive.
The fact that the DVI output behaves differently with and w/o the drive makes me think of a power problem, but I can’t tell if it’s external (the power supply) or internal (design or hardware issue in the Pi).
For now the Pi is back in the garage with its good old SD card, I need to some time to gather the energy to mess things up again 
Franck
I use a device similar to this for just that purpose:
Got it from a flea-Bay vendor for ~5USD.
