System freezes with flash drive

Hi all,

I have replaced the SD card on my emonbase with a USB flash drive. Everything works great except that every 5 or 10 minutes the systems freezes for up to 2mn.
I have tried all the troubleshooting tools I could get my hands on and the only one that reports something suspicious is atop, which shows the flash disk being over 100% “Busy”. “iotop -o -P” running at the same time show Actual disk read/writes at 0 or a few bytes/s:


[email protected]:~ $ iostat -d -x /dev/sda
Linux 4.14.71-v7+ (emonpi)      14/07/19        _armv7l_        (4 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.06     0.18    0.17    0.59     3.26     1.40    12.20     5.22 6834.12  307.19 8697.44 329.19  25.15

When the system is unresponsive the queue depth on /dev/sda is around 40, and the latency (avio) goes up to 10s. According to iotop the two processes doing most of the writing are feedwriter.php and jdb2 (journaling).

The system is
Linux 4.14.71-v7+ with emoncms low-write 10.1.1

Here is the output of fdisk on the flash drive:

Disk /dev/sda: 28.9 GiB, 31029460992 bytes, 60604416 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x50c2568e

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sda1          8192    96663    88472 43.2M  c W95 FAT32 (LBA)
/dev/sda2         98304  8290303  8192000  3.9G 83 Linux
/dev/sda3       8290304 60604415 52314112   25G 83 Linux

I am running out of ideas on how to troubleshoot this, any suggestion welcome, thanks!

Franck

Hello Frank, could there be a hardware issue with the flash drive, that its for some reason unable to write to a sector or something like that?

Alternatively if the disk is fine, it might be worth trying an ext2 formatted partition EmonScripts/readme.md at master · openenergymonitor/EmonScripts · GitHub but I’ve had no problem running on ext4 in the past either…

The filesystem is already ext2 - I used the emonSD-30Oct18 image and it must be what it had:

Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4      3.9G  1.7G  2.0G  46% /
/dev/sda1      vfat       43M   22M   21M  52% /boot
/dev/sda3      ext2       25G  243M   24G   2% /home/pi/data

Running e2fsck on the mounted /dev/sda3 partition reported errors, so I unmounted, fixed the errors, and rebooted.

As soon as I plug-in the devices that feed in inputs to emoncms, the problem appears:

DSK | sda | busy 102% | read 0 | write 1 | KiB/r 0 | KiB/w 1 | MBr/s 0.0 | MBw/s 0.0 | avio 10000 ms

feedwriter.php is the only process writing to the disk, but it wrote about 700k over 5mn, so I suspect there is a problem with the driver or with the drive.
The problem is, I would rather not go through a complete re-install on another flash drive without understand what is happening first…

Franck

Here is the iostat output for the disk: the number of transactions seems ok, but the wait times are definitely not:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda
                  2.30     0.21    4.60    0.48   102.10     1.38    40.67    13.61 2624.42  159.55 26095.77 141.79  72.15
sda2
                  1.85     0.10    4.20    0.09    97.15     0.84    45.69     1.94  446.39  139.79 14762.51 152.67  65.48
sda3
                  0.13     0.11    0.37    0.39     4.24     0.54    12.60    11.66 15058.36  402.72 28706.41 788.52  59.87
sda1
                  0.32     0.00    0.03    0.00     0.45     0.00    28.77     0.01  185.94    5.24 11570.00 184.06   0.58

Franck

I am going back to the SD card for now, I can’t get the two USB sticks I have (Datatraveler G3 and DataTraveler G4) to work reliably.
If I plug-in a keyboard and a DVI monitor to the Pi:

  • the SD card boots fine and the boot sequence appears on the screen
  • with either of the USB drives nothing shows up on the monitor and the unit won’t boot
  • if I unplug keyboard and monitor the Pi boots from my original USB stick; it won’t boot from the copy I created using dd

I suspect something is going on with power consumption, so until I get a powered hub I’ll revert to the SD card.

Franck

Which model Pi are you trying this on?
Starting with the B+ and the Pi2 you can double the amount of current available to a
USB port by setting max_usb_current=1 in /boot/config.txt

Keep in mind you’ll need a PSU capable of supplying said current…

Thanks Bill,

That is good to know, although a USB drive should consume at most around 100mA max, well under the 600mA the USB port is supposed to support.

I have a 2.5A power supply that looks good on paper, but my next round of experiments will probably involve a scope to try and detect any voltage drop cause by the drive.
The fact that the DVI output behaves differently with and w/o the drive makes me think of a power problem, but I can’t tell if it’s external (the power supply) or internal (design or hardware issue in the Pi).

For now the Pi is back in the garage with its good old SD card, I need to some time to gather the energy to mess things up again :slight_smile:
Franck

I use a device similar to this for just that purpose:
image
Got it from a flea-Bay vendor for ~5USD.