No disk space although enough room

2022-12-03 12:11:58.409|ERROR|emoncms_mqtt.php|ErrorException: mysqli::query(): (HY000/3): Error writing file ‘/tmp/MYos5EUQ’ (Errcode: 28 “No space left on device”) in /var/www/emoncms/Modules/input/input_model.php:734
Stack trace:

Got this message from the log file. Restarted the system and mosquito but didn’t help.

What can I do?

What version of EmonSD are you running? What is the hardware?

There was an issue a while back with the log files not rotating correctly and filling up the space.

what does df -h tell you?

hi Brian,

I have [low-write 11.0.9] running on an EmonPi.

When I turn on the autorefresh of the emoncms log, I get the message every 1,5sec

Where can I enter the command you mention (d f -h)? I’'m not familiar with this.

thhnx,
Vincent

The df command is entered at the command line. It shows how much disk is free (df, disk free).

You could get similar details by going to the Admin: System Info in the emoncms left menu and looking under the Disk heading. Paste it here for helpers to read, please.

This is a screenshot of the disk space, to me looks that should not be the problem.

Before I turned to the forum I tried a complete reboot of the system but the message did not disappear.

In the admin page, click on ‘Copy as Markdown’ and paste here (no further formatting required)

image

no because…

The error is for tmp which is odd.

How old is the EmonPi?

Have you installed anything other than EmonCMS on the EmonPi?

Sorry. You need to SSH into the EmonPi to issue that command on the command line.

Server Information

Server Information

Services

  • emonhub :- Active Running

  • emoncms_mqtt :- Active Running

  • feedwriter :- Active Running - sleep 300s 0 feed points pending write

  • service-runner :- Active Running

  • emonPiLCD :- Active Running

  • redis-server :- Active Running

  • mosquitto :- Active Running

  • demandshaper :- Active Running

Emoncms

Server

  • CPU :- 1 Threads(s) | 4 Core(s) | 1 Sockets(s) | Cortex-A53 | 76.80MIPS |
  • OS :- Linux 5.10.17-v7+
  • Host :- emonpi | emonpi | (192.168.2.47)
  • Date :- 2022-12-04 15:38:15 UTC
  • Uptime :- 15:38:15 up 1 day, 18:50, 0 users, load average: 1.01, 1.02, 1.00

Memory

  • RAM :- Used: 20.47%
    • Total :- 924.21 MB
    • Used :- 189.18 MB
    • Free :- 735.04 MB
  • Swap :- Used: 0.00%
    • Total :- 100 MB
    • Used :- 0 B
    • Free :- 100 MB

Disk

  • **** :- - / :- Used: 52.42%
    • Total :- 4.07 GB
    • Used :- 2.13 GB
    • Free :- 1.74 GB
    • Read Load :- 35.94 B/s
    • Write Load :- 985.47 B/s
    • Load Time :- 1 days 3 hours 26 mins
  • /var/opt/emoncms :- Used: 0.86%
    • Total :- 9.84 GB
    • Used :- 86.88 MB
    • Free :- 9.26 GB
    • Read Load :- 9.17 B/s
    • Write Load :- 0 B/s
    • Load Time :- 1 days 3 hours 26 mins
  • /boot :- Used: 18.90%
    • Total :- 252.05 MB
    • Used :- 47.65 MB
    • Free :- 204.4 MB
    • Read Load :- 0 B/s
    • Write Load :- 0 B/s
    • Load Time :- 1 days 3 hours 26 mins
  • /var/log :- Used: 18.34%
    • Total :- 50 MB
    • Used :- 9.17 MB
    • Free :- 40.83 MB
    • Read Load :- n/a
    • Write Load :- n/a
    • Load Time :- n/a

HTTP

  • Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.3.29-MariaDB-0+deb10u1
  • Host :- 127.0.0.1 (127.0.0.1)
  • Date :- 2022-12-04 15:38:15 (UTC 00:00‌​)
  • Stats :- Uptime: 154609 Threads: 11 Questions: 337462 Slow queries: 0 Opens: 48 Flush tables: 1 Open tables: 41 Queries per second avg: 2.182

Redis

  • Version :-
    • Redis Server :- 5.0.3
    • PHP Redis :- 5.3.4
  • Host :- localhost:6379
  • Size :- 86 keys (757.59K)
  • Uptime :- 1 days

MQTT Server

  • Version :- Mosquitto 1.5.7
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.3.29-1~deb10u1 (Zend Version 3.3.29)
  • Run user :- User: www-data Group: www-data video Script Owner: pi
  • Modules :- apache2handlercalendar Core ctype curl date dom v20031129exif fileinfo filter ftp gd gettext hash iconv json v1.7.0libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $openssl pcre PDO pdo_mysql Phar posix readline redis v5.3.4Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

  • Model :- Raspberry Pi 3 Model B Rev 1.2 - 1GB (Sony UK)

  • Serial num. :- 4D2CE44B

  • CPU Temperature :- 58.00°C

  • GPU Temperature :- 58.0°C

  • emonpiRelease :- emonSD-21Jul21

  • File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Linux; Android 7.1.1; Lenovo TB-X704F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
  • Language :- nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7,de;q=0.6,fr;q=0.5

Window

  • Size :- 1280 x 632

Screen

  • Resolution :- 1280 x 800

To fix this, you will need to SSH into the shell.

I think something has happened such that a process has left some files in the directory that need to be manually removed.

A bit of digging and I find that the /tmp folder should be cleaned regularly by a service called systemd-tmpfiles.

You can check when it was last run with

pi@emonpi:~ $ systemctl list-timers

Mine looks like this.

NEXT                         LEFT       LAST                         PASSED    UNIT                         ACTIVATES
Sun 2022-12-04 17:07:01 GMT  10min left Sun 2022-12-04 00:03:37 GMT  16h ago   apt-daily.timer              apt-daily.service
Sun 2022-12-04 17:09:00 GMT  12min left Sun 2022-12-04 16:39:37 GMT  16min ago phpsessionclean.timer        phpsessionclean.service
Mon 2022-12-05 00:00:00 GMT  7h left    Sun 2022-12-04 00:00:37 GMT  16h ago   logrotate.timer              logrotate.service
Mon 2022-12-05 00:00:00 GMT  7h left    Sun 2022-12-04 00:00:37 GMT  16h ago   man-db.timer                 man-db.service
Mon 2022-12-05 06:08:15 GMT  13h left   Sun 2022-12-04 06:09:37 GMT  10h ago   apt-daily-upgrade.timer      apt-daily-upgrade.service
Mon 2022-12-05 10:38:37 GMT  17h left   Sun 2022-12-04 10:38:37 GMT  6h ago    systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service

Checking on my system and I see there were 2 files sitting there that are not being cleaned up - so it does happen. I’ve now removed them.

You need to SSH in and list the files in the /tmp folder and we can see what needs to be deleted.

My /tmp folder currently looks like this.

pi@emonpi:~ $ ls -la /tmp
total 4
drwxrwxrwt 10 root root  200 Dec  4 16:55 .
drwxr-xr-x 21 root root 4096 Oct 17  2019 ..
drwxrwxrwt  2 root root   40 Sep 18 09:17 .font-unix
drwxrwxrwt  2 root root   40 Sep 18 09:17 .ICE-unix
drwx------  3 root root   60 Nov 25 10:21 systemd-private-36f58b8800834f268c77452ef788441e-apache2.service-Do59qG
drwx------  3 root root   60 Sep 18 09:47 systemd-private-36f58b8800834f268c77452ef788441e-redis-server.service-IF8TxJ
drwx------  3 root root   60 Sep 18 09:17 systemd-private-36f58b8800834f268c77452ef788441e-systemd-timesyncd.service-ZIrgqt
drwxrwxrwt  2 root root   40 Sep 18 09:17 .Test-unix
drwxrwxrwt  2 root root   40 Sep 18 09:17 .X11-unix
drwxrwxrwt  2 root root   40 Sep 18 09:17 .XIM-unix

[edit]
Interestingly, it looks like the standard configuration expressly tells the clean process not to clean the tmp folder

pi@emonpi:~ $ cat /usr/lib/tmpfiles.d/tmp.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

# Clear tmp directories separately, to make them easier to override
D /tmp 1777 root root -

As far as I can deduce from the man file, the trailing - means no automatic clean-up is done. :man_facepalming:

Is tmp cleared on reboot?

The controlling of clearing is done by this service. Some file (those still in my tmp folder are specifically only removed/updated at boot. So if not cleared on the timer, then no they won’t be cleared on boot (as the OP has discovered).

I checked in the ssh and find the same timers and files as you do. All timers have benn passed in less than 24h

What else is in tmp folder?

Only the files you showed

Please select the text in the terminal window. Once selected, it is automatically put into the paste buffer (clipboard).

You can then paste it as ‘Preformatted’ text (between 3 backticks) or use the button in the ribbon.

Can you provide the df -h output, please?

pi@emonpi:~ $ ls -la /tmp
total 4
drwxrwxrwt 10 root root  200 Dec  4 21:42 .
drwxr-xr-x 18 root root 4096 Jul 21  2021 ..
drwxrwxrwt  2 root root   40 Dec  2 20:41 .font-unix
drwxrwxrwt  2 root root   40 Dec  2 20:41 .ICE-unix
drwx------  3 root root   60 Dec  2 20:41 systemd-private-0439c32c8cbb4c8fbb667eee0e61aafc-apache2.service-CvGQfu
drwx------  3 root root   60 Dec  2 20:41 systemd-private-0439c32c8cbb4c8fbb667eee0e61aafc-redis-server.service-8JI2es
drwx------  3 root root   60 Dec  2 20:41 systemd-private-0439c32c8cbb4c8fbb667eee0e61aafc-systemd-timesyncd.service-AIpiVR
drwxrwxrwt  2 root root   40 Dec  2 20:41 .Test-unix
drwxrwxrwt  2 root root   40 Dec  2 20:41 .X11-unix
drwxrwxrwt  2 root root   40 Dec  2 20:41 .XIM-unix
pi@emonpi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.1G  2.2G  1.8G  56% /
devtmpfs        430M     0  430M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M   12M  451M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
tmpfs           1.0M  4.0K 1020K   1% /var/lib/php/sessions
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            30M   26M  4.2M  87% /tmp
/dev/mmcblk0p3  9.9G   87M  9.3G   1% /var/opt/emoncms
/dev/mmcblk0p1  253M   48M  205M  19% /boot
log2ram          50M   11M   40M  22% /var/log
tmpfs            93M     0   93M   0% /run/user/1000

I can’t change the format …

In the ribbon, click on this button (when creating the message)
image

and paste the text between the 2 sets of backticks (leave them on a line on their own).

I’ve edited the format.

Back to the problem. I’m stumped. tmp has plenty of room, so why you are getting that error is unknown.

All I can suggest is get a new SDCard (you can buy one ready to go) and migrate your data etc.

It could be the card is failing, but that is unusual. @Gwil

PS

I have learnt something so not a complete waste of time :slight_smile:

in my analysis I used 87% of /tmp allready. Which is limited to 30M …

as I’m a starter, there is not valuable data on it. As I’ve some free SD-cards hanging around (of a SLR camera), which is the easiest way to migrate to a new SD-Card

is there a topic on migrating?
will 64G suite my system?

Yes, of course, I was just looking at the files listed.

This is my output and as you can see, apparently the same files but 0% use.

pi@emonpi:/tmp $ ls -la
total 4
drwxrwxrwt 10 root root  220 Dec  4 21:47 .
drwxr-xr-x 21 root root 4096 Oct 17  2019 ..
drwxrwxrwt  2 root root   40 Sep 18 09:17 .font-unix
drwxrwxrwt  2 root root   40 Sep 18 09:17 .ICE-unix
drwx------  3 root root   60 Nov 25 10:21 systemd-private-36f58b8800834f268c77452ef788441e-apache2.service-Do59qG
drwx------  3 root root   60 Sep 18 09:47 systemd-private-36f58b8800834f268c77452ef788441e-redis-server.service-IF8TxJ
drwx------  3 root root   60 Sep 18 09:17 systemd-private-36f58b8800834f268c77452ef788441e-systemd-timesyncd.service-ZIrgqt
-rw-r--r--  1 root root    0 Dec  4 17:19 test-root.txt
drwxrwxrwt  2 root root   40 Sep 18 09:17 .Test-unix
drwxrwxrwt  2 root root   40 Sep 18 09:17 .X11-unix
drwxrwxrwt  2 root root   40 Sep 18 09:17 .XIM-unix
pi@emonpi:/tmp $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  2.6G  1.2G  69% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           488M     0  488M   0% /dev/shm
tmpfs           488M   56M  433M  12% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           488M     0  488M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/lib/php/sessions
tmpfs            30M     0   30M   0% /tmp
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p1  253M   52M  202M  21% /boot
/dev/mmcblk0p3   10G  798M  8.7G   9% /var/opt/emoncms
log2ram          50M  5.2M   45M  11% /var/log
tmpfs            98M     0   98M   0% /run/user/1000

You could try sudo ls -la otherwise I do not understand it.

Use this guide to updating Update & Upgrade - Guide | OpenEnergyMonitor

Note, @TrystanLea is about to release a new EmonSD image imminently.