Import from Archive fails

Hi

I am unable to restore my data from a previous archived backup… The file appears to be loading ok…

I would REAALYY hate to lose my data and configuration… Help !!!

Any pointers?

Thanks
P

The log:

=== Emoncms import start ===
2024-11-14-07:43:44
Backup module version:
    "version"      : "2.3.3",
EUID: 1000
Reading /opt/emoncms/modules/backup/config.cfg....
Location of data databases: /var/opt/emoncms
Location of emonhub.conf: /etc/emonhub
Location of Emoncms: /var/www/emoncms
Backup destination: /var/opt/emoncms/backup
Backup source path: /var/opt/emoncms/backup/uploads
Starting import from /var/opt/emoncms/backup/uploads to /var/opt/emoncms/backup...
Image version: emonSD-20Nov23
new image
Backup found: /var/opt/emoncms/backup/uploads/emoncms-backup-emonserver-2024-11-07(1).tar.gz starting import..
Read MYSQL authentication details from settings.php
Decompressing backup..
emoncms.sql
settings.ini
phpfina/
phpfina/227.dat
phpfina/294.meta
phpfina/201.meta
phpfina/195.dat
phpfina/69.meta
phpfina/68.meta
phpfina/259.meta
phpfina/6.meta
phpfina/264.meta
phpfina/1.dat
phpfina/323.dat
phpfina/227.meta
phpfina/318.meta
phpfina/5.dat
phpfina/315.meta
phpfina/260.dat
phpfina/2.dat
phpfina/197.dat
phpfina/225.dat
phpfina/226.meta
phpfina/307.dat
phpfina/252.dat
phpfina/201.dat
phpfina/289.dat
phpfina/290.dat
phpfina/199.meta
phpfina/315.dat
phpfina/325.dat
phpfina/196.dat
phpfina/251.dat
phpfina/237.dat
phpfina/292.meta
phpfina/6.dat
phpfina/290.meta
phpfina/250.meta
phpfina/68.dat
phpfina/294.dat
phpfina/9.meta
phpfina/7.meta
phpfina/200.dat
phpfina/264.dat
phpfina/200.meta
phpfina/195.meta
phpfina/199.dat
phpfina/268.meta
phpfina/3.meta
phpfina/248.dat
phpfina/295.dat
phpfina/295.meta
phpfina/261.dat
phpfina/2.meta
phpfina/225.meta
phpfina/270.dat
phpfina/255.meta
phpfina/196.meta
phpfina/261.meta
phpfina/272.dat
phpfina/224.dat
phpfina/8.dat
phpfina/319.dat
phpfina/4.meta
phpfina/268.dat
phpfina/250.dat
phpfina/324.dat
phpfina/320.dat
phpfina/262.meta
phpfina/224.meta
phpfina/323.meta
phpfina/260.meta
phpfina/249.meta
phpfina/266.dat
phpfina/226.dat
phpfina/307.meta
phpfina/255.dat
phpfina/262.dat
phpfina/291.meta
phpfina/8.meta
phpfina/248.meta
phpfina/249.dat
phpfina/251.meta
phpfina/325.meta
phpfina/5.meta
phpfina/69.dat
phpfina/270.meta
phpfina/293.dat
phpfina/3.dat
phpfina/266.meta
phpfina/7.dat
phpfina/1.meta
phpfina/317.dat
phpfina/259.dat
phpfina/289.meta
phpfina/237.meta
phpfina/293.meta
phpfina/319.meta
phpfina/324.meta
phpfina/317.meta
phpfina/291.dat
phpfina/252.meta
phpfina/272.meta
phpfina/197.meta
phpfina/318.dat
phpfina/9.dat
phpfina/292.dat
phpfina/253.dat
phpfina/253.meta
phpfina/320.meta
phpfina/4.dat
phpfiwa/
phptimeseries/
phptimeseries/feed_321.MYD
Removing compressed backup to save disk space..
Stopping services..
Emoncms MYSQL database import...
ERROR at line 1: Unknown command '\-'.
Error: failed to import mysql data
Import failed

I tried importing my data in the Nov 23 and Jan 24 versions - before AND after full updates and nada…

Hello @phamiltonsmith it seems to be failing on the line that imports the mysql database:

mysql -u"${username}" -p"${password}" "${database}" < "${import_location}/emoncms.sql" 

https://github.com/emoncms/backup/blob/master/emoncms-import.sh#L122.

Do you still have the old SD card? could you try the USB reader import method?

It might be worth trying the following manually from inside the exported archive directory:

mysql -uemoncms -pemonpiemoncmsmysql2016 emoncms < emoncms.sql

It might give some insight as to what is going wrong…

@TrystanLea

Tks for the reply.

No… The backup was made from an Ubuntu instal

I will try this - ill update then

TKs

Edited post - BT, Moderator

Did you check the backup log to make sure it had worked?

Yes, I got a backup successful msg at the end for both of the backups I made…

Ah but were there any error messages. That simply means the script ran successfully, not that it backed up successfully!

[edit]
BTW, click on the reply button under my message (not the big blue one) or quote the message else the person you reply to, doesn’t know you replied to them.

hmmm… dont recall any error messages… !

1 Like

Ok, so some updates…

  1. When doing import archive and export data I can ONLY do it manually via SSH. The web interface does not work (when clicking on the options, nothing happens)

  2. I manually exported an archive from my existing PI install and then immediately manually re-imported and all was good. So, thats positive…

  3. When I try to import any of my OLD backups, NONE work. From my previous Ubuntu install I had kep the previous 4 backups - none worked. As @borpin mentioned, its possible thats a backup has errors but I would argue its unlikely all 4 do so…

  4. I find interesting that the names generated are different…
    emoncms-backup-emonpi-2024-11-21.tar.gz << from existing emon PI install
    emoncms-backup-emonserver-2024-11-07.tar.gz << from previous Ubuntu install.

Is it possible naming conventions are affecting my data imports?

I am reaching the limit of my ideas… Altough I have backups from previous install, my old data appears to be lost…

P

That part of the name is simply the server / host name.

Actually, if there is something wrong with the backup script, then I think it is highly likely :slight_smile:

That is a service runner issue.

I suspect there is a problem with the backup in Ubuntu. We know there are permission issues with an Ubuntu install. I never use emoncms backup on my Ubuntu setup (just get PVE to do a LXC backup).

Do you still have the old system? You could try and fix the backup.

First rule of backups is to not assume they will work :wink:

Indeed depressing - I dont have my old system… I have since formatted that machine

And, yes, I DID assume backups were good…

I have learned my lesson!! :sob:

Could you share your LXC backup process? What are your export time intervals? when You import that data to a new system how do you setup the imputs so the new feeds linkup seamlessly?

Tks
P

Automated with PVE daily to my NAS. Click on the ‘Datacentre’ (if using PVE, then backup.

Did the data files at least survive? You said it was the SQL bit that failed? I think you can reconnect to the data files (the Feed data) - not completely sure - @TrystanLea?

BTW, you can just highlight the bit you want to Quote and click on ‘Quote’ in the pop-up dialogue.

Hi,

I had the same problem after migrating from Bullseye to Bookworm OS and was able to solve it as follows:

  • Extract “emoncms.sql” from your “emoncms-backup-name-date.tar.gz” backup archive
  • Edit this file and remove the first line “/*M!999999- enable the sandbox mode */ ”
  • Save this file back into the archive
  • Import the modified archive

An explanation of the compatibility problem can be found here

2 Likes

Woah!!! MUCH MUCH MUCH appreciated!! It worked a charm!!!

I was able to recover MANY MONTHS (and some threads YEARS) of data!!

:grinning: :grinning: :grinning: :grinning: :grinning: :grinning: :grinning: :grinning:

@TrystanLea - need to note this in the docs somewhere!!

From the link @physikus posted one of the the following needs to be added to the backup/import script.

  • Remove the line at backup time with something like mariadb-dump|tail +2.
  • Remove the line at import time with something like tail +2|mariadb.

Import of backup fails due to MAriaDB version incompatibility · Issue #80 · emoncms/backup

1 Like

Thanks @borpin @physikus

Would I add this here? https://github.com/emoncms/backup/blob/master/emoncms-import.sh#L122.

    echo "Emoncms MYSQL database import..."
    tail +2|emoncms.sql
    mysql -u"${username}" -p"${password}" "${database}" < "${import_location}/emoncms.sql"
1 Like

You will need to test it and see I think.

You will need to check the file to see if the line exists as I don’t think it will in a backup from a new system. You can’t assume the line to remove does exist!

I think this problem will occur if it is a backup from an older system into a new system.

It might be better to remove the line on creating the backup.

New MariaDB versions create dump files with this sandbox mode command. Older versions do not understand this command, and they will error on it. Import will not proceed in this case.

In my example I have tried to import a backup created with 10.5.26-MariaDB-0+deb11u2 (with sandbox feature) on Pi OS 11 (bullseye) after upgrade to Pi OS 12 (bookworm) with 10.11.3-MariaDB-1+rpi1 (unfortunately without sandbox feature). Using an original Pi OS image I have run the EmonCMS Installation Script (init.sh) instead of using the official emonSD-01Feb24 image. So I am not sure if this problem is relevant for many users.

Today, I have tested my system with the addidional “tail +2|emoncms.sql” line that did not work. So it was not that easy.

Finally, I found a working solution by adding four lines to the import script after line 122, also including the conditional query, if the line does exist:

if grep -q "enable the sandbox mode" ${import_location}/emoncms.sql ; then
	echo "Warning: removing sandbox mode command from emoncms.sql to be compatible with older MariaDB versions"
	sed -i 1,1d ${import_location}/emoncms.sql
fi

BTW, after successful import, the ${import_location} folder still contained one file and one folder: The empty folder “phpfiwa” and “settings.ini” were not automatically removed by the import script. Maybe this can be added to the # cleanup section after line 152.

1 Like

Thanks @physikus is there any chance that you can create a pull request with this change? or copy the full files in a zip file attached here?