Manually editing "rogue" data + upgrading from 10.0.1

Hi there,

Part one: Rogue data

I am currently running version: 10.0.1 on a Raspberry Pi which I installed myself (it’s not emonSD). For quite a few years it has worked mostly very well monitoring my mains usage, DIY solar generation etc.

Back a while some rogue data crept in. On the graphs this manifests itself as one day with a massive spike of positive “usage” and one day with a massive anti-spike of negative “usage”.

Normal days:

Rogue data on graph:

More normal days after the aberration:

I had suffered several power blips on/around the strange days, and I think this was the cause; as you can see it normalised afterwards and has been working fine ever since.

I don’t mind the lost data, really. I can just accept that. But I would like to zap or zero the data on those two days so that when I’m looking at a broad sweep of time, it doesn’t ruin the scale.

In my quest to do this I had a look at the tables in the mariadb and couldn’t really make head nor tail of where to start with it. In doing some fiddling to achieve this I noticed I was far far behind on my EmonStuff versions, and probably ought to upgrade.

Part two: Upgrading

I upgraded the OS successfully just with apt-get upgrade.

However the buttons on the /emoncms/admin/view page do not work. They do nothing. I searched this forum and the first response everyone suggests is to post logs. But there is also nothing in the logs indicating why the upgrade buttons do not work. Perhaps this is only a function if you’ve used EmonSD? I am not sure.

My final thought was:

Since the Pi only does EmonStuff, should I, for an easy life, simply run EmonSD? And if I do that, is there an easy way to import the data from the old installation? I also searched this forum for how to export the data, and it seemed to refer to functions my admin page simply does not have… Which makes me wonder if I also didn’t install it properly in the first place. I literally just have the /emoncms/admin/view page, and no other admin pages, or an admin toolbar etc.

Sorry for the ramble. Any help anyone can throw on any of these matters would be much appreciated.

The data isn’t stored in the MariaDB database.

That will not update emoncms. Update to emoncms 11.x or start with a freash card and the emonSD image (or get one from the shop) and import the old data by USB (see below).

Have you tried clearing the Browser Cache?

Yes. The benefit is likely to be longer life for your SD Card.

https://docs.openenergymonitor.org/emoncms/import.html

Your version is possibly too old. Use the USB import method.

1 Like

That handles the “import”. But before I do an import, I need to do the “export”!

One problem with this is it seems to require me to install the backup tool, which (as a prerequisite) insists I need a latest version of the base package installed. So I’m recycling back to the need to upgrade my existing installation, I think. At which point, I might as well stick with the updated version.

Not if you use a new card (which you should), as you import via USB from the old card.

1 Like

Ah! Sorry! So you insert the old card into a USB<>SD reader, put that in the USB port on the Pi, which by then has the “new” EmonSD?

Have I understood correctly?

Yes, exactly.

1 Like

Would you view it as sensible to find a way to cure my rogue data before doing this, or wait until after? Example: Maybe the new version has some tools to do manual edits?

Correcting data is more tricky. Have a search on editing the data. The old way via a visulaisation is currently broken (@TrystanLea).

2 Likes

I think what I’m getting is broadly, that it doesn’t really matter whether I attempt to edit it before or after upgrading (it’s a pain either way!) but that maybe there are some new tools in later versions, although they might be a bit broken just now.

I think my plan of attack is :

  1. buy a new EmonSD. update: now ordered from OpenEnergyMonitor shop
  2. try it, and if possible, do the import from old SD-via-USB
  3. see about editing the old data

Many thanks for your help.

2 Likes

The easiest way to edit at the moment is using the graph CSV export interface to extract a section of data that you would like to edit and then the feeds page CSV import tool to import that data to overwrite the existing data for those timestamps.

2 Likes

Ok this is excellent, thank you.

You’ve totally steered me in the direction of “get onto the newest Emon$stuff” … then once on that, there’s a nice method of editing.

Fantastic. Much obliged.

I’ve already ordered the new EmonSD from the shop. So once that’s here, I’ll have a go at importing the old (inc. duff) data from my old install, then editing it.

Thanks again.

2 Likes

The EmonSD arrived, and I did an update first thing. Looks great, very easy.

I did attempt to import the install from the old SD card but got this :

=== USB Emoncms import start ===
2023-03-08-15:26:10
Backup module version:
cat: /opt/emoncms/modules/backup/backup-module/module.json: No such file or directory
EUID: 1000
Reading /opt/emoncms/modules/backup/config.cfg....
Location of data databases: /var/opt/emoncms
Location of emonhub.conf: /etc/emonhub
Location of Emoncms: /var/www/emoncms

Scanning for USB card reader:
- Unmounted disk: /dev/disk/by-id/usb-Kingston_Multi-Reader_-0_08738180105837-0:0 at /dev/sda
Error: Error opening /dev/sda: No medium found
- Error: Unmounted disk has incorrect number of partitions
- Unmounted disk: /dev/disk/by-id/usb-Kingston_Multi-Reader_-1_08738180105837-0:1 at /dev/sdb
Error: Error opening /dev/sdb: No medium found
- Error: Unmounted disk has incorrect number of partitions
- Unmounted disk: /dev/disk/by-id/usb-Kingston_Multi-Reader_-2_08738180105837-0:2 at /dev/sdc
Error: Error opening /dev/sdc: No medium found
- Error: Unmounted disk has incorrect number of partitions
USB drive not found

Are Kingston readers just not supported? Is there one that is “known good”?

Also, and I’ve hunted high and low for this… But to no avail.

What is the username/password pair for ssh? I enabled it correctly by adding the ssh file to the boot volume, but none that I try (including ones I’ve found searching in the docs and on the forums) work.

EDIT: sorrry, I didn’t realise I had to open out the Nov22 thing.

The answer is pi / emonsd

1 Like

Ok, I’ve managed to get in via SSH and do a dmesg :

[  465.182216] usb 1-1.2: new high-speed USB device number 5 using dwc_otg
[  465.340573] usb 1-1.2: New USB device found, idVendor=11b0, idProduct=6368, bcdDevice=12.27
[  465.340612] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  465.340630] usb 1-1.2: Product: Multi-Reader
[  465.340644] usb 1-1.2: Manufacturer: Kingston
[  465.340659] usb 1-1.2: SerialNumber: 08738180105837
[  465.347417] usb-storage 1-1.2:1.0: USB Mass Storage device detected
[  465.350135] scsi host0: usb-storage 1-1.2:1.0
[  465.420773] usbcore: registered new interface driver uas
[  466.401813] scsi 0:0:0:0: Direct-Access              Multi-Reader  -0 1.00 PQ: 0 ANSI: 4
[  466.406128] sd 0:0:0:0: [sda] Media removed, stopped polling
[  466.414398] scsi 0:0:0:1: Direct-Access              Multi-Reader  -1 1.00 PQ: 0 ANSI: 4
[  466.419264] sd 0:0:0:1: [sdb] Media removed, stopped polling
[  466.421362] sd 0:0:0:0: [sda] Attached SCSI removable disk
[  466.427801] sd 0:0:0:1: [sdb] Attached SCSI removable disk
[  466.436048] scsi 0:0:0:2: Direct-Access              Multi-Reader  -2 1.00 PQ: 0 ANSI: 4
[  466.439434] sd 0:0:0:2: [sdc] Media removed, stopped polling
[  466.441516] sd 0:0:0:0: Attached scsi generic sg0 type 0
[  466.441752] sd 0:0:0:1: Attached scsi generic sg1 type 0
[  466.441951] sd 0:0:0:2: Attached scsi generic sg2 type 0
[  466.447734] scsi 0:0:0:3: Direct-Access              Multi-Reader  -3 1.00 PQ: 0 ANSI: 4
[  466.448435] sd 0:0:0:3: Attached scsi generic sg3 type 0
[  466.452927] sd 0:0:0:2: [sdc] Attached SCSI removable disk
[  466.455904] sd 0:0:0:3: [sdd] Media removed, stopped polling
[  466.460240] sd 0:0:0:3: [sdd] Attached SCSI removable disk
[  467.566672] w1_master_driver w1_bus_master1: Attaching one wire slave 00.100000000000 crc 9d
[  467.573663] w1_master_driver w1_bus_master1: Family 0 for 00.100000000000.9d is not registered.
[  473.848394] sd 0:0:0:3: [sdd] 31116288 512-byte logical blocks: (15.9 GB/14.8 GiB)
[  473.850342] sdd: detected capacity change from 0 to 31116288
[  473.858158]  sdd: sdd1 sdd2
[  506.697024] w1_master_driver w1_bus_master1: Attaching one wire slave 00.900000000000 crc 11
[  506.707380] w1_master_driver w1_bus_master1: Family 0 for 00.900000000000.11 is not registered.
[  581.917836] w1_master_driver w1_bus_master1: Attaching one wire slave 00.500000000000 crc db
[  581.927855] w1_master_driver w1_bus_master1: Family 0 for 00.500000000000.db is not registered.
[  633.888500] w1_master_driver w1_bus_master1: Attaching one wire slave 00.d00000000000 crc 57
[  633.897562] w1_master_driver w1_bus_master1: Family 0 for 00.d00000000000.57 is not registered.
[  660.348954] w1_master_driver w1_bus_master1: Attaching one wire slave 00.300000000000 crc be
[  660.358584] w1_master_driver w1_bus_master1: Family 0 for 00.300000000000.be is not registered.
[  709.720105] w1_master_driver w1_bus_master1: Attaching one wire slave 00.b00000000000 crc 32
[  709.729868] w1_master_driver w1_bus_master1: Family 0 for 00.b00000000000.32 is not registered.
[  748.999849] w1_master_driver w1_bus_master1: Attaching one wire slave 00.700000000000 crc f8
[  749.009837] w1_master_driver w1_bus_master1: Family 0 for 00.700000000000.f8 is not registered.
[  800.970443] w1_master_driver w1_bus_master1: Attaching one wire slave 00.f00000000000 crc 74
[  800.980366] w1_master_driver w1_bus_master1: Family 0 for 00.f00000000000.74 is not registered.
[  863.440566] w1_master_driver w1_bus_master1: Attaching one wire slave 00.080000000000 crc c2
[  863.450557] w1_master_driver w1_bus_master1: Family 0 for 00.080000000000.c2 is not registered.
[  916.730796] w1_master_driver w1_bus_master1: Attaching one wire slave 00.880000000000 crc 4e
[  916.737645] w1_master_driver w1_bus_master1: Family 0 for 00.880000000000.4e is not registered.
[  954.531114] w1_master_driver w1_bus_master1: Attaching one wire slave 00.480000000000 crc 84
[  954.541063] w1_master_driver w1_bus_master1: Family 0 for 00.480000000000.84 is not registered.
[ 1005.201534] w1_master_driver w1_bus_master1: Attaching one wire slave 00.c80000000000 crc 08
[ 1005.211630] w1_master_driver w1_bus_master1: Family 0 for 00.c80000000000.08 is not registered.
[ 1081.881860] w1_master_driver w1_bus_master1: Attaching one wire slave 00.280000000000 crc e1
[ 1081.891855] w1_master_driver w1_bus_master1: Family 0 for 00.280000000000.e1 is not registered.
[ 1145.402435] w1_master_driver w1_bus_master1: Attaching one wire slave 00.a80000000000 crc 6d
[ 1145.412619] w1_master_driver w1_bus_master1: Family 0 for 00.a80000000000.6d is not registered.
[ 1208.932846] w1_master_driver w1_bus_master1: Attaching one wire slave 00.680000000000 crc a7
[ 1208.944060] w1_master_driver w1_bus_master1: Family 0 for 00.680000000000.a7 is not registered.

So maybe, if I’m reading it correctly, it is mounting it on /dev/sdd ? it looks that way.

But the import system only seems to go as far as /dev/sdc?

At least it seems to give up after sdc…

Is there any way to get it to try sdd?

Ok. This is weird. I found an adaptor to convert the micro SD to a standard SD, figuring that might solve it. Shutdown the pi from SSH, booted it with the card removed.

Now it won’t let me in saying I have the wrong password. Despite the fact I saved it in my password manager.

Any way to reset the password for my user from ssh? Again, searched all the docs; nothing for “forgot password” / “lost password” etc.

Ok, when I search the source of the login page, it definitely has some stuff about password recovery. But that isn’t actually displayed, and it’s javascript, so it’s not something I can poke.

Can I just zap the password field in the users table in the mysql? Or will that break things much much worse?

Sigh. I did a “delete from users;” … I was hoping to put it back to a “no users / not yet configured” state. This didn’t work.