How much data are you archiving?
Right now it’s 1.7mb but I expect that to grow. I only have a few weeks of data so far.
Pipe takes the output of one command and feeds it as input to another command.
If your archive destination was on the same network as your archive source, then it would
be a matter of changing directory to the destination and running the commands on the source.
But as the second link I posted mentions, scp doesn’t know how to deal with STDIN/STDOUT
but ssh does. Have you had a look at the second link?
And yet another:
The article at the last link I posted sounds exactly like what you’re after.
Bill,
I looked at the 2nd one. I’ve also seen many on the web. My problem is that I’m not that experienced in linux commands. I’ve tried many of the solutions offered, all to no avail.
I’ll try this again later and will reply here if I make progress.
Thanks
OK. I posted a third link while you were typing…
Thanks. I’ll explore the ssh option rather than scp.
Have a great day.
Rather than bother trying to compress across the network, another technique would be to simply compress it first on the pi and then just copy the compressed file across. Since you don’t want to write it to the SD card, just write the compressed file to one of the many tmpfs (i.e. RAM-backed filesystems) on an emon system. /tmp stands out as an obvious choice.
Also, I’m not sure what Node-RED is helping with. Isn’t it simpler to just write a script and then put it in a cron job? Or even a systemd timer unit if you’re feeling ultra-modern (and brave).
1 Like
sure that is viable ? I have feeds running for over 2 years now (just emoncms on a real server) , one of them is 787 Mb … the second is 383 Mb … not sure even compressed that will fit in ram tmp file … or how long the pi would work on those kinda files …
1 Like
Hello Dave,
This I understand and can make work. Thanks
Your right, making Node-RED do it made me realize I could make the emoncms-export.sh script do the same thing. I’ll read up on system timer. I’m feeling brave… 
Regards
If you have certbot installed, it does just this for the certificate renewal - a good example to follow.
There is an old dropbox backup thread here Emoncms backup to Dropbox if anyone is interested.
Only problem is Dropbox now enforce a 3 machine limit on the free accounts for syncing.
Anyone looked at using GoogleDrive from the command line? I know it is possible as my WordPress site and my HA installation backs up to my GoogleDrive using an API key.
Ya musta’ missed that one, Dave… 
In his first post he said:
I thought about suggesting that too. But, as Eric alluded to, the OP said
So while it may fit now, the file may outgrow tempfs.
Sounds like he’s still a bit new at it, or at least not comfortable with the advanced aspects of Linux
I’d say my experience in Linux is ‘patchy’. But I don’t mind pressing buttons and asking why later
Good attitude!
Ya gotta be careful with that, though. It can bite ya in the backside when ya least expect it.
It would be nice if we could plugin a usb stick in to the Pi and run a nightly backup
Options as i see them are:
attach a USB stick on the Pi (which is easy - format it/mount it/ add an entry in /etc/fstab)
Change the backup script 
First set up some ssh-keys from the Pi to the backup server.
Have a base directory on the remote server to hold as “todays” backup area
($date is defined in the backup script)
define a new variable BACKUP_SERVER for your backup server
ssh $BACKUP_SERVER "cd $BACKUP_AREA ; mkdir $date"
edit the backup script
change mysqldump -u$username -p$password emoncms > $backup_location/emoncms.sql
to
mysqldump -u$username -p$password emoncms | ssh $BACKUP_SERVER "cd $BACKUP_AREA/$date ; > emoncms.sql"
change
to
tar -cf - $backup_location/emoncms.sql $emonhub_config_path/emonhub.conf $emoncms_location/settings.php --transform 's?.*/??g' 2>&1 | ssh $BACKUP_SERVER "tar -C $BACKUP_AREA/$date -xf - "
similarly
change
tar -vr --file=$backup_location/emoncms-backup-$date.tar -C $mysql_path phpfina
to (this one i’m guessing)
tar -cf - $mysql_path/phpfina | ssh $BACKUP_SERVER "tar -C $BACKUP_AREA/$date -xf - " ```
again same as above
tar -vr --file=$backup_location/emoncms-backup-$date.tar -C $mysql_path phptimeseries
to
tar -cf - $mysql_path/phptimeseries | ssh $BACKUP_SERVER "tar -C $BACKUP_AREA/$date -xf - " ```
In essence we are using unix’s power of redirecting stdout over the ssh tunnel to the remote server.
With the adjustments above you’ll not write to disc on the pi for any of the output.
worth a try 
Thanks Dave,
There’s a lot there for me to digest. I’ll work on this over the weekend and report back.
Neil
2019-04-12T04:00:00Z
I had it figured out for the compression parts but not yet this part:
I’m having trouble generating SSH Keys for ssh’ing into a MacBook. Works when I ssh into a RPi but not the other way around. I don’t want my server pw exposed in plain text. I’ll figure that out.
For now I found a spare USB stick and used that. Simples
Thanks to everyone for your input. I’m amazed at how helpful everyone is, its a great learning experience for me.
Best wishes