Simple Data Backup with rsync

I struggled for years to manage simple home data backups effectively but a nice Linux tool (rsync) exists to make it very manageable, here are some use cases I use frequently to make it a breeze:

A straight copy of one drive to another:

  • n – this will show the output without doing anything (dry run), remove this to run the backup.
  • r – this means recursive, basically it will catch all files.
  • u – this skips files that are newer on ‘drive2’, I use this to ensure files are a true copy in case of a mix-up.
  • v – makes the output verbose, gives lots of information on progress.
rsync -nruv /media/drive1/ /media/drive2

As of recent I started using the –checksum argument due to not following my own rules and doing a backup that overwrote modification time of all files, this argument looks at the file checksum (unique identifier) as opposed to the modification time to do a backup:

rsync -nruv --checksum /media/drive1/ /media/drive2

This was supposed to just get you on your feet with backups, a lot more options are available, read on here perhaps:

By modifying a tutorial on the opensource blog I was really able to streamline and speed up the backups, save the below as a shell script, it makes light work of multiple Terabytes of data:

DIRS="directory_to_copy"
SRC="/media/drive1"
DEST="/media/drive2"

for DIR in $DIRS; do
     cd "$SRC"/$DIRS
     rsync -cdlptgov --delete . /"$DEST"/$DIR
     find . -maxdepth 1 -type d -not -name "." -exec rsync -crlptgov --delete {} /"$DEST"/$DIR \;
done

Resources I used:
https://ss64.com/bash/rsync.html
https://opensource.com/article/19/5/advanced-rsync
https://www.computerhope.com/unix/rsync.htm

That’s it!

Easy Media Management with exiftool

I struggled for years to manage pictures effectively but a nice Linux tool (exiftool) exists to make it very manageable, here are some use cases I use frequently:

Change all times of all pictures in directory by one hour:
(change number/sign for any other hours, it is clever and will adjust day if crosses midnight etc.)

$ exiftool -AllDates+=1 -overwrite_original *

Remove all EXIF metadata from images with “.jpeg” extensions only:

exiftool -all= *.jpeg

Add if statements to operations if required: (for example only adjust picture taken with Canon Cameras)

$ exiftool -AllDates+=1 -overwrite_original -if '$make eq "Canon"' -r *

Some other rename commands that help with naming if required:

Cuts start of filename: (by 4 letters, adjust as required) (-n = dry run)

$ rename -n -v  's/^(.{4})//' *

Changes file extensions from upper case to lower case.

rename 'y/A-Z/a-z/' *.JPG

That’s it for now!

Flight Tracking on RPi Zero. (Updated to include ADSB Exchange)

Originally I ‘built my own’ FlightRadar24 node using their straightforward tutorial. Its cool and you get free premium membership on their mobile app but a lot of flights are censored which is silly as I can read them directly off my node but FR24 won’t let me see them on the app. So I ended up doing a fresh install and now I feed FR24 and also ADSB Exchange which don’t censor any flights and have some awesome local interfaces, see below for tutorial:

Start with a fresh Raspbian install on RPI. Then we install:

sudo apt-get install dump1090-fa
sudo apt-get install piaware #Not 100% sure if this line required
sudo bash -c "$(wget -O - http://repo.feed.flightradar24.com/install_fr24_rpi.sh)"

The FR24 setup should guide you through first setup if its your first time, other wise you can reconfigure if you have your existing FR24 key:

sudo fr24feed --reconfigure --fr24key=your_key

I restarted the service, not sure if necessary, everything should be feeding FR24 now.

sudo systemctl restart fr24feed

See if service is running:

fr24feed-status

You can also see status if you go the FR24 local web GUI at http://192.168.1.XXX:8754/

Next we setup feed to ADSB Exchange:

sudo bash -c "$(wget -nv -O - https://raw.githubusercontent.com/adsbxchange/adsb-exchange/master/install.sh)"

It should also guide you through first setup, very straight forward, when finished you can check if service running:

sudo systemctl status adsbexchange-feed

You can see if your data is getting to ADSB Exchange by going to https://www.adsbexchange.com/myip/ and you can view the global aggregated data at https://tar1090.adsbexchange.com/

There are a few other fantastic packages to install to see current stats of your system:

Tar1090 is an amazing package that shows you what your node is currently seeing, install below and then use web interface at http://192.168.1.XXX/tar1090/

sudo bash -c "$(wget -q -O - https://raw.githubusercontent.com/wiedehopf/tar1090/master/install.sh)"

Timelapse1090 shows historical flights and you can replay flights etc, works well but data only stored in RAM so lost over reboot, my next plan is to write this to a database (now done). (viewable at: http://192.168.1.XXX/timelapse/)

sudo bash -c "$(wget -q -O - https://raw.githubusercontent.com/wiedehopf/timelapse1090/master/install.sh)"

Graphs1090 is supposed to show you performance of your node, number of aircraft seen etc., didn’t get it working yet but works well for others: (viewable at: http://192.168.1.XXX/graphs1090/)

sudo bash -c "$(wget -q -O - https://raw.githubusercontent.com/wiedehopf/graphs1090/master/install.sh)"

You can also see the data stream from your node by the below CLI command:

nc localhost 30003

As an FYI below you can see the screenshots of the FR24 app showing the performance of my node a week after I set it up:

RPi Network CCTV Stream

I used motioneyeos for a number of years on Raspberry Pi’s (as both a Fast Network Camera and a NVR on separate devices) and while it was helpful for live viewing, the RPI really struggled on the recording frame rate.

I have since invested in a professional NVR but since I had the RPI’s lying around I decided to set them up to stream on my network and let them be captured by my new NVR or any other device that I want.

Start with a fresh install on the RPI and run the below commands:

raspi-config
#enable the camera if not done so already
sudo apt-get install ntpdate
sudo apt-get install vlc

Create a file on the Desktop called stream-rtsp.sh as per the below:

nano stream-rtsp.sh
!/bin/bash
raspivid -o - -t 0 -w 1296 -h 972 -fps 8 -b 2500000 -rot 180 -a 12 | cvlc -vvv stream:///dev/stdin --sout '#rtp{access=udp,sdp=rtsp://:8554/stream}' :demux=h264

Make the file execturable:

chmod +x stream-rtsp.sh

Test the script by running it manually:

./stream-rtsp.sh

Making the script run on startup by creating the file:

sudo nano /etc/systemd/system/stream-rtsp.service
[Unit]
Description=auto start stream
After=multi-user.target

[Service]
Type=simple
ExecStart=/home/pi/stream-rtsp.sh
User=pi
WorkingDirectory=/home/pi
Restart=on-failure

[Install]
WantedBy=multi-user.target

Set the service to auto start:

sudo systemctl enable stream-rtsp.service

Reboot the system and confirm the system started the service by:

sudo systemctl start stream-rtsp.service

Now lets check if the device is successfully streaming by on a different device launching VLC and navigating to Media –> Open Network Stream, enter the below (modify for your IP address) and click play:

rtsp://192.168.1.xxx:8554/stream

You should now see your camera screen. I will show you how to add this to Hikvision NVR in a later post.