Complete Influx Ver. 1.x TICK Stack Disaster Recovery

Note: This only works for Version 1.x (1.6 specifically for me), I have since upgraded to Version 2.x.

My entire system became corrupt one day and while it was technically booting it was not functioning. I did not have proper backups so the road to recovery was long & painful. I now have better emphasis on backups.

Typically all Influx data is backed up by:

influxd backup -portable /media/usb/drive

and restored with

influxd restore -portable /media/usb/drive

I did not have this luxury so I started with copying all the main files to and external drive, these were:


Okay we are now finished with the corrupted image, do a full fresh install of your system. (tutorial)

Great we are now all setup, insert USB where files were backed up to before. We need to find the UUID of the USB and then edit fstab to mount the USB automatically:

ls -l /dev/disk/by-uuid/
sudo nano /etc/fstab

Add below line to fstab: (edit UUID for your device)

UUID=18A9-9943 /media/usb vfat auto,nofail,noatime,users,rw,uid=pi,gid=pi 0 0

Now we need to tell Influx config to look at memory stick, edit the below file with:

  #dir = "/var/lib/influxdb/meta"
  dir = "/media/usb/drive/meta"

  #dir = "/var/lib/influxdb/data"
  dir = "/media/usb/drive/data"

  #wal-dir = "/var/lib/influxdb/wal"
  wal-dir = "/media/usb/drive/wal"

I also needed to change the user of the files on the USB by:

sudo chown -R influxdb:influxdb /media/usb/drive

We will revert some of the above changes later on.
Note: My original plan was to have all files on the USB drive permanently but as soon as I added the data source in Chronograf everything broke so I undid this. I just used this step to export the data properly.

Reboot system.

Now all your old data should be loaded.

Now we will create a proper backup of the data with the below:

influxd backup -portable /media/usb/drive

Revert all changes in the influxdb.conf file:

sudo nano /etc/influxdb/influxdb.conf 

Now restore all data back to the default locations by:

influxd restore -portable /media/usb/drive

Since it took me a few days to figure out how to restore data I already had the system back up recording data, the above restore does not work if a database is already created so I had to side-load all databases in with:

USE my_data_bak
DROP DATABASE my_data_bak

Finally add back in your Chronograf alerts etc. by:

sudo mv /var/lib/kapacitor/kapacitor.db /var/lib/kapacitor/kapacitor_orig.db 
sudo mv /media/usb/drive/kapacitor.db /var/lib/kapacitor/
sudo chown -R kapacitor:kapacitor /var/lib/kapacitor/kapacitor.db

Future planning would be to keep regular backups with: (you need to do this individually for all databases). See my other post on this.

influxd backup -portable /media/usb-influx/backup
kapacitor backup /media/usb/drive/kapacitor.db

Reboot and we are done!

Resources used to create this page:

Leave a Reply

Your email address will not be published. Required fields are marked *