Complete Influx TICK Stack Disaster Recovery

My entire system became corrupt one day and while it was technically booting it was not functioning. I did not have proper backups so the road to recovery was long & painful. I now have better emphasis on backups.

Typically all Influx data is backed up by:

influxd backup -portable /media/usb/drive

and restored with

influxd restore -portable /media/usb/drive

I did not have this luxury so I started with copying all the main files to and external drive, these were:

/var/lib/influxdb/data
/var/lib/influxdb/wal
/var/lib/influxdb/meta
/var/lib/kapacitor/kapacitor.db

Okay we are now finished with the corrupted image, do a full fresh install of your system. (tutorial)

Great we are now all setup, insert USB where files were backed up to before, we need to tell Influx config to look at memory stick, edit the below file with:

sudo nano /etc/influxdb/influxdb.conf 
[meta]
  #dir = "/var/lib/influxdb/meta"
  dir = "/media/usb/drive/meta"

[data]
  #dir = "/var/lib/influxdb/data"
  dir = "/media/usb/drive/data"

  #wal-dir = "/var/lib/influxdb/wal"
  wal-dir = "/media/usb/drive/wal"

I also needed to change the user of the files on the USB by:

sudo chown -R influxdb:influxdb /media/usb/drive

We will revert some of the above changes later on.
Note: My original plan was to have all files on the USB drive permanently but as soon as I added the data source in Chronograf everything broke so I undid this. I just used this step to export the data properly.

Reboot system.

Now all your old data should be loaded.

Now we will create a proper backup of the data with the below:

influxd backup -portable /media/usb/drive

Revert all changes in the influxdb.conf file:

sudo nano /etc/influxdb/influxdb.conf 

Now restore all data back to the default locations by:

influxd restore -portable /media/usb/drive

Since it took me a few days to figure out how to restore data I already had the system back up recording data, the above restore does not work if a database is already created so I had to side-load all databases in with:

influxd
CREATE my_data_bak
USE my_data_bak
SELECT * INTO my_data..:MEASUREMENT FROM /.*/ GROUP BY *
DROP DATABASE my_data_bak
exit

Finally add back in your Chronograf alerts etc. by:

sudo mv /var/lib/kapacitor/kapacitor.db /var/lib/kapacitor/kapacitor_orig.db 
sudo mv /media/usb/drive/kapacitor.db /var/lib/kapacitor/
sudo chown -R kapacitor:kapacitor /var/lib/kapacitor/kapacitor.db

Future planning would be to keep regular backups with: (you need to do this individually for all databases). See my other post on this.

influxd backup -portable /media/usb-influx/backup
kapacitor backup /media/usb/drive/kapacitor.db

Reboot and we are done!

Using integral function on Grafana (covert Watt to kWh)

After fighting for longer than I’d like to admit with this function I finally managed to get it working.

I use a single stat visualation and the below queries to give me energy usage in Watt/Hour from my data stored in Watts.

1hr Usage: (Relative time over-ride = 1h)

SELECT integral("Energy_Usage",1h) FROM "esp" WHERE ("Device" = 'esp_03') AND $timeFilter GROUP BY time(3h) 

24hr Usage: (Relative time over-ride = 24h)

SELECT integral("Energy_Usage",1h) FROM "esp" WHERE ("Device" = 'esp_03') AND $timeFilter GROUP BY time(3d) 

7 Day Usage: (Relative time over-ride = 7d)

SELECT integral("Energy_Usage",1h) FROM "esp" WHERE ("Device" = 'esp_03') AND $timeFilter GROUP BY time(21d) 

That’s it!

Installing TICK Stack on RPi4

Nothing complicated this time, just commands I use to setup my Influx TICK stack from fresh install.

sudo apt-get update
sudo apt-get upgrade
wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/os-release
test $VERSION_ID = "10" && echo "deb https://repos.influxdata.com/debian buster stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get install influxdb
sudo apt install influxdb-client
sudo apt-get update
sudo apt-get install telegraf
sudo apt-get install chronograf
sudo apt-get install kapacitor
sudo systemctl unmask influxdb.service 
sudo systemctl start influxdb 
sudo apt-get install fail2ban
sudo apt-get install ntp
sudo apt-get install ntpstat
systemctl stop systemd-timesyncd
systemctl disable systemd-timesyncd
/etc/init.d/ntp stop
/etc/init.d/ntp start
sudo reboot

Confirm everything is working:

sudo service kapacitor status
sudo service chronograf status
sudo service influxdb status
sudo service telegraf status
ntpstat

You can also head to the Chronograf configuration page on: http://192.168.1.xxx:8888

That’s it!