Setting up home NAS - Part 5 - e-mailing rsync report

This is the continuation of the "Setting up home NAS" series. Better read the below posts to get context.

Setting up home NAS - Part 0 - RAID v/s Backup rsync v/s Raspberry Pi v/s Cloud Storage for Youtuber in the year 2020

Setting up home NAS - Part 1 - Scientific debugging to get USB drive visible as \\share in network

Setting up home NAS - Part 2 - PowerShell arrange media date bases files into YYYY\MM\DD folders

Setting up home NAS - Part 3 - Dealing with duplicates

Setting up home NAS - Patt 4 - Raspberry Pi to build locally redundant NAS for home

Setting up home NAS - Part 5 - e-mailing rsync report (This post)

Introduction

In the previous post, we saw how a RaspberryPi can be used to build locally redundant NAS. We used rsync command to copy folders and files from the primary hard disk to the secondary. There was no way to monitor whether it works every day. This post is to explain a monitoring mechanism that is nothing but sending a summary mail at end of every sync operation daily.

Approach

The high-level solution goes as follows

• Have the rsync commands output logs to a folder
• Merge the individual rsync reports into a single file by taking only what we need. This is what we are going to email every day. Not the raw rsync output.
• Send the report using the appropriate email provider. 

Prepare the report

This includes the generation and formatting of rsync report

rsync to output logs

The rsync command that we saw in the previous post needs to be modified a little bit as follows

rsync /media/usbseagate1tbc/media/ /mnt/router/media/ -av --progress --delete --out-format="%t %f %''b" >> "/var/log/j3dnas/$(date +%F)/Media.log" 2>&1

This means the output of rsync will be saved to a dated folder with the required format. The path needs to be changed as per our setup. 

The rsync command has a lot of options to choose the log format. Experiment with those to tune the format.

create directory if not exist

In case the directory named with the current date is not there, create it as follows.

mkdir "/var/log/j3dnas/$(date +%F)" -p

Merge rsync logs with grep

If there are multiple folders we are syncing it will produce multiple log files. Those can be merged using the below simple grep command.

grep "$(date +%Y/%m/%d)\|deleting\|error\|failed" --no-filename /var/log/j3dnas/$(date +%F)/*.log >> "/var/log/j3dnas/$(date +%F)/status.txt"

grep has many options similar to rsync. Explore to tune the merge output. Before appending the logs add the subject line. The mailing libraries will take the first line as the subject of the mail.

echo -e "Subject: J3D Family NAS rsync status $(date +%F)\r\n" >> "/var/log/j3dnas/$(date +%F)/status.txt"

The merged status.txt file is what we will be sending in the email.

Sending eMail

Choosing mail provider

Gmail comes to our mind when we think about mails. We can use it to send mails here. It uses 587 port. Based on our experience it can be any mail provider. The main point to consider is they support SMTP to send mails.

Once the service provider is decided, create a new email account or use any existing account that is already in use to send these types of notifications. 

Mailing library

Next, we need to decide on what email library/command to be used. Personally, I started with ssmtp but later I found that ssmtp won't work in the latest Raspberry Pi. Finally ended up with msmtp. More on how to send mail from RasPi using msmtp can be found here. The credentials need to be stored in the /etc/msmtprc file.

In case there are issues in using the plain password of the email account, use the app password method of Google to send the email.

cat /var/log/j3dnas/$(date +%F)/status.txt | msmtp  --debug --from=default -t joymon@gmail.com

The command to send mail will be like the above.

Final script

The final script will look something like the below.

#!/bin/bash

mkdir "/var/log/j3dnas/$(date +%F)" -p

# rsync and log output to datewise folder

rsync /media/usbseagate1tbc/users/ /mnt/router/users/ -av --progress --delete --out-format="%t %f %''b" >> "/var/log/j3dnas/$(date +%F)/users.log" 2>&1

rsync /media/usbseagate1tbc/Media/ /mnt/router/Media/ -av --progress --delete --out-format="%t %f %''b" >> "/var/log/j3dnas/media_$(date +%F)/Media.log" 2>&1

# Push the email subject and modified file list to the status file

echo "Subject: J3D NAS Sync Status $(date +%F)" > "/var/log/j3dnas/$(date +%F)/status.txt"

grep "$(date +%Y/%m/%d)\|deleting" /var/log/j3dnas/$(date +%F)/*.log >> "/var/log/j3dnas/$(date +%F)/status.txt"

# Send mail

cat status.txt | msmtp  --debug --from=default -t joymon@gmail.com