Setting up home NAS - Part 3 - Dealing with duplicates

This is the fourth post on Setting up home NAS. Below are the other posts on this series

Setting up home NAS - Part 0 - RAID v/s Backup rsync v/s Raspberry Pi v/s Cloud Storage for Youtuber in the year 2020

Setting up home NAS - Part 1 - Scientific debugging to get USB drive visible as \\share in network

Setting up home NAS - Part 2 - PowerShell arrange media date bases files into YYYY\MM\DD folders

Setting up home NAS - Part 3 - Dealing with duplicated (this post)

Setting up home NAS - Patt 4 - Raspberry Pi to build locally redundant NAS for home

Introduction

This post is not directly related to NAS (Network Attached Storage). If you are looking for NAS related stuff, you may stop reading from here onwards.

As mentioned in the previous post, my current set-up is.

• A router attached USB drive working as NAS. \\192.168.1.1\share. It is of 1TB capacity
• Sync the files to my personal computer that has 500GB drive. So effectively getting only 500GB which is backed up. SyncToy is working fine for on-demand sync
• When I complete editing the video files, render and publish to YouTube, I move files out from the above setup. Combine all the raw videos to single and upload to YouTube in private visibility as well as back up to another external disk that has 2TB.
• That means during editing time, 2 copies are in router attached USB drive and laptop. After publishing, one uploaded to YouTube and other copy in the different 2 TB hard disk.

This setup is throttled at 500GB. If I am able to process videos in a timely fashion and move to the second system of storage this will work. Else need to find some other mechanism to have a storage mechanism with backup. As we are seeing its not enterprise-level data storage. At least we don't lose data on single hard disk failure. 

Problem

When we set up NAS for home the main files stored will be media files such as photos, audio, and video files. They come from different sources such as multiple cameras, social media. The last post was about organizing media. That will show some duplicates and help us to remove them. But when the same file may come from different family and friends groups if there are mutual members. Though they are less compared to the file size from DJI Osmo Pocket and RAW files, they constitute a lot in the long run.

The solution that I am using is below.

Solution

This time, I am not writing any PowerShell program to deal with duplicates. There is a tool out there that is doing the job for me. 

Its name is AllDup, its free and working fast at least to my expectation. It supports the comparison of images as well as videos. There is an option to preview before we delete it. Plus a lot of options to control search behavior. Below is a screenshot showing different options.

If there is any better solution, please comment.