Copy Files with No Duplicates.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Copy Files with No Duplicates!\n",
    "\n",
    "## Instructions\n",
    "\n",
    "To use this notebook, run all cells in order, up to and including Python cell 8, which runs the main function.\n",
    "\n",
    "Follows the steps below to perform different actions.\n",
    "\n",
    "### Deduplicating a Directory\n",
    "\n",
    "Suppose I have some directory, `Directory 2022`.\n",
    "\n",
    "I want to create a copy of that directory which contains no duplicates.\n",
    "Note that here, duplicates mean files which are exactly the same content-wise (file name, age, and location are not taken into consideration).\n",
    "\n",
    "1. Run all cells, and select **COPY** mode\n",
    "2. Enter a hash table file, e.g. \"MyDirectoryHashed.json\"\n",
    "3. Select directory 2022 as your source (e.g. `C:\\My Files\\Directory 2022`)\n",
    "4. Select a new, empty directory as your destination (e.g. `C:\\My Files\\Deduplicated Directory`)\n",
    "5. (Optional) Enter *YES* to store additional info in the hash table file\n",
    "\n",
    "The deduplicated directory will contain all files in directory 2022, minus duplicates.\n",
    "In the case of a duplicate, only the oldest file will be kept.\n",
    "All file names (and additional info, if desired) will be kept in the hash table file.\n",
    "\n",
    "The source directory will be untouched.\n",
    "\n",
    "*Note: Do not specify the same directory as source and destination.  This will not deduplicate the directory as duplicates are never removed, only not copied.*\n",
    "\n",
    "\n",
    "### Merging a Directory into an Already Deduplicated Directory\n",
    "\n",
    "Suppose I have another directory, `Directory 2023`, which is a later version of the same directory from part 1.\n",
    "\n",
    "It will have a lot of the same files, which we don't want to keep, but also some new files.\n",
    "\n",
    "1. Run all cells, and select **COPY** mode\n",
    "2. Enter the existing hash table file created in part 1, e.g. \"MyDirectoryHashed.json\"\n",
    "3. Select directory 2023 as your source (e.g. `C:\\My Files\\Directory 2023`)\n",
    "4. Select the directory created in part 1 as your destination (e.g. `C:\\My Files\\Deduplicated Directory`)\n",
    "5. (Optional) Enter *YES* to store additional info in the hash table file; if you did this in part 1, it makes sense to do it again for consistency\n",
    "\n",
    "The deduplicated directory will now contain all files in directory 2022 and directory 2023, minus duplicates.\n",
    "\n",
    "The source directory will be untouched.\n",
    "\n",
    "\n",
    "### Taking Note of Duplicates Without Moving Files\n",
    "\n",
    "Suppose I have a directory, `My Stuff`, and I want to create a record of its contents and get a sense of how many duplicates there are.\n",
    "\n",
    "1. Run all cells, and select **MARK** mode\n",
    "2. Enter a hash table file, e.g. \"MyStuffHashed.json\"\n",
    "3. Select the directory as your source (e.g. `C:\\My Files\\My Stuff`)\n",
    "4. (Optional) Enter *YES* to store additional info in the hash table file\n",
    "\n",
    "All file names (and additional info, if desired) will be kept in the hash table file.  No files will have been copied.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "\n",
    "This notebook only uses Python Standard Library imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hashlib\n",
    "import json\n",
    "import os\n",
    "import shutil\n",
    "import time\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serializing and Deserializing the Hash Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_hash_table(hash_table_path: str) -> dict:\n",
    "    \"\"\"Loads a hash table dict from a file.\"\"\"\n",
    "\n",
    "    try:\n",
    "        with open(hash_table_path, 'r') as f_json:\n",
    "            return json.load(f_json)\n",
    "    except FileNotFoundError:\n",
    "        return {}\n",
    "\n",
    "\n",
    "def save_hash_table(hash_table_path: str, hash_table: dict):\n",
    "    \"\"\"Loads a hash table dict from a file.\"\"\"\n",
    "    \n",
    "    DEFAULT = 'file_data.json'\n",
    "\n",
    "    try:\n",
    "        with open(hash_table_path, 'w') as f_json:\n",
    "            json.dump(hash_table, f_json, indent=4, sort_keys=True)\n",
    "    except FileNotFoundError:\n",
    "        try:\n",
    "            with open(DEFAULT, 'w') as f_json:\n",
    "                json.dump(hash_table, f_json, indent=4, sort_keys=True)\n",
    "                print(f'Could not create json file; used default name ({DEFAULT}) instead.')\n",
    "        except FileNotFoundError:\n",
    "            global unsaved_hash_table\n",
    "            unsaved_hash_table = hash_table\n",
    "            print('Could not create json file.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hashing Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_md5(file_path: str) -> str:\n",
    "    \"\"\"Create an MD5 hash of a file's contents.\"\"\"\n",
    "    \n",
    "    hash_md5 = hashlib.md5()\n",
    "    try:\n",
    "        with open(file_path, 'rb') as f_to_hash:\n",
    "            for chunk in iter(lambda: f_to_hash.read(4096), b''):\n",
    "                hash_md5.update(chunk)\n",
    "    except FileNotFoundError:\n",
    "        return 'BADHASH'\n",
    "    \n",
    "    return hash_md5.hexdigest()\n",
    "\n",
    "\n",
    "def hash_file(hash_table: dict, file_path_object, dir_path: str = None, extra_details: bool = False ) -> bool:\n",
    "    \"\"\"Attempt to hash a file and return true iff it was not already hashed.\"\"\"\n",
    "\n",
    "    file_path = str(file_path_object)\n",
    "    file_size = str(os.path.getsize(file_path))\n",
    "    file_name = file_path_object.name\n",
    "    file_hash = create_md5(file_path)\n",
    "    combined_hash = f'{file_hash}{file_size}'\n",
    "    \n",
    "    if extra_details:\n",
    "        relative_path = file_path.replace(dir_path, \"\") if dir_path else file_path\n",
    "        \n",
    "        if combined_hash in hash_table:\n",
    "            if file_name not in hash_table[combined_hash]['names']:\n",
    "                hash_table[combined_hash]['names'].append(file_name)\n",
    "            if relative_path not in hash_table[combined_hash]['paths']:\n",
    "                hash_table[combined_hash]['paths'].append(relative_path)\n",
    "            return False\n",
    "        else:\n",
    "            hash_table[combined_hash] = {\n",
    "                'md5': file_hash,\n",
    "                'size': file_size,\n",
    "                'names': [file_name],\n",
    "                'paths': [relative_path],\n",
    "                'created': get_earliest_age(file_path_object)\n",
    "            }\n",
    "            return True\n",
    "\n",
    "    else:\n",
    "        if combined_hash in hash_table:\n",
    "            if file_name not in hash_table[combined_hash]['names']:\n",
    "                hash_table[combined_hash]['names'].append(file_name)\n",
    "            return False\n",
    "        else:\n",
    "            hash_table[combined_hash] = {\n",
    "                'md5': file_hash,\n",
    "                'size': file_size,\n",
    "                'names': [file_name]\n",
    "            }\n",
    "            return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Moving Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def move_files_relative(old_dir_path: str, new_dir_path: str, file_path: str):\n",
    "    \"\"\"Moves a file from one directory to another, keeping its relative file structure.\"\"\"\n",
    "\n",
    "    if os.path.isfile(file_path):\n",
    "        relative_path = file_path.replace(old_dir_path, \"\")\n",
    "        new_path = f'{new_dir_path}{relative_path}'\n",
    "        os.makedirs(os.sep.join(new_path.split(os.sep)[:-1]), exist_ok=True)\n",
    "        shutil.copyfile(file_path, new_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterating Through Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_earliest_age(file_path_object) -> int:\n",
    "    \"\"\"Returns the earliest of created and modified time for a file path object.\"\"\"\n",
    "    \n",
    "    try:\n",
    "        return min(\n",
    "            os.path.getmtime(str(file_path_object)),\n",
    "            os.path.getctime(str(file_path_object))\n",
    "        )\n",
    "    except FileNotFoundError:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hash_files_in_directory(hash_table: dict, dir_path: str, extra_details: bool = False):\n",
    "    \"\"\"Hashes all files in a directory.\"\"\"\n",
    "    \n",
    "    hashed = 0\n",
    "    duplicate = 0\n",
    "    \n",
    "    file_paths = sorted(Path(dir_path).glob('**/*'), key=get_earliest_age)\n",
    "    for file_path in file_paths:\n",
    "        if os.path.isdir(str(file_path)):\n",
    "            continue\n",
    "        \n",
    "        if hash_file(hash_table, file_path, dir_path, extra_details):\n",
    "            hashed += 1\n",
    "            print(f'Hashed: {file_path}')\n",
    "        else:\n",
    "            duplicate += 1\n",
    "            print(f'Hashed, Duplicate: {file_path}')\n",
    "    \n",
    "    print(f'\\n{hashed + duplicate} total files, {hashed} new hashes, {duplicate} duplicates')\n",
    "\n",
    "\n",
    "def hash_and_copy_files_in_directory(hash_table: dict, old_dir_path: str, new_dir_path: str, extra_details: bool = False):\n",
    "    \"\"\"Hashes all files in a directory.\"\"\"\n",
    "    \n",
    "    hashed = 0\n",
    "    duplicate = 0\n",
    "    failures = []\n",
    "    \n",
    "    file_paths = sorted(Path(old_dir_path).glob('**/*'), key=get_earliest_age)\n",
    "    for file_path in file_paths:\n",
    "        if os.path.isdir(str(file_path)):\n",
    "            continue\n",
    "        \n",
    "        try:\n",
    "            if hash_file(hash_table, file_path, old_dir_path, extra_details):\n",
    "                move_files_relative(old_dir_path, new_dir_path, str(file_path))\n",
    "                print(f'Hashed, Moved: {file_path}')\n",
    "                hashed += 1\n",
    "            else:\n",
    "                print(f'Hashed, Duplicate: {file_path}')\n",
    "                duplicate += 1\n",
    "        except FileNotFoundError:\n",
    "            print(f'FAILED: {file_path}')\n",
    "            failures.append(str(file_path))\n",
    "    \n",
    "    fail_count = len(failures)\n",
    "    print(f'\\n{hashed + duplicate + fail_count} total files, {hashed} moved, {duplicate} duplicates')\n",
    "    if failures:\n",
    "        print(f'\\n{fail_count} files failed:')\n",
    "        for fail in failures:\n",
    "            print(f'* {fail}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_paths(directory_path: str) -> str:\n",
    "    \"\"\"Remove trailing and leading slashes.\"\"\"\n",
    "    \n",
    "    return directory_path.strip('/').strip('\\\\')\n",
    "    \n",
    "\n",
    "def mark_files():\n",
    "    \"\"\"Add files in a directory to a hash table.\"\"\"\n",
    "    \n",
    "    hash_table_file = input('Hash table file: ')\n",
    "    source_directory = process_paths(input('Source directory: '))\n",
    "    \n",
    "    do_extra_details = input('Would you like to record earliest time of creation and all file paths? ')\n",
    "    extra_details = do_extra_details.lower()[0] == 'y'\n",
    "    \n",
    "    print('\\nProcessing...\\n')\n",
    "    \n",
    "    hash_table = load_hash_table(hash_table_file)\n",
    "    hash_files_in_directory(hash_table, source_directory, extra_details)\n",
    "    save_hash_table(hash_table_file, hash_table)\n",
    "    \n",
    "    print('\\nDone!')\n",
    "\n",
    "\n",
    "def copy_files():\n",
    "    \"\"\"Hash and copy unhashed files in a directory to another directory.\"\"\"\n",
    "\n",
    "    hash_table_file = input('Hash table file: ')\n",
    "    source_directory = process_paths(input('Source directory: '))\n",
    "    destination_directory = process_paths(input('Destination directory: '))\n",
    "    \n",
    "    do_extra_details = input('Would you like to record earliest time of creation and all file paths? ')\n",
    "    extra_details = do_extra_details.lower()[0] == 'y'\n",
    "    \n",
    "    print('\\nProcessing...\\n')\n",
    "    \n",
    "    hash_table = load_hash_table(hash_table_file)\n",
    "    hash_and_copy_files_in_directory(hash_table, source_directory, destination_directory, extra_details)\n",
    "    save_hash_table(hash_table_file, hash_table)\n",
    "    \n",
    "    print('\\nDone!')\n",
    "\n",
    "    \n",
    "def debug():\n",
    "    print('Hello worlds, we are in debug mode!  Stick code here to mess around with this notebook.')\n",
    "    print('If this isn\\'t what you wanted, just re-run this cell.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use absolute paths:\n",
    "\n",
    "```\n",
    "* C:\\My Stuff\\My Secret Stuff\n",
    "* D:\\Backups\\PC Backups\\1996\n",
    "```\n",
    "\n",
    "Or you can use relative paths:\n",
    "\n",
    "```\n",
    "* Temp Directory\n",
    "* ..\\Git Repos\\Some Other Project\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def main():\n",
    "    print('Copy files with no duplicates!')\n",
    "    print('Note: You can use relative or absolute paths.')\n",
    "    mode = input('Mode (MARK or COPY): ')\n",
    "    print('')\n",
    "    if mode.lower() == 'mark':\n",
    "        confirm = input('Are you sure you want to do MARK mode, and not COPY? ')\n",
    "        if confirm.lower()[0] == 'y':\n",
    "            mark_files()\n",
    "    elif mode.lower() == 'copy':\n",
    "        copy_files()\n",
    "    elif mode.lower() == 'debug':\n",
    "        debug()\n",
    "    else:\n",
    "        print('No action taken!')\n",
    "\n",
    "\n",
    "main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Documentation\n",
    "\n",
    "### Tips\n",
    "\n",
    "For important files, you may want to use absolute paths to avoid making mistakes.\n",
    "Note that this notebook will not modify your source directories unless you accidentally specify them as destination directories.\n",
    "\n",
    "If you find yourself often merging into an existing hash table file, you may want to back it up in case you make a mistake in the future and fill it with data from the wrong directory.\n",
    "\n",
    "\n",
    "### Motivation\n",
    "\n",
    "The goal of this notebook is to have two modes, such that multiple directories can be merged without duplicates within each directory or between any of them:\n",
    "\n",
    "1. Mark only mode: Given a directory, hash each file using MD5 and file size\n",
    "2. Copy mode: Same as mark mode, plus copy each file not in the hash table to another directory\n",
    "\n",
    "This requires four pieces of functionality:\n",
    "\n",
    "1. Serializing and deserializing a Python dict containing the data (hash and size)\n",
    "2. Iterating through files\n",
    "3. Hashing the file in question and extracting its size\n",
    "4. Copying files from one directory to another while maintaining their relative directory structure\n",
    "\n",
    "\n",
    "### Sources\n",
    "\n",
    "* `file_age_directory.py` script used in the inner workings of https://kieferand.co\n",
    "* https://stackoverflow.com/questions/8858008/how-to-move-a-file-in-python#8858026\n",
    "* https://stackoverflow.com/questions/1072569/see-if-two-files-have-the-same-content-in-python\n",
    "* https://stackoverflow.com/questions/5787471/md5-and-sha-2-collisions-in-python\n",
    "* https://stackoverflow.com/questions/3431825/generating-an-md5-checksum-of-a-file\n",
    "* https://stackoverflow.com/questions/6773584/how-is-pythons-glob-glob-ordered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}