Created using Colaboratory

Author: Animeshcoder

Python_MySQL_P6.ipynb

@@ -0,0 +1,136 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "authorship_tag": "ABX9TyOqp9cUjqipp0bbLoFB3F7n",
+      "include_colab_link": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "view-in-github",
+        "colab_type": "text"
+      },
+      "source": [
+        "<a href=\"https://colab.research.google.com/github/Animeshcoder/MySQL-Python/blob/main/Python_MySQL_P6.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **Introduction:**\n",
+        "This project is a Python script that demonstrates how to identify and remove duplicate rows from a MySQL database table. The script defines a list of SQL statements that are used to perform this task in three steps. The first step uses a self-join to identify rows in the table that have the same values for the Name, Phone No, and Age columns. The second step updates NULL values in one row with values from the other row. The third step removes duplicate rows from the table. This project can serve as a starting point for building more complex data processing pipelines that involve cleaning and deduplicating data in a MySQL database.\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "cstqU4qOs9_7"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### **Steps Involved:**\n",
+        "\n",
+        "**Step 1: Connect to the database**\n",
+        "\n",
+        "The first part of the script sets up the connection details for the MySQL database and creates a connection object using PyMySQL’s connect function. The user, password, host, and database arguments are passed to this function as a dictionary to specify the connection details.\n",
+        "\n",
+        "**Step 2: Define SQL statements**\n",
+        "\n",
+        "Next, a list of SQL statements is defined that will be used to identify and remove duplicate rows from the table. The first statement is a SELECT statement that uses a self-join to identify rows in the table that have the same values for the Name, Phone No, and Age columns. The second statement is an UPDATE statement that updates NULL values in one row with values from the other row. The third statement is a DELETE statement that removes duplicate rows from the table.\n",
+        "\n",
+        "**Step 3: Execute SQL statements**\n",
+        "\n",
+        "A cursor object is created using the cursor method of the connection object. This cursor is used to execute the first SQL statement in the list and fetch all rows of the result set. The remaining SQL statements in the list are then executed one by one using a loop.\n",
+        "\n",
+        "**Step 4: Commit changes and close connection**\n",
+        "\n",
+        "After all SQL statements have been executed, any changes made to the database are committed by calling the connection object’s commit method. Finally, both cursor and connection objects are closed by calling their respective close methods."
+      ],
+      "metadata": {
+        "id": "w_hb5MRXSAWL"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "34Oi4IcrslSU"
+      },
+      "outputs": [],
+      "source": [
+        "import pymysql\n",
+        "\n",
+        "# Replace these values with your MySQL connection details\n",
+        "db_config = {\n",
+        "    'user': 'youruser',\n",
+        "    'password': 'yourpassword@123',\n",
+        "    'host': 'yourhost',\n",
+        "    'database': 'yourdatabasename'\n",
+        "}\n",
+        "\n",
+        "# Connect to the MySQL database\n",
+        "cnx = pymysql.connect(**db_config)\n",
+        "cursor = cnx.cursor()\n",
+        "\n",
+        "# Define the SQL statements\n",
+        "sql_statements = [\n",
+        "    \"\"\"\n",
+        "    -- Step 1: Identify matching rows using a self-join\n",
+        "    SELECT t1.*, t2.*\n",
+        "    FROM table t1\n",
+        "    JOIN table t2\n",
+        "    ON t1.`Name` = t2.`Name` AND t1.`Phone No` = t2.`Phone No` AND t1.`Age` = t2.`Age`\n",
+        "    WHERE t1.id < t2.id;\n",
+        "    \"\"\",\n",
+        "    \"\"\"\n",
+        "    -- Step 2: Update NULL values in one row with values from the other row\n",
+        "    UPDATE table t1\n",
+        "    JOIN table t2\n",
+        "    ON t1.`Name` = t2.`Name` AND t1.`Phone No` = t2.`Phone No` AND t1.`Age` = t2.`Age`\n",
+        "    SET t1.`Date of Birth` = COALESCE(t1.`Date of Birth`, t2.`Date of Birth`),\n",
+        "    WHERE t1.id < t2.id;\n",
+        "\n",
+        "\"\"\",\n",
+        "\"\"\"\n",
+        "-- Step 3: Delete duplicate rows from the table\n",
+        "DELETE FROM ntable\n",
+        "WHERE id NOT IN (\n",
+        "SELECT * FROM (\n",
+        "SELECT MIN(id)\n",
+        "FROM ntable\n",
+        "GROUP BY `Name`, `Phone No`, `Age`\n",
+        ") AS x\n",
+        ");\n",
+        "\"\"\"\n",
+        "]\n",
+        "\n",
+        "# Execute the first SQL statement and fetch all rows of the result set\n",
+        "cursor.execute(sql_statements[0])\n",
+        "rows = cursor.fetchall()\n",
+        "\n",
+        "# Execute the remaining SQL statements one by one\n",
+        "for sql in sql_statements[1:]:\n",
+        "    cursor.execute(sql)\n",
+        "\n",
+        "# Commit the changes to the database\n",
+        "cnx.commit()\n",
+        "\n",
+        "# Close the cursor and connection\n",
+        "cursor.close()\n",
+        "cnx.close()\n"
+      ]
+    }
+  ]
+}