DA08_BaiTap12345_An.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BÀI TẬP NUMPY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Đề Bài"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thực hiện các yêu cầu sau với dữ liệu 'vn_housing_dataset.csv':\n",
    "1. Lấy ra thông tin từ các cột diện tích, chiều dài, chiều rộng, giá nhà.\n",
    "2. Kiểm tra các thông tin bất thường, loại bỏ nếu thấy bất hợp lý.\n",
    "3. Chuẩn hóa thông tin (ví dụ: giá nhà với đơn vị \"tỷ\" thì đổi hết sang đơn vị \"triệu\", nếu có).\n",
    "4. Lấy thêm thông tin từ cột ngày, format lại thành \"Ngày_Tháng_Năm\"\n",
    "5. Lưu lại dữ liệu đã làm sạch, chuẩn hóa, loại bỏ bất thường… vào file csv khác"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pandas import Timestamp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "US_L = 'm'  # standard unit of length\n",
    "US_S = 'm²'  # standard unit of area\n",
    "US_PR = 'triệu/m²'  # standard unit of price\n",
    "UL_PR = 'tỷ/m²'  # large unit of length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "CN_S = 'Diện tích'  # area column name\n",
    "CN_L = 'Dài'  # length column name\n",
    "CN_W = 'Rộng'  # width column name\n",
    "CN_PR = 'Giá'  # price column name\n",
    "CN_PR_SRC = 'Giá/m2'  # price column name (source)\n",
    "CN_DT = 'Ngày'  # price column name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "FMT_DT_SRC = '%Y-%m-%d'  # format datime source\n",
    "FMT_DT_DST = '%d_%m_%Y'  # format datime destination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# String split\n",
    "def s_splt(s):\n",
    "    return s.str.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# String len\n",
    "def s_len(s):\n",
    "    return s.str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# String lower\n",
    "def s_low(s):\n",
    "    return s.str.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Header\n",
    "def hdr(s):\n",
    "    return s.split()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Footer\n",
    "def ftr(s):\n",
    "    return s.split()[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connector\n",
    "def con(s):\n",
    "    return s.str.join(' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Refactor\n",
    "def refactor(s):\n",
    "    return con(s_splt(s.str.strip()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try parse to number\n",
    "def try_prs_num(s):\n",
    "    return pd.to_numeric(s.str[0], errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try parse to datetime\n",
    "def try_prs_dt(s):\n",
    "    return pd.to_datetime(s, format=FMT_DT_SRC, errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try parse value\n",
    "def try_prs_val(s):\n",
    "    return s_splt(s).str[0].astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Round value and add standard unit\n",
    "def rnd_unit(val, unit):\n",
    "    return val.round(2).astype(str) + f' {unit}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Round value and add standard unit advanced\n",
    "def rnd_unit_adv(val, unit):\n",
    "    return s_splt(val).str[0].astype(float).round(2).astype(str).apply(lambda val: f'{val} {unit}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('vn_housing_dataset.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request 1\n",
    "length = refactor(df[CN_L])\n",
    "width = refactor(df[CN_W])\n",
    "area = refactor(df[CN_S])\n",
    "price = refactor(df[CN_PR_SRC])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request 2\n",
    "length_split = s_splt(length)\n",
    "width_split = s_splt(width)\n",
    "area_split = s_splt(area)\n",
    "price_split = s_splt(price)\n",
    "\n",
    "length_value = try_prs_num(length_split)\n",
    "width_value = try_prs_num(width_split)\n",
    "area_value = try_prs_num(area_split)\n",
    "price_value = try_prs_num(price_split)\n",
    "\n",
    "valids = (length_value.notna()\n",
    "              & width_value.notna()\n",
    "              & area_value.notna()\n",
    "              & price_value.notna()\n",
    "              & length_value.gt(0)\n",
    "              & width_value.gt(0)\n",
    "              & area_value.gt(0)\n",
    "              & price_value.gt(0)\n",
    "              & s_len(length_split).gt(1)\n",
    "              & s_len(width_split).gt(1)\n",
    "              & s_len(area_split).gt(1)\n",
    "              & s_len(price_split).gt(1)\n",
    "              & (np.ceil(length_value) * np.ceil(width_value)).ge(np.floor(area_value)))\n",
    "df_valid = df.loc[valids]\n",
    "\n",
    "length_valid = rnd_unit(length_value.loc[valids], US_L)\n",
    "width_valid = rnd_unit(width_value.loc[valids], US_L)\n",
    "area_valid = rnd_unit(area_value.loc[valids], US_S)\n",
    "price_valid = df[CN_PR_SRC].loc[valids]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request 3\n",
    "price_valid = rnd_unit_adv(con(s_low(price_valid).apply(lambda x: [str(float(hdr(x)) * 1000) if ftr(x) == UL_PR else hdr(x), ftr(x)])), US_PR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request 4\n",
    "df_date = df.loc[valids][CN_DT]\n",
    "date_value = try_prs_dt(df_date)\n",
    "\n",
    "valids &= (date_value.notna()) & date_value.lt(Timestamp.today())\n",
    "invalids = ~valids\n",
    "\n",
    "length_valid = length_valid.loc[valids]\n",
    "width_valid = width_valid.loc[valids]\n",
    "area_valid = area_valid.loc[valids]\n",
    "price_valid = price_valid.loc[valids]\n",
    "date_valid = pd.to_datetime(date_value.loc[valids]).dt.strftime(FMT_DT_DST)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request 5\n",
    "pd.concat([date_valid, length_valid, width_valid, area_valid, price_valid], axis=1).to_csv('data_valid.csv')\n",
    "df.loc[invalids].to_csv('data_invalid.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}