softskill_ner_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6gH5aFmb55kr"
   },
   "source": [
    "# Fine-tune a pretrained 🤗 model for SoftSkill NER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iaIRjigt55ks"
   },
   "source": [
    "This notebook shows how to fine-tune custom NER model for soft skills using 🤗 Huggingface pretrained model [distilbert](thttps://huggingface.co/distilbert-base-uncased). \n",
    "\n",
    "We will use 🤗 Transformers [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) for this. it is the simplest way to fine-tune a 🤗 Transformer model. You can however choose to fine tune models using pytorch and tensorflow way which gives you flexibility to write your own custom training loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Y69s0RT455ko",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Transformers installation\n",
    "! pip install transformers datasets\n",
    "! pip install seqeval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mJkR2e8E55kt"
   },
   "source": [
    "## Load custom softskills dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NabEwdNa55ku"
   },
   "source": [
    "We will use a custom created, tokenized and annotated dataset for Softskill NER as such a dataset is not available in the open domain. \n",
    "\n",
    "There are only 119 sentences with one or more NERs annotated in each sentences. This dataset is good enough to run training and get the decent results however for production usecases it is advisable to compile more data depending upon type and number of NERs the model should be able to classify.\n",
    "\n",
    "Training data has some of the typical Softskills \"positive attitude\", \"leadership\", \"customer focus\" etc. you may want to take a look at 'raw_training_sentences.csv' and 'train_ner.json' in this repo to get good idea of how the custom NER training data has been prepared."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "HWMNRhmG55ku",
    "outputId": "1b627051-fe84-41b0-bdc9-0ae8580c0f98"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\ashut\\anaconda3\\envs\\skillner\\lib\\site-packages\\tqdm\\auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "Using custom data configuration default-e2cfc8a42dad81ca\n",
      "Reusing dataset json (C:\\Users\\ashut\\.cache\\huggingface\\datasets\\json\\default-e2cfc8a42dad81ca\\0.0.0\\ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b)\n",
      "100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 336.00it/s]\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "import torch\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "#Separate in train and test datasets\n",
    "data_files = {\"train\": \"./data/train_ner.json\", \"test\": \"./data/test_ner.json\"}\n",
    "\n",
    "#load custom NER dataset using Huggingface datasets liberary\n",
    "skillner = load_dataset('json', data_files=data_files)\n",
    "\n",
    "#lets define label names\n",
    "label_names = ['O','SoftSkill']\n",
    "\n",
    "#Lets set the device to cpu\n",
    "device =  torch.device('cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r2Ip9K7F55kv"
   },
   "source": [
    "we need a tokenizer to process the text and include a padding and truncation strategy to handle any variable sequence lengths. To process the dataset in one step, use 🤗 Datasets [`map`](https://huggingface.co/docs/datasets/process.html#map) method to apply a preprocessing function over the entire dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "nMxHed_f55kv"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at C:\\Users\\ashut\\.cache\\huggingface\\datasets\\json\\default-e2cfc8a42dad81ca\\0.0.0\\ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b\\cache-d0401a394c2ae949.arrow\n",
      "Loading cached processed dataset at C:\\Users\\ashut\\.cache\\huggingface\\datasets\\json\\default-e2cfc8a42dad81ca\\0.0.0\\ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b\\cache-8995eebec30ed675.arrow\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "#load distilbert tokenizer \n",
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\n",
    "\n",
    "# Our training data is already tokenized into words and labeled however we will need to \n",
    "# tokenize it to add special start and end tokens and furter tokenize like the way pretrained model was trained (wordPiece)\n",
    "# realign the labels with expended tokens\n",
    "def tokenize_and_align_labels(examples):\n",
    "    tokenized_inputs = tokenizer(examples[\"tokens\"], truncation=True, is_split_into_words=True)\n",
    "\n",
    "    labels = []\n",
    "    for i, label in enumerate(examples[f\"ner_tags\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.\n",
    "        previous_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids:  # Set the special tokens to -100.\n",
    "            if word_idx is None:\n",
    "                label_ids.append(-100)\n",
    "            elif word_idx != previous_word_idx:  # Only label the first token of a given word.\n",
    "                label_ids.append(label[word_idx])\n",
    "            else:\n",
    "                label_ids.append(-100)\n",
    "            previous_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "\n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs\n",
    "\n",
    "#Apply the tokenization and alignment to all the rows of training data in one go\n",
    "tokenized_skillner = skillner.map(tokenize_and_align_labels, batched=True)\n",
    "\n",
    "\n",
    "from transformers import DataCollatorForTokenClassification\n",
    "\n",
    "#Batch the data for training \n",
    "data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D7LPR9HI55kx"
   },
   "source": [
    "## Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lrpINyuW55kx"
   },
   "source": [
    "🤗 Transformers provides a [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) API supports a wide range of training options and features such as logging, gradient accumulation, and mixed precision.\n",
    "\n",
    "Let's load the model and number of labels. Since we are only classifying 1) softskill or 2) other than softskill Token, we will have only two labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "a5WZF9pP55ky"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_transform.bias', 'vocab_transform.weight', 'vocab_projector.bias']\n",
      "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForTokenClassification\n",
    "\n",
    "#load distilbert base uncased model to be used for token classification. Number of labels = 2 as we are only recognizing Softskill token as 1 and rest all as 0\n",
    "model = AutoModelForTokenClassification.from_pretrained(\"distilbert-base-uncased\", num_labels=2).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H_2Ok1tr55ky"
   },
   "source": [
    "<Tip>\n",
    "\n",
    "You will see a warning about some of the pretrained weights not being used and some weights being randomly\n",
    "initialized. Don't worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. we will fine-tune this new model head on token classification task, transferring the knowledge of the pretrained model to it.\n",
    "\n",
    "</Tip>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y99_OzGu55ky"
   },
   "source": [
    "### Training hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ir5MLHqu55ky"
   },
   "source": [
    "Next, let's create a [TrainingArguments](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments) class which contains all the hyperparameters we can tune as well as flags for activating different training options. For this tutorial we can start with the default training [hyperparameters](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments), but feel free to experiment with these to find your optimal settings.\n",
    "\n",
    "Specify where to save the checkpoints from this training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "UF6vVNnh55ky"
   },
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    "\n",
    "#default training args\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=\"./skillner_model\",\n",
    "    evaluation_strategy=\"epoch\",\n",
    "    learning_rate=2e-5,\n",
    "    per_device_train_batch_size=10,\n",
    "    per_device_eval_batch_size=10,\n",
    "    num_train_epochs=10,\n",
    "    weight_decay=0.01,\n",
    "    logging_dir='./logs',\n",
    "    logging_steps=10,    \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zTUy2nxu55ky"
   },
   "source": [
    "### Metrics config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m3kXNfve55kz"
   },
   "source": [
    "[Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) does not automatically evaluate model performance during training. we will need to pass [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) a function to compute and report metrics. The 🤗 Datasets library provides a simple [`accuracy`](https://huggingface.co/metrics/accuracy) function we can load with the `load_metric` (see this [tutorial](https://huggingface.co/docs/datasets/metrics.html) for more information) function:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C_t6Ngu555kz"
   },
   "source": [
    "Call `compute` on `metric` to calculate the accuracy of your predictions. Before passing your predictions to `compute`, you need to convert the predictions to logits (remember all 🤗 Transformers models return logits):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "anpL1URQ55kz"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from datasets import load_metric\n",
    "\n",
    "metric = load_metric(\"seqeval\")\n",
    "\n",
    "\n",
    "def compute_metrics(eval_preds):\n",
    "    logits, labels = eval_preds\n",
    "    predictions = np.argmax(logits, axis=-1)\n",
    "\n",
    "    # Remove ignored index (special tokens) and convert to labels\n",
    "    true_labels = [[label_names[l] for l in label if l != -100] for label in labels]\n",
    "    true_predictions = [\n",
    "        [label_names[p] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    all_metrics = metric.compute(predictions=true_predictions, references=true_labels)\n",
    "    return {\n",
    "        \"precision\": all_metrics[\"overall_precision\"],\n",
    "        \"recall\": all_metrics[\"overall_recall\"],\n",
    "        \"f1\": all_metrics[\"overall_f1\"],\n",
    "        \"accuracy\": all_metrics[\"overall_accuracy\"],\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wBLuhzuM55kz"
   },
   "source": [
    "### Trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GsA_sitL55k0"
   },
   "source": [
    "Create a [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) object with your model, training arguments, training and test datasets, and evaluation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "qSPluo7k55k0"
   },
   "outputs": [],
   "source": [
    "from transformers import Trainer\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_skillner[\"train\"],\n",
    "    eval_dataset=tokenized_skillner[\"test\"],\n",
    "    tokenizer=tokenizer,\n",
    "    data_collator=data_collator,\n",
    "    compute_metrics=compute_metrics,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S1mNaAVY55k0"
   },
   "source": [
    "Then fine-tune your model by calling [train()](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.train):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "P8DSF_v455k0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The following columns in the training set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "C:\\Users\\ashut\\anaconda3\\envs\\skillner\\lib\\site-packages\\transformers\\optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n",
      "***** Running training *****\n",
      "  Num examples = 101\n",
      "  Num Epochs = 10\n",
      "  Instantaneous batch size per device = 10\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 10\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 110\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='110' max='110' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [110/110 00:15, Epoch 10/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.549900</td>\n",
       "      <td>0.449371</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.820628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.399400</td>\n",
       "      <td>0.361132</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.820628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.319400</td>\n",
       "      <td>0.301087</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.177778</td>\n",
       "      <td>0.834081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.239700</td>\n",
       "      <td>0.243414</td>\n",
       "      <td>0.785714</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>0.647059</td>\n",
       "      <td>0.892377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.183600</td>\n",
       "      <td>0.211946</td>\n",
       "      <td>0.828571</td>\n",
       "      <td>0.725000</td>\n",
       "      <td>0.773333</td>\n",
       "      <td>0.923767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.138300</td>\n",
       "      <td>0.193942</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.779221</td>\n",
       "      <td>0.923767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.102100</td>\n",
       "      <td>0.183426</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.779221</td>\n",
       "      <td>0.923767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.087000</td>\n",
       "      <td>0.176324</td>\n",
       "      <td>0.794872</td>\n",
       "      <td>0.775000</td>\n",
       "      <td>0.784810</td>\n",
       "      <td>0.923767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.077300</td>\n",
       "      <td>0.178077</td>\n",
       "      <td>0.810811</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.779221</td>\n",
       "      <td>0.923767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.068300</td>\n",
       "      <td>0.176858</td>\n",
       "      <td>0.789474</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.769231</td>\n",
       "      <td>0.919283</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "C:\\Users\\ashut\\anaconda3\\envs\\skillner\\lib\\site-packages\\seqeval\\metrics\\sequence_labeling.py:171: UserWarning: SoftSkill seems not to be NE tag.\n",
      "  warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
      "C:\\Users\\ashut\\anaconda3\\envs\\skillner\\lib\\site-packages\\seqeval\\metrics\\v1.py:57: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "The following columns in the evaluation set  don't have a corresponding argument in `DistilBertForTokenClassification.forward` and have been ignored: ner_tags, tokens, id. If ner_tags, tokens, id are not expected by `DistilBertForTokenClassification.forward`,  you can safely ignore this message.\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 19\n",
      "  Batch size = 10\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=110, training_loss=0.20243914885954423, metrics={'train_runtime': 17.7708, 'train_samples_per_second': 56.835, 'train_steps_per_second': 6.19, 'total_flos': 5427972833556.0, 'train_loss': 0.20243914885954423, 'epoch': 10.0})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets save the checkpoints for fine tuned NER model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./skillner_model\n",
      "Configuration saved in ./skillner_model\\config.json\n",
      "Model weights saved in ./skillner_model\\pytorch_model.bin\n",
      "tokenizer config file saved in ./skillner_model\\tokenizer_config.json\n",
      "Special tokens file saved in ./skillner_model\\special_tokens_map.json\n"
     ]
    }
   ],
   "source": [
    "#save the model in \"skillner_model\" directory\n",
    "trainer.save_model() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets check our fine tuned model inference with a sample sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'entity': 'LABEL_0',\n",
       "  'score': 0.9943357,\n",
       "  'index': 1,\n",
       "  'word': 'john',\n",
       "  'start': 0,\n",
       "  'end': 4},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.9899693,\n",
       "  'index': 2,\n",
       "  'word': 'doe',\n",
       "  'start': 5,\n",
       "  'end': 8},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.9952485,\n",
       "  'index': 3,\n",
       "  'word': 'is',\n",
       "  'start': 9,\n",
       "  'end': 11},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.99341005,\n",
       "  'index': 4,\n",
       "  'word': 'known',\n",
       "  'start': 12,\n",
       "  'end': 17},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.99029845,\n",
       "  'index': 5,\n",
       "  'word': 'to',\n",
       "  'start': 18,\n",
       "  'end': 20},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.9671793,\n",
       "  'index': 6,\n",
       "  'word': 'be',\n",
       "  'start': 21,\n",
       "  'end': 23},\n",
       " {'entity': 'LABEL_1',\n",
       "  'score': 0.9086044,\n",
       "  'index': 7,\n",
       "  'word': 'composed',\n",
       "  'start': 24,\n",
       "  'end': 32},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.98399967,\n",
       "  'index': 8,\n",
       "  'word': 'and',\n",
       "  'start': 33,\n",
       "  'end': 36},\n",
       " {'entity': 'LABEL_1',\n",
       "  'score': 0.8795789,\n",
       "  'index': 9,\n",
       "  'word': 'confident',\n",
       "  'start': 37,\n",
       "  'end': 46},\n",
       " {'entity': 'LABEL_1',\n",
       "  'score': 0.5955358,\n",
       "  'index': 10,\n",
       "  'word': 'professional',\n",
       "  'start': 47,\n",
       "  'end': 59},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.9862076,\n",
       "  'index': 11,\n",
       "  'word': 'at',\n",
       "  'start': 60,\n",
       "  'end': 62},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.9948212,\n",
       "  'index': 12,\n",
       "  'word': 'work',\n",
       "  'start': 63,\n",
       "  'end': 67},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.99487203,\n",
       "  'index': 13,\n",
       "  'word': ',',\n",
       "  'start': 67,\n",
       "  'end': 68},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.99508435,\n",
       "  'index': 14,\n",
       "  'word': 'he',\n",
       "  'start': 69,\n",
       "  'end': 71},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.99558073,\n",
       "  'index': 15,\n",
       "  'word': 'also',\n",
       "  'start': 72,\n",
       "  'end': 76},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.99546534,\n",
       "  'index': 16,\n",
       "  'word': 'has',\n",
       "  'start': 77,\n",
       "  'end': 80},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.99142504,\n",
       "  'index': 17,\n",
       "  'word': 'strong',\n",
       "  'start': 81,\n",
       "  'end': 87},\n",
       " {'entity': 'LABEL_1',\n",
       "  'score': 0.79402506,\n",
       "  'index': 18,\n",
       "  'word': 'leadership',\n",
       "  'start': 88,\n",
       "  'end': 98},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.98130417,\n",
       "  'index': 19,\n",
       "  'word': 'qualities',\n",
       "  'start': 99,\n",
       "  'end': 108},\n",
       " {'entity': 'LABEL_0',\n",
       "  'score': 0.988927,\n",
       "  'index': 20,\n",
       "  'word': '.',\n",
       "  'start': 108,\n",
       "  'end': 109}]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "getNER = pipeline(\"ner\", model=model.to(device), tokenizer=tokenizer)\n",
    "example = \"John Doe is known to be composed and confident professional at work, he also has strong leadership qualities.\"\n",
    "\n",
    "ner_results = getNER(example)\n",
    "\n",
    "ner_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets parse the results and check if the model has correctly identified the soft skills. There could be a better way to map the label ids to the text labels which is #ToDo "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NER =  composed\n",
      "NER =  confident\n",
      "NER =  professional\n",
      "NER =  leadership\n"
     ]
    }
   ],
   "source": [
    "for key in ner_results:\n",
    "        if(key['entity'] ==\"LABEL_1\"):\n",
    "            print(\"NER = \",key['word'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "name": "training.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "skillner",
   "language": "python",
   "name": "skillner"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}