03NamedEntityRecognition.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "03NamedEntityRecognition.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XkG3HwP_bQq3",
        "colab_type": "text"
      },
      "source": [
        "# 1) Basics of Named Entity Recognition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mSyz-NMSbVNw",
        "colab_type": "text"
      },
      "source": [
        "Named Entity Recognition is a subtask of information extraction that classify named entities into pre-defined categories such as names of persons, organizations, locations\n",
        "\n",
        "spaCy features an extremely fast statistical entity recognition system, that assigns labels to contiguous spans of tokens\n",
        "\n",
        "The default model identifies a variety of named and numeric entities, including companies, locations, organizations and products"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6leZUM06YGBq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# officaial documentation \n",
        "# https://spacy.io/usage/linguistic-features/#named-entities\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hLaxb0FrUgk0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import spaCy\n",
        "import spacy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CLgvvWbc_EMS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# load the English language library\n",
        "nlp = spacy.load(name='en_core_web_sm')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LkQthSvc_FX5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create a simple doc object\n",
        "doc = nlp(\"Apple is looking at buying U.K. startup for $1 billion\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6EFOdItIVH7a",
        "colab_type": "code",
        "outputId": "d8cdb53f-9946-4647-89db-0bc81f582ce8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "for ent in doc.ents:\n",
        "    print(ent.text, ent.start_char, ent.end_char, ent.label_, str(spacy.explain(ent.label_)))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Apple 0 5 ORG Companies, agencies, institutions, etc.\n",
            "U.K. 27 31 GPE Countries, cities, states\n",
            "$1 billion 44 54 MONEY Monetary values, including unit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jx3tVFL2XmVz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create another doc object\n",
        "doc_2 = nlp(\"San Francisco considers banning sidewalk delivery robots\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HNuFuvSLXvxr",
        "colab_type": "code",
        "outputId": "ee88b7ef-9a88-4584-c59d-8de00064e31b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "for ent in doc_2.ents:\n",
        "    print(ent.text, ent.start_char, ent.end_char, ent.label_, str(spacy.explain(ent.label_)))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "San Francisco 0 13 GPE Countries, cities, states\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OtQdLE7QYY3Z",
        "colab_type": "text"
      },
      "source": [
        "# 2) Adding Named Entity to Span"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oPAoXWlRYbd5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc_3 = nlp(\"facebook is hiring a new vice president in U.S.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sqPQStshZCnh",
        "colab_type": "code",
        "outputId": "78f781a1-87a9-4dc3-ae8c-fd6dbf7f74cf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "for ent in doc_3.ents:\n",
        "    print(ent.text, ent.label_, str(spacy.explain(ent.label_)))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "U.S. GPE Countries, cities, states\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bxCtW38MZxgT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# we will add Facebook as Named Entity as a company"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HIfjy4wNaN_u",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from spacy.tokens import Span"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cVUb4_y6aSdI",
        "colab_type": "code",
        "outputId": "498c2c72-48b6-49ed-92be-cf963058c42b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "# Get the hash value of ORG entity label\n",
        "ORG = doc_3.vocab.strings['ORG']\n",
        "print(ORG)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "383\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jn_mWXC4aUZv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create a Span for new entity\n",
        "new_ent = Span(doc_3, 0, 1, label=ORG)\n",
        "# Index locations from 0 to 1 (excludes 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sc9WJmAFaViZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Add the entity to the existing Doc object\n",
        "doc_3.ents = list(doc_3.ents) + [new_ent]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VmwR3Qi8a76t",
        "colab_type": "code",
        "outputId": "93e2dbfa-de14-4a34-dd60-ca0d1cf39017",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "for ent in doc_3.ents:\n",
        "    print(ent.text, ent.label_, str(spacy.explain(ent.label_)))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "facebook ORG Companies, agencies, institutions, etc.\n",
            "U.S. GPE Countries, cities, states\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ojSF7HyU5O3I",
        "colab_type": "text"
      },
      "source": [
        "# 3) Visualizing Named Entities"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C3T03Lqv5QfX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import spaCy\n",
        "import spacy\n",
        "# load the English language library\n",
        "nlp = spacy.load(name='en_core_web_sm')\n",
        "# Import the displaCy library\n",
        "from spacy import displacy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GhzHpYhY5T80",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc = nlp(\"Apple is looking at buying U.K. startup for $1 billion\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KECwGAYs5aWI",
        "colab_type": "code",
        "outputId": "99dcbbf3-d88a-4022-b319-6702e7c13cd8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "displacy.render(docs=doc,style='ent',jupyter=True)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " is looking at buying \n",
              "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    U.K.\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
              "</mark>\n",
              " startup for \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    $1 billion\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
              "</mark>\n",
              "</div></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pJEm7ewU5a1i",
        "colab_type": "code",
        "outputId": "5a51a712-da72-43a5-c7e4-1a8776d1f123",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "# Viewing Specific Entities\n",
        "options = {'ents': ['ORG', 'MONEY']}\n",
        "displacy.render(docs=doc,style='ent',jupyter=True,options=options)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " is looking at buying U.K. startup for \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    $1 billion\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
              "</mark>\n",
              "</div></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}