LM-ontology-concept-placement.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## LM-ontology-concept-placement\n",
        "### A Guide for Data Processing and Running the Project\n",
        "\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/KRR-Oxford/LM-ontology-concept-placement/blob/main/LM-ontology-concept-placement.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>\n",
        "\n",
        "This Google Colab notebook provides a quick guide to download data from Zenodo and process the data into correct folder names to run the provided bash (.sh) scripts.\n",
        "\n",
        "Set the runtime to be T4 GPU.  \n"
      ],
      "metadata": {
        "id": "c81Oqo1GrQuT"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A4eYr7ObEu6_",
        "outputId": "48ac435f-7625-40e5-dd20-22ce601a7721"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "rm: cannot remove 'LM-ontology-concept-placement': No such file or directory\n",
            "Cloning into 'LM-ontology-concept-placement'...\n",
            "remote: Enumerating objects: 255, done.\u001b[K\n",
            "remote: Counting objects: 100% (255/255), done.\u001b[K\n",
            "remote: Compressing objects: 100% (216/216), done.\u001b[K\n",
            "remote: Total 255 (delta 82), reused 185 (delta 39), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (255/255), 446.26 KiB | 1.72 MiB/s, done.\n",
            "Resolving deltas: 100% (82/82), done.\n"
          ]
        }
      ],
      "source": [
        "!rm -r LM-ontology-concept-placement # remove if existed previously\n",
        "!git clone https://github.com/KRR-Oxford/LM-ontology-concept-placement.git"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jfpGz4TyUYc4"
      },
      "source": [
        "Data download from Zenodo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JU-CoVt1Jbti",
        "outputId": "83dbb96a-bbe2-4107-84ae-e880dd458980"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting zenodo_get\n",
            "  Downloading zenodo_get-1.6.1-py2.py3-none-any.whl.metadata (1.1 kB)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from zenodo_get) (2.31.0)\n",
            "Collecting wget (from zenodo_get)\n",
            "  Downloading wget-3.2.zip (10 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: humanize in /usr/local/lib/python3.10/dist-packages (from zenodo_get) (4.10.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->zenodo_get) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->zenodo_get) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->zenodo_get) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->zenodo_get) (2024.7.4)\n",
            "Downloading zenodo_get-1.6.1-py2.py3-none-any.whl (17 kB)\n",
            "Building wheels for collected packages: wget\n",
            "  Building wheel for wget (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9656 sha256=00066123fb200faa81d3e6520aa7bb50950954151d6c042af5055c81d20a82ee\n",
            "  Stored in directory: /root/.cache/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769\n",
            "Successfully built wget\n",
            "Installing collected packages: wget, zenodo_get\n",
            "Successfully installed wget-3.2 zenodo_get-1.6.1\n",
            "Title: Ontology Enrichment from Texts (OET): A Biomedical Dataset for Concept Discovery and Placement\n",
            "Keywords: ontology learning, biomedical ontologies, text mining, entity linking, concept placement, SNOMED CT, ontology enrichment, ontology engineering\n",
            "Publication date: 2023-12-26\n",
            "DOI: 10.5281/zenodo.10432003\n",
            "Total size: 595.4 MB\n",
            "\n",
            "Link: https://zenodo.org/records/10432003/files/OET-data-ver4.zip   size: 595.4 MB\n",
            "\n",
            "Checksum is correct. (d3a4312627b6d8b86c1391281b49e3b6)\n",
            "All files have been downloaded.\n"
          ]
        }
      ],
      "source": [
        "!pip install zenodo_get # install zenodo\n",
        "!zenodo_get 10432003 # get dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-BgMiKIMUPkW"
      },
      "source": [
        "Data unzip to the data folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "118NyE3QLzDr",
        "outputId": "936e2674-00df-4d76-af60-523e53237b84"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "rm: cannot remove 'LM-ontology-concept-placement/data': No such file or directory\n",
            "Archive:  OET-data-ver4.zip\n",
            "  inflating: LM-ontology-concept-placement/data/readme.md  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/._readme.md  \n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-Disease/\n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/._MM-S14-Disease  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/._.DS_Store  \n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/\n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/\n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)/\n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-Disease/ontology/\n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/test-in-KB.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._test-in-KB.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/test-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._test-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/test-NIL-all-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._test-NIL-all-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/valid-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._valid-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/test-NIL-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._test-NIL-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/train-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._train-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/valid-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._valid-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/test-in-KB-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._test-in-KB-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level/valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-edge-pair-level/._valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(out-of-KB-discovery)/._.DS_Store  \n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_full/\n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/\n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)/train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(concept-placement)/._train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)/test-in-KB.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(concept-placement)/._test-in-KB.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(concept-placement)/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)/test-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(concept-placement)/._test-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)/test-NIL-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(concept-placement)/._test-NIL-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)/valid-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(concept-placement)/._valid-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)/valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(concept-placement)/._valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/ontology/SNOMEDCT-US-20140901-Disease-final.owl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/ontology/._SNOMEDCT-US-20140901-Disease-final.owl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/ontology/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/ontology/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/ontology/SNOMEDCT-US-20140901-Disease-edges-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/ontology/._SNOMEDCT-US-20140901-Disease-edges-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/ontology/SNOMEDCT-US-20140901-Disease-edges-atomic.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/ontology/._SNOMEDCT-US-20140901-Disease-edges-atomic.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/ontology/SNOMEDCT-US-20140901-Disease_syn_attr_hyp-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/ontology/._SNOMEDCT-US-20140901-Disease_syn_attr_hyp-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/ontology/SNOMEDCT-US-20140901-Disease_with_NIL_syn_attr_hyp-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/ontology/._SNOMEDCT-US-20140901-Disease_with_NIL_syn_attr_hyp-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/ontology/SNOMEDCT-US-20170301-Disease-final.owl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/ontology/._SNOMEDCT-US-20170301-Disease-final.owl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_full/train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_full/._train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_full/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_full/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/._train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/test.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/._test.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr/._valid.jsonl  \n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-CPP/\n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/._.DS_Store  \n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/\n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/\n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)/\n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-CPP/ontology/\n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/test-in-KB.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._test-in-KB.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/test-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._test-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/test-NIL-all-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._test-NIL-all-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/valid-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._valid-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/test-NIL-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._test-NIL-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/train-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._train-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/valid-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._valid-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/test-in-KB-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._test-in-KB-complex.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level/valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-edge-pair-level/._valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(out-of-KB-discovery)/._.DS_Store  \n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_full/\n",
            "   creating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/\n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)/train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(concept-placement)/._train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)/test-in-KB.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(concept-placement)/._test-in-KB.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(concept-placement)/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)/test-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(concept-placement)/._test-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)/test-NIL-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(concept-placement)/._test-NIL-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)/valid-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(concept-placement)/._valid-NIL.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)/valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(concept-placement)/._valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/ontology/SNOMEDCT-US-20140901-CPP_with_NIL_syn_attr_hyp-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/ontology/._SNOMEDCT-US-20140901-CPP_with_NIL_syn_attr_hyp-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/ontology/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/ontology/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/ontology/SNOMEDCT-US-20140901-CPP-final.owl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/ontology/._SNOMEDCT-US-20140901-CPP-final.owl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/ontology/SNOMEDCT-US-20170301-CPP-final.owl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/ontology/._SNOMEDCT-US-20170301-CPP-final.owl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/ontology/SNOMEDCT-US-20140901-CPP-edges-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/ontology/._SNOMEDCT-US-20140901-CPP-edges-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/ontology/SNOMEDCT-US-20140901-CPP-edges-atomic.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/ontology/._SNOMEDCT-US-20140901-CPP-edges-atomic.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/ontology/SNOMEDCT-US-20140901-CPP_syn_attr_hyp-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/ontology/._SNOMEDCT-US-20140901-CPP_syn_attr_hyp-all.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_full/train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_full/._train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_full/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_full/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/._train.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/._.DS_Store  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/test.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/._test.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/valid.jsonl  \n",
            "  inflating: LM-ontology-concept-placement/data/__MACOSX/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr/._valid.jsonl  \n"
          ]
        }
      ],
      "source": [
        "!rm -r LM-ontology-concept-placement/data # remove if existed previously\n",
        "!mkdir LM-ontology-concept-placement/data\n",
        "!unzip OET-data-ver4.zip -d LM-ontology-concept-placement/data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FoZlG0t4OuLp"
      },
      "source": [
        "**Data name and folder processing**\n",
        "\n",
        "Note that we renamed the sub-folder names after running the scripts for easier understanding. The original sub-folder names are below (see `readme.md` in the data folder):\n",
        "\n",
        "* st21pv_syn_attr-all-complexEdge-filt -> mention-level-(out-of-KB-discovery)/syn_attr\n",
        "\n",
        "* st21pv_syn_full-all-complexEdge-filt -> mention-level-(out-of-KB-discovery)/syn_full\n",
        "\n",
        "* st21pv_syn_attr-all-complexEdge-unsup -> mention-level-(concept-placement)\n",
        "\n",
        "* st21pv_syn_attr-all-complexEdge-edges-final -> mention-edge-pair-level\n",
        "\n",
        "Then rename and replace the folders of `CPP` and `Disease` under `data/MedMentions-preprocessed+`.\n",
        "\n",
        "Then moving ontology files into the `ontologies` folder.\n",
        "\n",
        "Finally create a sample validation in-KB data file of random 100 data lines (which is need for debugging mode in inference)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "crDy1d_8O3sl"
      },
      "outputs": [],
      "source": [
        "# rename data files\n",
        "!mv \"LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_attr\" \"LM-ontology-concept-placement/data/MM-S14-Disease/st21pv_syn_attr-all-complexEdge-filt\"\n",
        "!mv \"LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(out-of-KB-discovery)/syn_full\" \"LM-ontology-concept-placement/data/MM-S14-Disease/st21pv_syn_full-all-complexEdge-filt\"\n",
        "!mv \"LM-ontology-concept-placement/data/MM-S14-Disease/mention-level-(concept-placement)\" \"LM-ontology-concept-placement/data/MM-S14-Disease/st21pv_syn_attr-all-complexEdge-unsup\"\n",
        "!mv \"LM-ontology-concept-placement/data/MM-S14-Disease/mention-edge-pair-level\" \"LM-ontology-concept-placement/data/MM-S14-Disease/st21pv_syn_attr-all-complexEdge-edges-final\"\n",
        "\n",
        "!mv \"LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_attr\" \"LM-ontology-concept-placement/data/MM-S14-CPP/st21pv_syn_attr-all-complexEdge-filt\"\n",
        "!mv \"LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(out-of-KB-discovery)/syn_full\" \"LM-ontology-concept-placement/data/MM-S14-CPP/st21pv_syn_full-all-complexEdge-filt\"\n",
        "!mv \"LM-ontology-concept-placement/data/MM-S14-CPP/mention-level-(concept-placement)\" \"LM-ontology-concept-placement/data/MM-S14-CPP/st21pv_syn_attr-all-complexEdge-unsup\"\n",
        "!mv \"LM-ontology-concept-placement/data/MM-S14-CPP/mention-edge-pair-level\" \"LM-ontology-concept-placement/data/MM-S14-CPP/st21pv_syn_attr-all-complexEdge-edges-final\"\n",
        "\n",
        "# create folder and move data files\n",
        "!mkdir LM-ontology-concept-placement/data/MedMentions-preprocessed+\n",
        "!mkdir LM-ontology-concept-placement/data/MedMentions-preprocessed+/CPP\n",
        "!mkdir LM-ontology-concept-placement/data/MedMentions-preprocessed+/Disease\n",
        "\n",
        "!mv LM-ontology-concept-placement/data/MM-S14-CPP/* \"LM-ontology-concept-placement/data/MedMentions-preprocessed+/CPP\"\n",
        "!mv LM-ontology-concept-placement/data/MM-S14-Disease/* \"LM-ontology-concept-placement/data/MedMentions-preprocessed+/Disease\"\n",
        "\n",
        "# create folder and move ontology files\n",
        "!mkdir LM-ontology-concept-placement/ontologies\n",
        "!mv LM-ontology-concept-placement/data/MedMentions-preprocessed+/CPP/ontology/* LM-ontology-concept-placement/ontologies\n",
        "!mv LM-ontology-concept-placement/data/MedMentions-preprocessed+/Disease/ontology/* LM-ontology-concept-placement/ontologies\n",
        "\n",
        "# create a sample validation in-KB data file of random 100 data lines (which is need for debugging mode in inference)\n",
        "!shuf -n 100 LM-ontology-concept-placement/data/MedMentions-preprocessed+/Disease/st21pv_syn_attr-all-complexEdge-edges-final/valid.jsonl > LM-ontology-concept-placement/data/MedMentions-preprocessed+/Disease/st21pv_syn_attr-all-complexEdge-edges-final/valid-sample.jsonl\n",
        "\n",
        "!mkdir LM-ontology-concept-placement/results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NlJIGc_TUpsb"
      },
      "source": [
        "Set the focused folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ikeCxX0371gU",
        "outputId": "edb9f536-0e1f-4bc8-a25a-81082a9a820f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/LM-ontology-concept-placement\n"
          ]
        }
      ],
      "source": [
        "%cd LM-ontology-concept-placement"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vf_S44crUzDH"
      },
      "source": [
        "Download necessary libraries with pip\n",
        "\n",
        "Do not restart the program if there is a numpy installation issue"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OKVJ_CzbM3Mz",
        "outputId": "15b59e75-49fd-4825-fa9d-004be23dc517"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting torch==1.11.0 (from -r requirements.txt (line 4))\n",
            "  Downloading torch-1.11.0-cp310-cp310-manylinux1_x86_64.whl.metadata (24 kB)\n",
            "Collecting pysolr==3.8.1 (from -r requirements.txt (line 5))\n",
            "  Downloading pysolr-3.8.1-py2.py3-none-any.whl.metadata (8.3 kB)\n",
            "Collecting emoji==0.5.3 (from -r requirements.txt (line 6))\n",
            "  Downloading emoji-0.5.3.tar.gz (43 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.4/43.4 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting regex==2019.8.19 (from -r requirements.txt (line 7))\n",
            "  Downloading regex-2019.08.19.tar.gz (654 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m654.6/654.6 kB\u001b[0m \u001b[31m17.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting tqdm==4.66.1 (from -r requirements.txt (line 9))\n",
            "  Downloading tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.6/57.6 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nltk==3.6.3 (from -r requirements.txt (line 10))\n",
            "  Downloading nltk-3.6.3-py3-none-any.whl.metadata (3.0 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 11)) (1.26.4)\n",
            "Collecting segtok==1.5.7 (from -r requirements.txt (line 12))\n",
            "  Downloading segtok-1.5.7.tar.gz (24 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting flair==0.6.1 (from -r requirements.txt (line 13))\n",
            "  Downloading flair-0.6.1-py3-none-any.whl.metadata (12 kB)\n",
            "Collecting pytorch-transformers==1.2.0 (from -r requirements.txt (line 14))\n",
            "  Downloading pytorch_transformers-1.2.0-py3-none-any.whl.metadata (21 kB)\n",
            "Collecting transformers==4.29.0 (from -r requirements.txt (line 15))\n",
            "  Downloading transformers-4.29.0-py3-none-any.whl.metadata (111 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.9/111.9 kB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting colorama==0.4.3 (from -r requirements.txt (line 16))\n",
            "  Downloading colorama-0.4.3-py2.py3-none-any.whl.metadata (14 kB)\n",
            "Collecting termcolor==1.1.0 (from -r requirements.txt (line 17))\n",
            "  Downloading termcolor-1.1.0.tar.gz (3.9 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting faiss-cpu>=1.6.1 (from -r requirements.txt (line 18))\n",
            "  Downloading faiss_cpu-1.8.0.post1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.7 kB)\n",
            "Collecting pytorch_metric_learning==1.4.0 (from -r requirements.txt (line 19))\n",
            "  Downloading pytorch_metric_learning-1.4.0-py3-none-any.whl.metadata (16 kB)\n",
            "Collecting prettytable==3.3.0 (from -r requirements.txt (line 22))\n",
            "  Downloading prettytable-3.3.0-py3-none-any.whl.metadata (24 kB)\n",
            "Collecting rank_bm25==0.2.2 (from -r requirements.txt (line 23))\n",
            "  Downloading rank_bm25-0.2.2-py3-none-any.whl.metadata (3.2 kB)\n",
            "Collecting deeponto==0.8.8 (from -r requirements.txt (line 24))\n",
            "  Downloading deeponto-0.8.8-py3-none-any.whl.metadata (13 kB)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch==1.11.0->-r requirements.txt (line 4)) (4.12.2)\n",
            "Requirement already satisfied: requests>=2.9.1 in /usr/local/lib/python3.10/dist-packages (from pysolr==3.8.1->-r requirements.txt (line 5)) (2.31.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.3->-r requirements.txt (line 10)) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.3->-r requirements.txt (line 10)) (1.4.2)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (2.8.2)\n",
            "Requirement already satisfied: gensim>=3.4.0 in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (4.3.3)\n",
            "Requirement already satisfied: pytest>=5.3.2 in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (7.4.4)\n",
            "Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (3.7.1)\n",
            "Collecting mpld3==0.3 (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading mpld3-0.3.tar.gz (788 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m788.5/788.5 kB\u001b[0m \u001b[31m41.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (1.3.2)\n",
            "Collecting sqlitedict>=1.6.0 (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading sqlitedict-2.1.0.tar.gz (21 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting deprecated>=1.2.4 (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading Deprecated-1.2.14-py2.py3-none-any.whl.metadata (5.4 kB)\n",
            "Requirement already satisfied: hyperopt>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (0.2.7)\n",
            "Collecting bpemb>=0.3.2 (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading bpemb-0.3.5-py3-none-any.whl.metadata (19 kB)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (0.9.0)\n",
            "Collecting langdetect (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading langdetect-1.0.9.tar.gz (981 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m981.5/981.5 kB\u001b[0m \u001b[31m58.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (4.9.4)\n",
            "Collecting ftfy (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading ftfy-6.2.3-py3-none-any.whl.metadata (7.8 kB)\n",
            "Requirement already satisfied: sentencepiece!=0.1.92 in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (0.1.99)\n",
            "Collecting konoha<5.0.0,>=4.0.0 (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading konoha-4.6.5-py3-none-any.whl.metadata (10.0 kB)\n",
            "Collecting janome (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading Janome-0.5.0-py2.py3-none-any.whl.metadata (2.6 kB)\n",
            "Requirement already satisfied: gdown in /usr/local/lib/python3.10/dist-packages (from flair==0.6.1->-r requirements.txt (line 13)) (5.1.0)\n",
            "Collecting boto3 (from pytorch-transformers==1.2.0->-r requirements.txt (line 14))\n",
            "  Downloading boto3-1.34.155-py3-none-any.whl.metadata (6.6 kB)\n",
            "Collecting sacremoses (from pytorch-transformers==1.2.0->-r requirements.txt (line 14))\n",
            "  Downloading sacremoses-0.1.1-py3-none-any.whl.metadata (8.3 kB)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0->-r requirements.txt (line 15)) (3.15.4)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0->-r requirements.txt (line 15)) (0.23.5)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0->-r requirements.txt (line 15)) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0->-r requirements.txt (line 15)) (6.0.1)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.29.0->-r requirements.txt (line 15))\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from pytorch_metric_learning==1.4.0->-r requirements.txt (line 19)) (0.18.1+cu121)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prettytable==3.3.0->-r requirements.txt (line 22)) (0.2.13)\n",
            "Collecting JPype1==1.3.0 (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading JPype1-1.3.0.tar.gz (820 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m820.3/820.3 kB\u001b[0m \u001b[31m45.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting yacs (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading yacs-0.1.8-py3-none-any.whl.metadata (639 bytes)\n",
            "Collecting anytree==2.8.0 (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading anytree-2.8.0-py2.py3-none-any.whl.metadata (7.3 kB)\n",
            "Collecting dill (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.8->-r requirements.txt (line 24)) (2.1.4)\n",
            "Collecting openprompt==1.0.0 (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading openprompt-1.0.0-py3-none-any.whl.metadata (10 kB)\n",
            "Collecting datasets (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading datasets-2.20.0-py3-none-any.whl.metadata (19 kB)\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.8->-r requirements.txt (line 24)) (3.7.5)\n",
            "Collecting pprintpp (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading pprintpp-0.4.0-py2.py3-none-any.whl.metadata (7.9 kB)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.8->-r requirements.txt (line 24)) (3.3)\n",
            "Collecting textdistance (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading textdistance-4.6.3-py3-none-any.whl.metadata (18 kB)\n",
            "Requirement already satisfied: ipywidgets==7.7.1 in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.8->-r requirements.txt (line 24)) (7.7.1)\n",
            "Requirement already satisfied: ipykernel in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.8->-r requirements.txt (line 24)) (5.5.6)\n",
            "Collecting enlighten (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading enlighten-1.12.4-py2.py3-none-any.whl.metadata (18 kB)\n",
            "Collecting rdflib (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading rdflib-7.0.0-py3-none-any.whl.metadata (11 kB)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from anytree==2.8.0->deeponto==0.8.8->-r requirements.txt (line 24)) (1.16.0)\n",
            "Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.2.0)\n",
            "Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (5.7.1)\n",
            "Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (3.6.8)\n",
            "Requirement already satisfied: ipython>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (7.34.0)\n",
            "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (3.0.11)\n",
            "Collecting sentencepiece!=0.1.92 (from flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading sentencepiece-0.1.96-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)\n",
            "Collecting tensorboardX (from openprompt==1.0.0->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading tensorboardX-2.6.2.2-py2.py3-none-any.whl.metadata (5.8 kB)\n",
            "Collecting rouge==1.0.0 (from openprompt==1.0.0->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading rouge-1.0.0-py3-none-any.whl.metadata (4.0 kB)\n",
            "Requirement already satisfied: pyarrow in /usr/local/lib/python3.10/dist-packages (from openprompt==1.0.0->deeponto==0.8.8->-r requirements.txt (line 24)) (14.0.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from openprompt==1.0.0->deeponto==0.8.8->-r requirements.txt (line 24)) (1.13.1)\n",
            "Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.10/dist-packages (from deprecated>=1.2.4->flair==0.6.1->-r requirements.txt (line 13)) (1.16.0)\n",
            "Requirement already satisfied: smart-open>=1.8.1 in /usr/local/lib/python3.10/dist-packages (from gensim>=3.4.0->flair==0.6.1->-r requirements.txt (line 13)) (7.0.4)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.29.0->-r requirements.txt (line 15)) (2024.6.1)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.10/dist-packages (from hyperopt>=0.1.1->flair==0.6.1->-r requirements.txt (line 13)) (1.0.0)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from hyperopt>=0.1.1->flair==0.6.1->-r requirements.txt (line 13)) (2.2.1)\n",
            "Requirement already satisfied: py4j in /usr/local/lib/python3.10/dist-packages (from hyperopt>=0.1.1->flair==0.6.1->-r requirements.txt (line 13)) (0.10.9.7)\n",
            "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.10/dist-packages (from ipykernel->deeponto==0.8.8->-r requirements.txt (line 24)) (6.1.12)\n",
            "Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.10/dist-packages (from ipykernel->deeponto==0.8.8->-r requirements.txt (line 24)) (6.3.3)\n",
            "Collecting importlib-metadata<4.0.0,>=3.7.0 (from konoha<5.0.0,>=4.0.0->flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading importlib_metadata-3.10.1-py3-none-any.whl.metadata (3.5 kB)\n",
            "Collecting overrides<4.0.0,>=3.0.0 (from konoha<5.0.0,>=4.0.0->flair==0.6.1->-r requirements.txt (line 13))\n",
            "  Downloading overrides-3.1.0.tar.gz (11 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.2.3->flair==0.6.1->-r requirements.txt (line 13)) (1.2.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.2.3->flair==0.6.1->-r requirements.txt (line 13)) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.2.3->flair==0.6.1->-r requirements.txt (line 13)) (4.53.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.2.3->flair==0.6.1->-r requirements.txt (line 13)) (1.4.5)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.2.3->flair==0.6.1->-r requirements.txt (line 13)) (9.4.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=2.2.3->flair==0.6.1->-r requirements.txt (line 13)) (3.1.2)\n",
            "Requirement already satisfied: iniconfig in /usr/local/lib/python3.10/dist-packages (from pytest>=5.3.2->flair==0.6.1->-r requirements.txt (line 13)) (2.0.0)\n",
            "Requirement already satisfied: pluggy<2.0,>=0.12 in /usr/local/lib/python3.10/dist-packages (from pytest>=5.3.2->flair==0.6.1->-r requirements.txt (line 13)) (1.5.0)\n",
            "Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /usr/local/lib/python3.10/dist-packages (from pytest>=5.3.2->flair==0.6.1->-r requirements.txt (line 13)) (1.2.2)\n",
            "Requirement already satisfied: tomli>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from pytest>=5.3.2->flair==0.6.1->-r requirements.txt (line 13)) (2.0.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.9.1->pysolr==3.8.1->-r requirements.txt (line 5)) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.9.1->pysolr==3.8.1->-r requirements.txt (line 5)) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.9.1->pysolr==3.8.1->-r requirements.txt (line 5)) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.9.1->pysolr==3.8.1->-r requirements.txt (line 5)) (2024.7.4)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.21.3->flair==0.6.1->-r requirements.txt (line 13)) (3.5.0)\n",
            "Collecting botocore<1.35.0,>=1.34.155 (from boto3->pytorch-transformers==1.2.0->-r requirements.txt (line 14))\n",
            "  Downloading botocore-1.34.155-py3-none-any.whl.metadata (5.7 kB)\n",
            "Collecting jmespath<2.0.0,>=0.7.1 (from boto3->pytorch-transformers==1.2.0->-r requirements.txt (line 14))\n",
            "  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)\n",
            "Collecting s3transfer<0.11.0,>=0.10.0 (from boto3->pytorch-transformers==1.2.0->-r requirements.txt (line 14))\n",
            "  Downloading s3transfer-0.10.2-py3-none-any.whl.metadata (1.7 kB)\n",
            "Collecting pyarrow (from openprompt==1.0.0->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.3 kB)\n",
            "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (0.6)\n",
            "Collecting requests>=2.9.1 (from pysolr==3.8.1->-r requirements.txt (line 5))\n",
            "  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)\n",
            "INFO: pip is looking at multiple versions of datasets to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting datasets (from deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading datasets-2.19.2-py3-none-any.whl.metadata (19 kB)\n",
            "Collecting xxhash (from datasets->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
            "Collecting multiprocess (from datasets->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)\n",
            "Collecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.11.0->transformers==4.29.0->-r requirements.txt (line 15))\n",
            "  Downloading fsspec-2024.3.1-py3-none-any.whl.metadata (6.8 kB)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (3.10.0)\n",
            "Collecting blessed>=1.17.7 (from enlighten->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading blessed-1.20.0-py2.py3-none-any.whl.metadata (13 kB)\n",
            "Collecting prefixed>=0.3.2 (from enlighten->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading prefixed-0.7.1-py2.py3-none-any.whl.metadata (10 kB)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from gdown->flair==0.6.1->-r requirements.txt (line 13)) (4.12.3)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->deeponto==0.8.8->-r requirements.txt (line 24)) (2024.1)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->deeponto==0.8.8->-r requirements.txt (line 24)) (2024.1)\n",
            "Collecting isodate<0.7.0,>=0.6.0 (from rdflib->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Downloading isodate-0.6.1-py2.py3-none-any.whl.metadata (9.6 kB)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (1.0.10)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (2.0.8)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (8.2.5)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (1.1.3)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (2.0.10)\n",
            "Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (0.4.1)\n",
            "Requirement already satisfied: typer<1.0.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (0.12.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (2.8.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (3.1.4)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (71.0.4)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (3.4.0)\n",
            "INFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting torchvision (from pytorch_metric_learning==1.4.0->-r requirements.txt (line 19))\n",
            "  Downloading torchvision-0.19.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.0 kB)\n",
            "  Downloading torchvision-0.18.1-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
            "  Downloading torchvision-0.18.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
            "  Downloading torchvision-0.17.2-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
            "  Downloading torchvision-0.17.1-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
            "  Downloading torchvision-0.17.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
            "  Downloading torchvision-0.16.2-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
            "INFO: pip is still looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while.\n",
            "  Downloading torchvision-0.16.1-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
            "  Downloading torchvision-0.16.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
            "  Downloading torchvision-0.15.2-cp310-cp310-manylinux1_x86_64.whl.metadata (11 kB)\n",
            "  Downloading torchvision-0.15.1-cp310-cp310-manylinux1_x86_64.whl.metadata (11 kB)\n",
            "  Downloading torchvision-0.14.1-cp310-cp310-manylinux1_x86_64.whl.metadata (11 kB)\n",
            "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n",
            "  Downloading torchvision-0.14.0-cp310-cp310-manylinux1_x86_64.whl.metadata (11 kB)\n",
            "  Downloading torchvision-0.13.1-cp310-cp310-manylinux1_x86_64.whl.metadata (10 kB)\n",
            "  Downloading torchvision-0.13.0-cp310-cp310-manylinux1_x86_64.whl.metadata (10 kB)\n",
            "  Downloading torchvision-0.12.0-cp310-cp310-manylinux1_x86_64.whl.metadata (10 kB)\n",
            "Requirement already satisfied: accelerate>=0.19.0 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]->deeponto==0.8.8->-r requirements.txt (line 24)) (0.32.1)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.19.0->transformers[torch]->deeponto==0.8.8->-r requirements.txt (line 24)) (5.9.5)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.19.0->transformers[torch]->deeponto==0.8.8->-r requirements.txt (line 24)) (0.4.3)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (2.3.4)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->deeponto==0.8.8->-r requirements.txt (line 24)) (4.0.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata<4.0.0,>=3.7.0->konoha<5.0.0,>=4.0.0->flair==0.6.1->-r requirements.txt (line 13)) (3.19.2)\n",
            "Collecting jedi>=0.16 (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24))\n",
            "  Using cached jedi-0.19.1-py2.py3-none-any.whl.metadata (22 kB)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (4.4.2)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.7.5)\n",
            "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (3.0.47)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (2.16.1)\n",
            "Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.2.0)\n",
            "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.1.7)\n",
            "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (4.9.0)\n",
            "Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.10/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (1.2.0)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (2.20.1)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (0.1.5)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (1.5.4)\n",
            "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (13.7.1)\n",
            "Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (0.18.1)\n",
            "Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.10/dist-packages (from widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (6.5.5)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->gdown->flair==0.6.1->-r requirements.txt (line 13)) (2.5)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (2.1.5)\n",
            "Requirement already satisfied: jupyter-core>=4.6.0 in /usr/local/lib/python3.10/dist-packages (from jupyter-client->ipykernel->deeponto==0.8.8->-r requirements.txt (line 24)) (5.7.2)\n",
            "Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.10/dist-packages (from jupyter-client->ipykernel->deeponto==0.8.8->-r requirements.txt (line 24)) (24.0.1)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown->flair==0.6.1->-r requirements.txt (line 13)) (1.7.1)\n",
            "Requirement already satisfied: protobuf>=3.20 in /usr/local/lib/python3.10/dist-packages (from tensorboardX->openprompt==1.0.0->deeponto==0.8.8->-r requirements.txt (line 24)) (3.20.3)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.8.4)\n",
            "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core>=4.6.0->jupyter-client->ipykernel->deeponto==0.8.8->-r requirements.txt (line 24)) (4.2.2)\n",
            "Requirement already satisfied: marisa-trie>=0.7.7 in /usr/local/lib/python3.10/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (1.2.0)\n",
            "Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (23.1.0)\n",
            "Requirement already satisfied: nbformat in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (5.10.4)\n",
            "Requirement already satisfied: nbconvert>=5 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (6.5.4)\n",
            "Requirement already satisfied: nest-asyncio>=1.5 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.6.0)\n",
            "Requirement already satisfied: Send2Trash>=1.8.0 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.8.3)\n",
            "Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.18.1)\n",
            "Requirement already satisfied: prometheus-client in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.20.0)\n",
            "Requirement already satisfied: nbclassic>=0.4.7 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.1.0)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.7.0)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (3.0.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy->deeponto==0.8.8->-r requirements.txt (line 24)) (0.1.2)\n",
            "Requirement already satisfied: notebook-shim>=0.2.3 in /usr/local/lib/python3.10/dist-packages (from nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.2.4)\n",
            "Requirement already satisfied: bleach in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (6.1.0)\n",
            "Requirement already satisfied: defusedxml in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.7.1)\n",
            "Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.4)\n",
            "Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.3.0)\n",
            "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.8.4)\n",
            "Requirement already satisfied: nbclient>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.10.0)\n",
            "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.5.1)\n",
            "Requirement already satisfied: tinycss2 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.3.0)\n",
            "Requirement already satisfied: fastjsonschema>=2.15 in /usr/local/lib/python3.10/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (2.20.0)\n",
            "Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.10/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (4.23.0)\n",
            "Requirement already satisfied: argon2-cffi-bindings in /usr/local/lib/python3.10/dist-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (21.2.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (2023.12.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.19.1)\n",
            "Requirement already satisfied: jupyter-server<3,>=1.8 in /usr/local/lib/python3.10/dist-packages (from notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.24.0)\n",
            "Requirement already satisfied: cffi>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.16.0)\n",
            "Requirement already satisfied: webencodings in /usr/local/lib/python3.10/dist-packages (from bleach->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (0.5.1)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (2.22)\n",
            "Requirement already satisfied: anyio<4,>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from jupyter-server<3,>=1.8->notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (3.7.1)\n",
            "Requirement already satisfied: websocket-client in /usr/local/lib/python3.10/dist-packages (from jupyter-server<3,>=1.8->notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.8.0)\n",
            "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.1.0->jupyter-server<3,>=1.8->notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.8->-r requirements.txt (line 24)) (1.3.1)\n",
            "Downloading torch-1.11.0-cp310-cp310-manylinux1_x86_64.whl (750.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m750.6/750.6 MB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pysolr-3.8.1-py2.py3-none-any.whl (16 kB)\n",
            "Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.3/78.3 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nltk-3.6.3-py3-none-any.whl (1.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m71.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading flair-0.6.1-py3-none-any.whl (331 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m331.5/331.5 kB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pytorch_transformers-1.2.0-py3-none-any.whl (176 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m176.4/176.4 kB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading transformers-4.29.0-py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m71.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading colorama-0.4.3-py2.py3-none-any.whl (15 kB)\n",
            "Downloading pytorch_metric_learning-1.4.0-py3-none-any.whl (110 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.3/110.3 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading prettytable-3.3.0-py3-none-any.whl (26 kB)\n",
            "Downloading rank_bm25-0.2.2-py3-none-any.whl (8.6 kB)\n",
            "Downloading deeponto-0.8.8-py3-none-any.whl (89.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m89.7/89.7 MB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading anytree-2.8.0-py2.py3-none-any.whl (41 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.7/41.7 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading openprompt-1.0.0-py3-none-any.whl (146 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m146.1/146.1 kB\u001b[0m \u001b[31m14.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading sentencepiece-0.1.96-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m58.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading rouge-1.0.0-py3-none-any.whl (14 kB)\n",
            "Downloading faiss_cpu-1.8.0.post1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (27.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m27.0/27.0 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading bpemb-0.3.5-py3-none-any.whl (19 kB)\n",
            "Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)\n",
            "Downloading konoha-4.6.5-py3-none-any.whl (20 kB)\n",
            "Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m83.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading boto3-1.34.155-py3-none-any.whl (139 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m139.2/139.2 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading datasets-2.19.2-py3-none-any.whl (542 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m542.1/542.1 kB\u001b[0m \u001b[31m27.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading requests-2.32.3-py3-none-any.whl (64 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m64.9/64.9 kB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading enlighten-1.12.4-py2.py3-none-any.whl (41 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.9/41.9 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading ftfy-6.2.3-py3-none-any.whl (43 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.0/43.0 kB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading Janome-0.5.0-py2.py3-none-any.whl (19.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.7/19.7 MB\u001b[0m \u001b[31m66.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pprintpp-0.4.0-py2.py3-none-any.whl (16 kB)\n",
            "Downloading rdflib-7.0.0-py3-none-any.whl (531 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m531.9/531.9 kB\u001b[0m \u001b[31m29.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading sacremoses-0.1.1-py3-none-any.whl (897 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m897.5/897.5 kB\u001b[0m \u001b[31m51.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading textdistance-4.6.3-py3-none-any.whl (31 kB)\n",
            "Downloading torchvision-0.12.0-cp310-cp310-manylinux1_x86_64.whl (21.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading yacs-0.1.8-py3-none-any.whl (14 kB)\n",
            "Downloading blessed-1.20.0-py2.py3-none-any.whl (58 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.4/58.4 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading botocore-1.34.155-py3-none-any.whl (12.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.5/12.5 MB\u001b[0m \u001b[31m82.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading fsspec-2024.3.1-py3-none-any.whl (171 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m172.0/172.0 kB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading importlib_metadata-3.10.1-py3-none-any.whl (14 kB)\n",
            "Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.7/41.7 kB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading jmespath-1.0.1-py3-none-any.whl (20 kB)\n",
            "Downloading prefixed-0.7.1-py2.py3-none-any.whl (13 kB)\n",
            "Downloading s3transfer-0.10.2-py3-none-any.whl (82 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.7/82.7 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m13.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tensorboardX-2.6.2.2-py2.py3-none-any.whl (101 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m101.7/101.7 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m18.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hUsing cached jedi-0.19.1-py2.py3-none-any.whl (1.6 MB)\n",
            "Building wheels for collected packages: emoji, regex, segtok, termcolor, JPype1, mpld3, sqlitedict, langdetect, overrides\n",
            "  Building wheel for emoji (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for emoji: filename=emoji-0.5.3-py3-none-any.whl size=42182 sha256=148d1d5baedcec3554bf2a03c7f58a9002b7e44ada3c9ccce4214d577d6f4268\n",
            "  Stored in directory: /root/.cache/pip/wheels/d6/4a/a4/bbc13c14cf69e4bdb342087e3560f27147d66abbe1717191bd\n",
            "  Building wheel for regex (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for regex: filename=regex-2019.8.19-cp310-cp310-linux_x86_64.whl size=745589 sha256=20691b4df69addb5169cbd571d3974a751e10ae8d3ca595addc9b73d138078e6\n",
            "  Stored in directory: /root/.cache/pip/wheels/12/1c/7b/5f9a9a14dad5e355925af4ed02a26cc8316afdadd76048c311\n",
            "  Building wheel for segtok (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for segtok: filename=segtok-1.5.7-py3-none-any.whl size=23236 sha256=16f4386dc1b745d02e671078e199141621c3b42d77744f4960487f4e9e62adfe\n",
            "  Stored in directory: /root/.cache/pip/wheels/41/d2/13/eceaf7b7e5d891e5b4972e3737ea42acf03062c809a8fc489b\n",
            "  Building wheel for termcolor (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4832 sha256=64ab878bbefb331592f3b8b63b5f7d3be0162e21c00c5cf5cd1e501265a1b13a\n",
            "  Stored in directory: /root/.cache/pip/wheels/a1/49/46/1b13a65d8da11238af9616b00fdde6d45b0f95d9291bac8452\n",
            "  Building wheel for JPype1 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for JPype1: filename=JPype1-1.3.0-cp310-cp310-linux_x86_64.whl size=454407 sha256=92d648d20ff28ec39a4ceb2e0adbd67748a692d506fe3e50fcd925d41b3f1f5d\n",
            "  Stored in directory: /root/.cache/pip/wheels/f5/c7/8f/c97c6c9868c256c8d17dabb772a2ca9002dcff2912fa8d7d58\n",
            "  Building wheel for mpld3 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for mpld3: filename=mpld3-0.3-py3-none-any.whl size=116684 sha256=7908ddc01ba4bebcc48a89d3321fbf75a58e795edaed6274dc7195f8e3e0d7cb\n",
            "  Stored in directory: /root/.cache/pip/wheels/9c/92/f7/45d9aac5dcfb1c2a1761a272365599cc7ba1050ce211a3fd9a\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sqlitedict: filename=sqlitedict-2.1.0-py3-none-any.whl size=16863 sha256=adf97f163686541a3f24cde5128efd97e6d8bc99e424564b0eb6fa7383c69f3b\n",
            "  Stored in directory: /root/.cache/pip/wheels/79/d6/e7/304e0e6cb2221022c26d8161f7c23cd4f259a9e41e8bbcfabd\n",
            "  Building wheel for langdetect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993221 sha256=96218574eeca8c20c047d9c14f385c341bb8d77ca65d8319271baabadec284f1\n",
            "  Stored in directory: /root/.cache/pip/wheels/95/03/7d/59ea870c70ce4e5a370638b5462a7711ab78fba2f655d05106\n",
            "  Building wheel for overrides (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for overrides: filename=overrides-3.1.0-py3-none-any.whl size=10172 sha256=cff70c8edf3bd6ed61c7d9b2d520bd5b4b23c70c712f50fd7d6f36c3a64fdc67\n",
            "  Stored in directory: /root/.cache/pip/wheels/bd/23/63/4d5849844f8f9d32be09e1b9b278e80de2d8314fbf1e28068b\n",
            "Successfully built emoji regex segtok termcolor JPype1 mpld3 sqlitedict langdetect overrides\n",
            "Installing collected packages: tokenizers, termcolor, sqlitedict, sentencepiece, regex, prefixed, pprintpp, overrides, mpld3, JPype1, janome, emoji, yacs, xxhash, tqdm, torch, textdistance, tensorboardX, segtok, rouge, requests, rank_bm25, prettytable, langdetect, jmespath, jedi, isodate, importlib-metadata, ftfy, fsspec, faiss-cpu, dill, deprecated, colorama, blessed, anytree, torchvision, sacremoses, rdflib, pysolr, nltk, multiprocess, konoha, enlighten, botocore, transformers, s3transfer, pytorch_metric_learning, bpemb, flair, datasets, boto3, pytorch-transformers, openprompt, deeponto\n",
            "  Attempting uninstall: tokenizers\n",
            "    Found existing installation: tokenizers 0.19.1\n",
            "    Uninstalling tokenizers-0.19.1:\n",
            "      Successfully uninstalled tokenizers-0.19.1\n",
            "  Attempting uninstall: termcolor\n",
            "    Found existing installation: termcolor 2.4.0\n",
            "    Uninstalling termcolor-2.4.0:\n",
            "      Successfully uninstalled termcolor-2.4.0\n",
            "  Attempting uninstall: sentencepiece\n",
            "    Found existing installation: sentencepiece 0.1.99\n",
            "    Uninstalling sentencepiece-0.1.99:\n",
            "      Successfully uninstalled sentencepiece-0.1.99\n",
            "  Attempting uninstall: regex\n",
            "    Found existing installation: regex 2024.5.15\n",
            "    Uninstalling regex-2024.5.15:\n",
            "      Successfully uninstalled regex-2024.5.15\n",
            "  Attempting uninstall: tqdm\n",
            "    Found existing installation: tqdm 4.66.4\n",
            "    Uninstalling tqdm-4.66.4:\n",
            "      Successfully uninstalled tqdm-4.66.4\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 2.3.1+cu121\n",
            "    Uninstalling torch-2.3.1+cu121:\n",
            "      Successfully uninstalled torch-2.3.1+cu121\n",
            "  Attempting uninstall: requests\n",
            "    Found existing installation: requests 2.31.0\n",
            "    Uninstalling requests-2.31.0:\n",
            "      Successfully uninstalled requests-2.31.0\n",
            "  Attempting uninstall: prettytable\n",
            "    Found existing installation: prettytable 3.10.2\n",
            "    Uninstalling prettytable-3.10.2:\n",
            "      Successfully uninstalled prettytable-3.10.2\n",
            "  Attempting uninstall: importlib-metadata\n",
            "    Found existing installation: importlib_metadata 8.2.0\n",
            "    Uninstalling importlib_metadata-8.2.0:\n",
            "      Successfully uninstalled importlib_metadata-8.2.0\n",
            "  Attempting uninstall: fsspec\n",
            "    Found existing installation: fsspec 2024.6.1\n",
            "    Uninstalling fsspec-2024.6.1:\n",
            "      Successfully uninstalled fsspec-2024.6.1\n",
            "  Attempting uninstall: torchvision\n",
            "    Found existing installation: torchvision 0.18.1+cu121\n",
            "    Uninstalling torchvision-0.18.1+cu121:\n",
            "      Successfully uninstalled torchvision-0.18.1+cu121\n",
            "  Attempting uninstall: nltk\n",
            "    Found existing installation: nltk 3.8.1\n",
            "    Uninstalling nltk-3.8.1:\n",
            "      Successfully uninstalled nltk-3.8.1\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.42.4\n",
            "    Uninstalling transformers-4.42.4:\n",
            "      Successfully uninstalled transformers-4.42.4\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "dask 2024.7.1 requires importlib-metadata>=4.13.0; python_version < \"3.12\", but you have importlib-metadata 3.10.1 which is incompatible.\n",
            "gcsfs 2024.6.1 requires fsspec==2024.6.1, but you have fsspec 2024.3.1 which is incompatible.\n",
            "google-colab 1.0.0 requires requests==2.31.0, but you have requests 2.32.3 which is incompatible.\n",
            "torchaudio 2.3.1+cu121 requires torch==2.3.1, but you have torch 1.11.0 which is incompatible.\n",
            "torchtext 0.18.0 requires torch>=2.3.0, but you have torch 1.11.0 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed JPype1-1.3.0 anytree-2.8.0 blessed-1.20.0 boto3-1.34.155 botocore-1.34.155 bpemb-0.3.5 colorama-0.4.3 datasets-2.19.2 deeponto-0.8.8 deprecated-1.2.14 dill-0.3.8 emoji-0.5.3 enlighten-1.12.4 faiss-cpu-1.8.0.post1 flair-0.6.1 fsspec-2024.3.1 ftfy-6.2.3 importlib-metadata-3.10.1 isodate-0.6.1 janome-0.5.0 jedi-0.19.1 jmespath-1.0.1 konoha-4.6.5 langdetect-1.0.9 mpld3-0.3 multiprocess-0.70.16 nltk-3.6.3 openprompt-1.0.0 overrides-3.1.0 pprintpp-0.4.0 prefixed-0.7.1 prettytable-3.3.0 pysolr-3.8.1 pytorch-transformers-1.2.0 pytorch_metric_learning-1.4.0 rank_bm25-0.2.2 rdflib-7.0.0 regex-2019.8.19 requests-2.32.3 rouge-1.0.0 s3transfer-0.10.2 sacremoses-0.1.1 segtok-1.5.7 sentencepiece-0.1.96 sqlitedict-2.1.0 tensorboardX-2.6.2.2 termcolor-1.1.0 textdistance-4.6.3 tokenizers-0.13.3 torch-1.11.0 torchvision-0.12.0 tqdm-4.66.1 transformers-4.29.0 xxhash-3.4.1 yacs-0.1.8\n"
          ]
        }
      ],
      "source": [
        "# !sudo apt-get install python3.8\n",
        "!pip install -r requirements.txt\n",
        "# do not restart due to the numpy issue"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Training Bi-encoder and Cross-encoder\n",
        "\n",
        "This shell script executes:\n",
        "(i) Edge search with Edge-Bi-encoder training, evaluation\n",
        "(ii) Edge enrichment\n",
        "(iii) Edge selection: Edge-Cross-encoder\n",
        "\n",
        "The script is Edge-Bi-enc+Cross-enc.sh \\$1 \\$2 \\$3 \\$4 \\$5 \\$6 \\$7 \\$8 \\$9\n",
        "\n",
        "Parameters (marked with \\$1 to \\$9):\n",
        "1. subset string: \"Disease\" or \"CPP\"\n",
        "2. whether to use context\n",
        "3. top-k value\n",
        "4. number of edge seeds before edge enrichment into top-k\n",
        "5. biencoder training batch size\n",
        "6. whether to use debugging mode in training (training biencoder with first debug_max_lines, which have default value as 1000)\n",
        "7. whether to use a sample of first n rows of mention-edge pairs for creating data for cross-encoder.\n",
        "8. n rows mention-edge pairs for creating data for cross-encoder\n",
        "9. whether to use debugging mode in inference (for in-KB, inference with only a random 100 data lines; for out-of-KB, still inference will all data lines)."
      ],
      "metadata": {
        "id": "zBO7IogTpEgu"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fkykflY28IJ3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f99ce0a8-a271-4823-c712-b5390dda3b72"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Edge-Bi-enc+Cross-enc.sh: line 27: activate: No such file or directory\n",
            "Namespace(max_seq_length=160, max_context_length=32, max_cand_length=128, path_to_model=None, bert_model='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', pull_from_layer=-1, lowercase=True, context_key='context', use_context=True, out_dim=1, add_linear=False, data_path='data/MedMentions-preprocessed+/Disease/st21pv_syn_attr-all-complexEdge-edges-final', output_path='models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug', use_triplet_loss_bi_enc=True, use_miner_bi_enc=False, use_ori_classification=False, use_NIL_classification=False, lambda_NIL=0.25, use_score_features=False, use_score_pooling=False, use_men_only_score_ft=False, use_extra_features=False, use_BM25=False, aggregating_factor=20, silent=False, debug=True, debug_max_lines=1000, data_parallel=True, no_cuda=False, top_k=10, seed=52313, zeshel=False, evaluate=False, output_eval_file=None, limit_by_max_lines=False, max_number_of_train_size=300000, train_batch_size=16, max_grad_norm=1.0, learning_rate=3e-05, num_train_epochs=1, limit_by_train_steps=False, max_num_train_steps=20000, print_interval=100, eval_interval=2000, save_interval=1, save_model_epoch_parts=False, warmup_proportion=0.1, gradient_accumulation_steps=1, type_optimization='all_encoder_layers', shuffle=True, optimize_NIL=False, use_preprocessed_data=False, use_full_training_data=False, fix_seeds=True, eval_batch_size=16, mode='valid', save_topk_result=False, encode_batch_size=8, use_synonyms=False, use_NIL_tag=False, use_NIL_desc=False, use_NIL_desc_tag=False, cand_pool_path=None, cand_encode_path=None, use_NIL_classification_infer=False, add_NIL_to_bi_enc_pred=False, NIL_ent_ind=64076)\n",
            "self.n_gpu: 1\n",
            "self.device in BiEncoderRanker: cuda\n",
            "params[\"lowercase\"]: True\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "tokenizer_config.json: 100% 198/198 [00:00<00:00, 1.18MB/s]\n",
            "config.json: 100% 462/462 [00:00<00:00, 3.08MB/s]\n",
            "vocab.txt: 100% 226k/226k [00:00<00:00, 3.51MB/s]\n",
            "special_tokens_map.json: 100% 112/112 [00:00<00:00, 746kB/s]\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model.safetensors: 100% 438M/438M [00:06<00:00, 64.8MB/s]\n",
            "model built\n",
            "model loaded\n",
            "moving model to cuda\n",
            "model moved to cuda\n",
            "n_gpu: 1\n",
            "100% 887840/887840 [00:10<00:00, 82822.56it/s]\n",
            "08/07/2024 14:01:39 - INFO - Blink -   Read 887840 train samples.\n",
            "num of samples: 887840\n",
            "100% 1000/1000 [00:01<00:00, 615.50it/s]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   ====Processed samples (first 10): ====\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis without mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 2979 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40179 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with intestinal manifestations ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 6139 10952 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40180 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40181 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis of the lung ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1927 1920 3614 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40182 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis without mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 2979 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40179 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with intestinal manifestations ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 6139 10952 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40180 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40181 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis of the lung ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1927 1920 3614 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40182 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cf [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 4920 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( [unused0] cf [unused1] ) patients is associated with worse long - term pulmonary disease and shorter survival [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 1922 11656 6769 13461 10830 12 3838 13 2909 1922 11656 6769 12 1 4920 1 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 1930 7310 3298 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( [unused0] cf [unused1] ) patients is associated with worse long - term pulmonary disease and shorter survival [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 1922 11656 6769 13461 10830 12 3838 13 2909 1922 11656 6769 12 1 4920 1 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 1930 7310 3298 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis without mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 2979 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40179 is in-KB\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention tokens : [unused0] cf [unused1]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Mention ids : 1 4920 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   (Context) tokens : [CLS] in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( [unused0] cf [unused1] ) patients is associated with worse long - term pulmonary disease and shorter survival [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Context ids : 2 1922 11656 6769 13461 10830 12 3838 13 2909 1922 11656 6769 12 1 4920 1 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 1930 7310 3298 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual tokens : [CLS] in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( [unused0] cf [unused1] ) patients is associated with worse long - term pulmonary disease and shorter survival [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Contextual ids : 2 1922 11656 6769 13461 10830 12 3838 13 2909 1922 11656 6769 12 1 4920 1 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 1930 7310 3298 3\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with intestinal manifestations ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 6139 10952 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:41 - INFO - Blink -   Label_id : 40180 is in-KB\n",
            "label_text_vecs: tensor([[    2, 12911,  3422,  ...,     0,     0,     0],\n",
            "        [    2, 12911,  3422,  ...,     0,     0,     0],\n",
            "        [    2, 12911,  3422,  ...,     0,     0,     0],\n",
            "        ...,\n",
            "        [    2,  2535,  2849,  ...,     0,     0,     0],\n",
            "        [    2,  2535,  2849,  ...,     0,     0,     0],\n",
            "        [    2,  2535,  2849,  ...,     0,     0,     0]])\n",
            "random shuffling\n",
            "100% 383457/383457 [00:05<00:00, 71310.58it/s]\n",
            "08/07/2024 14:01:49 - INFO - Blink -   Read 383457 valid samples.\n",
            "num of samples: 383457\n",
            "100% 1000/1000 [00:01<00:00, 561.93it/s]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   ====Processed samples (first 10): ====\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 475 ##05 ##003\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] stock ##holm syndrome ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 5592 4770 12 4770 13 1 8260 23608 4741 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 189459 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 475 ##05 ##003\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] acute post - trauma stress state ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 5592 4770 12 4770 13 1 3939 2753 17 7176 3071 3360 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 189460 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 475 ##05 ##003\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] acute posttraumatic stress disorder following military combat ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 5592 4770 12 4770 13 1 3939 20999 3071 4770 2894 15720 18565 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 189461 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 475 ##05 ##003\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] chronic post - traumatic stress disorder ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 5592 4770 12 4770 13 1 3724 2753 17 10242 3071 4770 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 189462 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 475 ##05 ##003\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] posttraumatic stress disorder , delayed onset ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 5592 4770 12 4770 13 1 20999 3071 4770 16 7083 4727 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 189463 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 475 ##05 ##003\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] complex posttraumatic stress disorder ( disorder ) [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 5592 4770 12 4770 13 1 2796 20999 3071 4770 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 189464 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] nosocomial pneumonia [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 22905 7868 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 425 ##46 ##400 ##7\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 24592 7772 10893 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] [unused0] nosocomial pneumonia [unused1] caused by methicillin - resistant staphylococcus aureus treated with line ##zo ##li ##d or vancomycin : a secondary economic analysis of resource use from a spanish [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 1 22905 7868 1 4189 2007 23025 17 4910 13452 8849 2868 1956 2823 13642 12191 1007 2014 16100 30 43 4067 7029 2333 1927 7965 2485 2037 43 13289 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] [unused0] nosocomial pneumonia [unused1] caused by methicillin - resistant staphylococcus aureus treated with line ##zo ##li ##d or vancomycin : a secondary economic analysis of resource use from a spanish [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 1 22905 7868 1 4189 2007 23025 17 4910 13452 8849 2868 1956 2823 13642 12191 1007 2014 16100 30 43 4067 7029 2333 1927 7965 2485 2037 43 13289 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] infective pneumonia ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 20905 7868 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 72468 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] nosocomial pneumonia [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 22905 7868 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 425 ##46 ##400 ##7\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 24592 7772 10893 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] [unused0] nosocomial pneumonia [unused1] caused by methicillin - resistant staphylococcus aureus treated with line ##zo ##li ##d or vancomycin : a secondary economic analysis of resource use from a spanish [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 1 22905 7868 1 4189 2007 23025 17 4910 13452 8849 2868 1956 2823 13642 12191 1007 2014 16100 30 43 4067 7029 2333 1927 7965 2485 2037 43 13289 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] [unused0] nosocomial pneumonia [unused1] caused by methicillin - resistant staphylococcus aureus treated with line ##zo ##li ##d or vancomycin : a secondary economic analysis of resource use from a spanish [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 1 22905 7868 1 4189 2007 23025 17 4910 13452 8849 2868 1956 2823 13642 12191 1007 2014 16100 30 43 4067 7029 2333 1927 7965 2485 2037 43 13289 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] nosocomial infectious disease ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 22905 7465 2573 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 177009 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] nosocomial pneumonia [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 22905 7868 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 425 ##46 ##400 ##7\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 24592 7772 10893 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] assess healthcare resource utilization ( hc ##ru ) and the costs of treating [unused0] nosocomial pneumonia [unused1] ( np ) produced by methicillin - resistant staphylococcus aureus ( mrsa ) [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 2634 5841 7965 8422 12 4387 3176 13 1930 1920 5982 1927 8247 1 22905 7868 1 12 5007 13 4152 2007 23025 17 4910 13452 8849 12 12836 13 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] assess healthcare resource utilization ( hc ##ru ) and the costs of treating [unused0] nosocomial pneumonia [unused1] ( np ) produced by methicillin - resistant staphylococcus aureus ( mrsa ) [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 2634 5841 7965 8422 12 4387 3176 13 1930 1920 5982 1927 8247 1 22905 7868 1 12 5007 13 4152 2007 23025 17 4910 13452 8849 12 12836 13 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] infective pneumonia ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 20905 7868 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 72468 is in-KB\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention tokens : [unused0] nosocomial pneumonia [unused1]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Mention ids : 1 22905 7868 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori tokens : 425 ##46 ##400 ##7\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Concept ID ori ids : 24592 7772 10893 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   (Context) tokens : [CLS] assess healthcare resource utilization ( hc ##ru ) and the costs of treating [unused0] nosocomial pneumonia [unused1] ( np ) produced by methicillin - resistant staphylococcus aureus ( mrsa ) [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Context ids : 2 2634 5841 7965 8422 12 4387 3176 13 1930 1920 5982 1927 8247 1 22905 7868 1 12 5007 13 4152 2007 23025 17 4910 13452 8849 12 12836 13 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual tokens : [CLS] assess healthcare resource utilization ( hc ##ru ) and the costs of treating [unused0] nosocomial pneumonia [unused1] ( np ) produced by methicillin - resistant staphylococcus aureus ( mrsa ) [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Contextual ids : 2 2634 5841 7965 8422 12 4387 3176 13 1930 1920 5982 1927 8247 1 22905 7868 1 12 5007 13 4152 2007 23025 17 4910 13452 8849 12 12836 13 3\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label tokens : [CLS] nosocomial infectious disease ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label ids : 2 22905 7465 2573 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "08/07/2024 14:01:51 - INFO - Blink -   Label_id : 177009 is in-KB\n",
            "label_text_vecs: tensor([[   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        [   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        [   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0],\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0],\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0]])\n",
            "Evaluation:   0% 0/63 [00:00<?, ?it/s]context_input: tensor([[    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,     1, 22905,  7868,     1,  4189,  2007, 23025,    17,  4910,\n",
            "         13452,  8849,  2868,  1956,  2823, 13642, 12191,  1007,  2014, 16100,\n",
            "            30,    43,  4067,  7029,  2333,  1927,  7965,  2485,  2037,    43,\n",
            "         13289,     3],\n",
            "        [    2,     1, 22905,  7868,     1,  4189,  2007, 23025,    17,  4910,\n",
            "         13452,  8849,  2868,  1956,  2823, 13642, 12191,  1007,  2014, 16100,\n",
            "            30,    43,  4067,  7029,  2333,  1927,  7965,  2485,  2037,    43,\n",
            "         13289,     3],\n",
            "        [    2,  2634,  5841,  7965,  8422,    12,  4387,  3176,    13,  1930,\n",
            "          1920,  5982,  1927,  8247,     1, 22905,  7868,     1,    12,  5007,\n",
            "            13,  4152,  2007, 23025,    17,  4910, 13452,  8849,    12, 12836,\n",
            "            13,     3],\n",
            "        [    2,  2634,  5841,  7965,  8422,    12,  4387,  3176,    13,  1930,\n",
            "          1920,  5982,  1927,  8247,     1, 22905,  7868,     1,    12,  5007,\n",
            "            13,  4152,  2007, 23025,    17,  4910, 13452,  8849,    12, 12836,\n",
            "            13,     3],\n",
            "        [    2,  8422,    12,  4387,  3176,    13,  1930,  1920,  5982,  1927,\n",
            "          8247, 22905,  7868,    12,     1,  5007,     1,    13,  4152,  2007,\n",
            "         23025,    17,  4910, 13452,  8849,    12, 12836,    13,  1922, 13123,\n",
            "          4440,     3],\n",
            "        [    2,  8422,    12,  4387,  3176,    13,  1930,  1920,  5982,  1927,\n",
            "          8247, 22905,  7868,    12,     1,  5007,     1,    13,  4152,  2007,\n",
            "         23025,    17,  4910, 13452,  8849,    12, 12836,    13,  1922, 13123,\n",
            "          4440,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3]], device='cuda:0') <class 'torch.Tensor'>\n",
            "candidate_input: tensor([[   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        [   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        [   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0]], device='cuda:0') <class 'torch.Tensor'>\n",
            "Evaluation:   2% 1/63 [00:00<00:32,  1.88it/s]context_input: tensor([[    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2,    16,  4825,    17,  7109,    16, 14620,  3193,    24,  2161,\n",
            "          1982,  4371,  2397,    18,     1, 22905,  7868,     1,  2810,  1942,\n",
            "         12836,  1922, 13123,  4440,    18,  1920,  4795, 17777,  1942,  2297,\n",
            "            12,     3],\n",
            "        [    2,    16,  4825,    17,  7109,    16, 14620,  3193,    24,  2161,\n",
            "          1982,  4371,  2397,    18,     1, 22905,  7868,     1,  2810,  1942,\n",
            "         12836,  1922, 13123,  4440,    18,  1920,  4795, 17777,  1942,  2297,\n",
            "            12,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3]], device='cuda:0') <class 'torch.Tensor'>\n",
            "candidate_input: tensor([[   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0]], device='cuda:0') <class 'torch.Tensor'>\n",
            "Evaluation:   3% 2/63 [00:00<00:23,  2.62it/s]context_input: tensor([[    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3]], device='cuda:0') <class 'torch.Tensor'>\n",
            "candidate_input: tensor([[   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0]], device='cuda:0') <class 'torch.Tensor'>\n",
            "Evaluation: 100% 63/63 [00:18<00:00,  3.49it/s]\n",
            "08/07/2024 14:02:09 - INFO - Blink -   Eval accuracy: 0.08900 (89/1000)\n",
            "08/07/2024 14:02:09 - INFO - Blink -   Starting training\n",
            "08/07/2024 14:02:09 - INFO - Blink -   device: cuda n_gpu: 1, distributed training: False\n",
            "type_optimization: all_encoder_layers\n",
            "The following parameters will be optimized WITH decay:\n",
            "module.context_encoder.bert_model.encoder.layer.0.attention.self.query.weight , module.context_encoder.bert_model.encoder.layer.0.attention.self.key.weight , module.context_encoder.bert_model.encoder.layer.0.attention.self.value.weight , module.context_encoder.bert_model.encoder.layer.0.attention.output.dense.weight , module.context_encoder.bert_model.encoder.layer.0.attention.output.LayerNorm.weight , ...and 187 more\n",
            "The following parameters will be optimized WITHOUT decay:\n",
            "module.context_encoder.bert_model.encoder.layer.0.attention.self.query.bias , module.context_encoder.bert_model.encoder.layer.0.attention.self.key.bias , module.context_encoder.bert_model.encoder.layer.0.attention.self.value.bias , module.context_encoder.bert_model.encoder.layer.0.attention.output.dense.bias , module.context_encoder.bert_model.encoder.layer.0.attention.output.LayerNorm.bias , ...and 187 more\n",
            "08/07/2024 14:02:09 - INFO - Blink -    Num optimization steps = 62\n",
            "08/07/2024 14:02:09 - INFO - Blink -    Num warmup steps = 6\n",
            "Epoch:   0% 0/1 [00:00<?, ?it/s]\n",
            "Batch:   0% 0/63 [00:00<?, ?it/s]\u001b[A/usr/local/lib/python3.10/dist-packages/pytorch_transformers/optimization.py:166: UserWarning: This overload of add_ is deprecated:\n",
            "\tadd_(Number alpha, Tensor other)\n",
            "Consider using one of the following signatures instead:\n",
            "\tadd_(Tensor other, *, Number alpha) (Triggered internally at  ../torch/csrc/utils/python_arg_parser.cpp:1055.)\n",
            "  exp_avg.mul_(beta1).add_(1.0 - beta1, grad)\n",
            "\n",
            "Batch:   2% 1/63 [00:00<00:42,  1.47it/s]\u001b[A\n",
            "Batch:   3% 2/63 [00:01<00:38,  1.58it/s]\u001b[A\n",
            "Batch:   5% 3/63 [00:01<00:37,  1.59it/s]\u001b[A\n",
            "Batch:   6% 4/63 [00:02<00:36,  1.61it/s]\u001b[A\n",
            "Batch:   8% 5/63 [00:03<00:35,  1.61it/s]\u001b[A\n",
            "Batch:  10% 6/63 [00:03<00:35,  1.61it/s]\u001b[A\n",
            "Batch:  11% 7/63 [00:04<00:34,  1.61it/s]\u001b[A\n",
            "Batch:  13% 8/63 [00:04<00:34,  1.61it/s]\u001b[A\n",
            "Batch:  14% 9/63 [00:05<00:33,  1.61it/s]\u001b[A\n",
            "Batch:  16% 10/63 [00:06<00:32,  1.61it/s]\u001b[A\n",
            "Batch:  17% 11/63 [00:06<00:32,  1.62it/s]\u001b[A\n",
            "Batch:  19% 12/63 [00:07<00:31,  1.63it/s]\u001b[A\n",
            "Batch:  21% 13/63 [00:08<00:30,  1.63it/s]\u001b[A\n",
            "Batch:  22% 14/63 [00:08<00:30,  1.62it/s]\u001b[A\n",
            "Batch:  24% 15/63 [00:09<00:29,  1.63it/s]\u001b[A\n",
            "Batch:  25% 16/63 [00:09<00:28,  1.63it/s]\u001b[A\n",
            "Batch:  27% 17/63 [00:10<00:28,  1.63it/s]\u001b[A\n",
            "Batch:  29% 18/63 [00:11<00:27,  1.63it/s]\u001b[A\n",
            "Batch:  30% 19/63 [00:11<00:27,  1.62it/s]\u001b[A\n",
            "Batch:  32% 20/63 [00:12<00:26,  1.62it/s]\u001b[A\n",
            "Batch:  33% 21/63 [00:12<00:25,  1.62it/s]\u001b[A\n",
            "Batch:  35% 22/63 [00:13<00:25,  1.63it/s]\u001b[A\n",
            "Batch:  37% 23/63 [00:14<00:24,  1.63it/s]\u001b[A\n",
            "Batch:  38% 24/63 [00:14<00:23,  1.63it/s]\u001b[A\n",
            "Batch:  40% 25/63 [00:15<00:23,  1.62it/s]\u001b[A\n",
            "Batch:  41% 26/63 [00:16<00:22,  1.61it/s]\u001b[A\n",
            "Batch:  43% 27/63 [00:16<00:22,  1.62it/s]\u001b[A\n",
            "Batch:  44% 28/63 [00:17<00:21,  1.61it/s]\u001b[A\n",
            "Batch:  46% 29/63 [00:17<00:21,  1.61it/s]\u001b[A\n",
            "Batch:  48% 30/63 [00:18<00:20,  1.61it/s]\u001b[A\n",
            "Batch:  49% 31/63 [00:19<00:19,  1.61it/s]\u001b[A\n",
            "Batch:  51% 32/63 [00:19<00:19,  1.60it/s]\u001b[A\n",
            "Batch:  52% 33/63 [00:20<00:18,  1.61it/s]\u001b[A\n",
            "Batch:  54% 34/63 [00:21<00:18,  1.61it/s]\u001b[A\n",
            "Batch:  56% 35/63 [00:21<00:17,  1.61it/s]\u001b[A\n",
            "Batch:  57% 36/63 [00:22<00:16,  1.61it/s]\u001b[A\n",
            "Batch:  59% 37/63 [00:22<00:16,  1.61it/s]\u001b[A\n",
            "Batch:  60% 38/63 [00:23<00:15,  1.61it/s]\u001b[A\n",
            "Batch:  62% 39/63 [00:24<00:14,  1.62it/s]\u001b[A\n",
            "Batch:  63% 40/63 [00:24<00:14,  1.62it/s]\u001b[A\n",
            "Batch:  65% 41/63 [00:25<00:13,  1.62it/s]\u001b[A\n",
            "Batch:  67% 42/63 [00:25<00:13,  1.61it/s]\u001b[A\n",
            "Batch:  68% 43/63 [00:26<00:12,  1.62it/s]\u001b[A\n",
            "Batch:  70% 44/63 [00:27<00:11,  1.61it/s]\u001b[A\n",
            "Batch:  71% 45/63 [00:27<00:11,  1.61it/s]\u001b[A\n",
            "Batch:  73% 46/63 [00:28<00:10,  1.61it/s]\u001b[A\n",
            "Batch:  75% 47/63 [00:29<00:09,  1.61it/s]\u001b[A\n",
            "Batch:  76% 48/63 [00:29<00:09,  1.60it/s]\u001b[A\n",
            "Batch:  78% 49/63 [00:30<00:08,  1.60it/s]\u001b[A\n",
            "Batch:  79% 50/63 [00:30<00:08,  1.59it/s]\u001b[A\n",
            "Batch:  81% 51/63 [00:31<00:07,  1.59it/s]\u001b[A\n",
            "Batch:  83% 52/63 [00:32<00:06,  1.59it/s]\u001b[A\n",
            "Batch:  84% 53/63 [00:32<00:06,  1.59it/s]\u001b[A\n",
            "Batch:  86% 54/63 [00:33<00:05,  1.59it/s]\u001b[A\n",
            "Batch:  87% 55/63 [00:34<00:05,  1.60it/s]\u001b[A\n",
            "Batch:  89% 56/63 [00:34<00:04,  1.60it/s]\u001b[A\n",
            "Batch:  90% 57/63 [00:35<00:03,  1.60it/s]\u001b[A\n",
            "Batch:  92% 58/63 [00:36<00:03,  1.60it/s]\u001b[A\n",
            "Batch:  94% 59/63 [00:36<00:02,  1.61it/s]\u001b[A\n",
            "Batch:  95% 60/63 [00:37<00:01,  1.61it/s]\u001b[A\n",
            "Batch:  97% 61/63 [00:37<00:01,  1.60it/s]\u001b[A\n",
            "Batch:  98% 62/63 [00:38<00:00,  1.60it/s]\u001b[A\n",
            "Batch: 100% 63/63 [00:38<00:00,  1.62it/s]\n",
            "08/07/2024 14:02:48 - INFO - Blink -   ***** Saving fine - tuned model *****\n",
            "\n",
            "Evaluation:   0% 0/63 [00:00<?, ?it/s]\u001b[Acontext_input: tensor([[    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,  2712,  1942,  5761,  3416,  7295,  1930, 13245,  9557,    18,\n",
            "          7213,  7745,  1930,     1, 20999,  3071,  3601,     1,  1985,  2458,\n",
            "          1956, 12454,  1922,  3686,  6687,  7970,  1922,  4201,  3069,    16,\n",
            "          2502,     3],\n",
            "        [    2,     1, 22905,  7868,     1,  4189,  2007, 23025,    17,  4910,\n",
            "         13452,  8849,  2868,  1956,  2823, 13642, 12191,  1007,  2014, 16100,\n",
            "            30,    43,  4067,  7029,  2333,  1927,  7965,  2485,  2037,    43,\n",
            "         13289,     3],\n",
            "        [    2,     1, 22905,  7868,     1,  4189,  2007, 23025,    17,  4910,\n",
            "         13452,  8849,  2868,  1956,  2823, 13642, 12191,  1007,  2014, 16100,\n",
            "            30,    43,  4067,  7029,  2333,  1927,  7965,  2485,  2037,    43,\n",
            "         13289,     3],\n",
            "        [    2,  2634,  5841,  7965,  8422,    12,  4387,  3176,    13,  1930,\n",
            "          1920,  5982,  1927,  8247,     1, 22905,  7868,     1,    12,  5007,\n",
            "            13,  4152,  2007, 23025,    17,  4910, 13452,  8849,    12, 12836,\n",
            "            13,     3],\n",
            "        [    2,  2634,  5841,  7965,  8422,    12,  4387,  3176,    13,  1930,\n",
            "          1920,  5982,  1927,  8247,     1, 22905,  7868,     1,    12,  5007,\n",
            "            13,  4152,  2007, 23025,    17,  4910, 13452,  8849,    12, 12836,\n",
            "            13,     3],\n",
            "        [    2,  8422,    12,  4387,  3176,    13,  1930,  1920,  5982,  1927,\n",
            "          8247, 22905,  7868,    12,     1,  5007,     1,    13,  4152,  2007,\n",
            "         23025,    17,  4910, 13452,  8849,    12, 12836,    13,  1922, 13123,\n",
            "          4440,     3],\n",
            "        [    2,  8422,    12,  4387,  3176,    13,  1930,  1920,  5982,  1927,\n",
            "          8247, 22905,  7868,    12,     1,  5007,     1,    13,  4152,  2007,\n",
            "         23025,    17,  4910, 13452,  8849,    12, 12836,    13,  1922, 13123,\n",
            "          4440,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3]], device='cuda:0') <class 'torch.Tensor'>\n",
            "candidate_input: tensor([[   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        [   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        [   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0]], device='cuda:0') <class 'torch.Tensor'>\n",
            "\n",
            "Evaluation:   2% 1/63 [00:00<00:23,  2.63it/s]\u001b[Acontext_input: tensor([[    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2, 13642, 12191,  1007,  2014, 16100,    18,  1925,  4391,  1977,\n",
            "          2222,  3915,  1927,  1920,     1,  4604,  4574,     1,  2769,  1930,\n",
            "          2712,  7029,  3685,  2192,  2161,  2671,    18,  1925,  7029,  2753,\n",
            "          8352,     3],\n",
            "        [    2,    16,  4825,    17,  7109,    16, 14620,  3193,    24,  2161,\n",
            "          1982,  4371,  2397,    18,     1, 22905,  7868,     1,  2810,  1942,\n",
            "         12836,  1922, 13123,  4440,    18,  1920,  4795, 17777,  1942,  2297,\n",
            "            12,     3],\n",
            "        [    2,    16,  4825,    17,  7109,    16, 14620,  3193,    24,  2161,\n",
            "          1982,  4371,  2397,    18,     1, 22905,  7868,     1,  2810,  1942,\n",
            "         12836,  1922, 13123,  4440,    18,  1920,  4795, 17777,  1942,  2297,\n",
            "            12,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3]], device='cuda:0') <class 'torch.Tensor'>\n",
            "candidate_input: tensor([[   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0]], device='cuda:0') <class 'torch.Tensor'>\n",
            "\n",
            "Evaluation:   3% 2/63 [00:00<00:20,  3.00it/s]\u001b[Acontext_input: tensor([[    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  3108,  2823, 13642, 12191,  1007,  2014, 16100,    16,  1930,\n",
            "          2192,  2132,  2746,  3530,     1,  4604,  4574,     1,  1930,  2690,\n",
            "          2746,  2811,  2084,    18,  2333,  1927,  4387,  3176,  3685,  1930,\n",
            "          5982,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3],\n",
            "        [    2,  1502,  1026,    16, 22421,    13,    12,    58,    33,    18,\n",
            "          5349,    13,    18,  1920,     1,  4604,  4574,     1,  2769,  1982,\n",
            "          2567,  2826,  1922,  1920,  2823, 13642, 12191,  1007,    17,  2868,\n",
            "          2132,     3]], device='cuda:0') <class 'torch.Tensor'>\n",
            "candidate_input: tensor([[   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0]], device='cuda:0') <class 'torch.Tensor'>\n",
            "\n",
            "Evaluation:   5% 3/63 [00:00<00:19,  3.14it/s]\u001b[A\n",
            "Evaluation:   6% 4/63 [00:01<00:18,  3.20it/s]\u001b[A\n",
            "Evaluation:   8% 5/63 [00:01<00:17,  3.25it/s]\u001b[A\n",
            "Evaluation:  10% 6/63 [00:01<00:17,  3.29it/s]\u001b[A\n",
            "Evaluation:  11% 7/63 [00:02<00:16,  3.31it/s]\u001b[A\n",
            "Evaluation:  13% 8/63 [00:02<00:16,  3.32it/s]\u001b[A\n",
            "Evaluation:  14% 9/63 [00:02<00:16,  3.31it/s]\u001b[A\n",
            "Evaluation:  16% 10/63 [00:03<00:15,  3.33it/s]\u001b[A\n",
            "Evaluation:  17% 11/63 [00:03<00:15,  3.35it/s]\u001b[A\n",
            "Evaluation:  19% 12/63 [00:03<00:15,  3.32it/s]\u001b[A\n",
            "Evaluation:  21% 13/63 [00:03<00:15,  3.31it/s]\u001b[A\n",
            "Evaluation:  22% 14/63 [00:04<00:14,  3.33it/s]\u001b[A\n",
            "Evaluation:  24% 15/63 [00:04<00:14,  3.34it/s]\u001b[A\n",
            "Evaluation:  25% 16/63 [00:04<00:14,  3.32it/s]\u001b[A\n",
            "Evaluation:  27% 17/63 [00:05<00:13,  3.32it/s]\u001b[A\n",
            "Evaluation:  29% 18/63 [00:05<00:13,  3.34it/s]\u001b[A\n",
            "Evaluation:  30% 19/63 [00:05<00:13,  3.33it/s]\u001b[A\n",
            "Evaluation:  32% 20/63 [00:06<00:12,  3.34it/s]\u001b[A\n",
            "Evaluation:  33% 21/63 [00:06<00:12,  3.34it/s]\u001b[A\n",
            "Evaluation:  35% 22/63 [00:06<00:12,  3.34it/s]\u001b[A\n",
            "Evaluation:  37% 23/63 [00:06<00:11,  3.34it/s]\u001b[A\n",
            "Evaluation:  38% 24/63 [00:07<00:11,  3.33it/s]\u001b[A\n",
            "Evaluation:  40% 25/63 [00:07<00:11,  3.34it/s]\u001b[A\n",
            "Evaluation:  41% 26/63 [00:07<00:11,  3.33it/s]\u001b[A\n",
            "Evaluation:  43% 27/63 [00:08<00:10,  3.34it/s]\u001b[A\n",
            "Evaluation:  44% 28/63 [00:08<00:10,  3.33it/s]\u001b[A\n",
            "Evaluation:  46% 29/63 [00:08<00:10,  3.34it/s]\u001b[A\n",
            "Evaluation:  48% 30/63 [00:09<00:09,  3.32it/s]\u001b[A\n",
            "Evaluation:  49% 31/63 [00:09<00:09,  3.32it/s]\u001b[A\n",
            "Evaluation:  51% 32/63 [00:09<00:09,  3.33it/s]\u001b[A\n",
            "Evaluation:  52% 33/63 [00:09<00:09,  3.33it/s]\u001b[A\n",
            "Evaluation:  54% 34/63 [00:10<00:08,  3.35it/s]\u001b[A\n",
            "Evaluation:  56% 35/63 [00:10<00:08,  3.36it/s]\u001b[A\n",
            "Evaluation:  57% 36/63 [00:10<00:08,  3.34it/s]\u001b[A\n",
            "Evaluation:  59% 37/63 [00:11<00:07,  3.31it/s]\u001b[A\n",
            "Evaluation:  60% 38/63 [00:11<00:07,  3.32it/s]\u001b[A\n",
            "Evaluation:  62% 39/63 [00:11<00:07,  3.32it/s]\u001b[A\n",
            "Evaluation:  63% 40/63 [00:12<00:06,  3.31it/s]\u001b[A\n",
            "Evaluation:  65% 41/63 [00:12<00:06,  3.31it/s]\u001b[A\n",
            "Evaluation:  67% 42/63 [00:12<00:06,  3.29it/s]\u001b[A\n",
            "Evaluation:  68% 43/63 [00:13<00:06,  3.29it/s]\u001b[A\n",
            "Evaluation:  70% 44/63 [00:13<00:05,  3.28it/s]\u001b[A\n",
            "Evaluation:  71% 45/63 [00:13<00:05,  3.29it/s]\u001b[A\n",
            "Evaluation:  73% 46/63 [00:13<00:05,  3.31it/s]\u001b[A\n",
            "Evaluation:  75% 47/63 [00:14<00:04,  3.29it/s]\u001b[A\n",
            "Evaluation:  76% 48/63 [00:14<00:04,  3.29it/s]\u001b[A\n",
            "Evaluation:  78% 49/63 [00:14<00:04,  3.32it/s]\u001b[A\n",
            "Evaluation:  79% 50/63 [00:15<00:03,  3.33it/s]\u001b[A\n",
            "Evaluation:  81% 51/63 [00:15<00:03,  3.30it/s]\u001b[A\n",
            "Evaluation:  83% 52/63 [00:15<00:03,  3.27it/s]\u001b[A\n",
            "Evaluation:  84% 53/63 [00:16<00:03,  3.28it/s]\u001b[A\n",
            "Evaluation:  86% 54/63 [00:16<00:02,  3.28it/s]\u001b[A\n",
            "Evaluation:  87% 55/63 [00:16<00:02,  3.25it/s]\u001b[A\n",
            "Evaluation:  89% 56/63 [00:16<00:02,  3.26it/s]\u001b[A\n",
            "Evaluation:  90% 57/63 [00:17<00:01,  3.27it/s]\u001b[A\n",
            "Evaluation:  92% 58/63 [00:17<00:01,  3.26it/s]\u001b[A\n",
            "Evaluation:  94% 59/63 [00:17<00:01,  3.27it/s]\u001b[A\n",
            "Evaluation:  95% 60/63 [00:18<00:00,  3.28it/s]\u001b[A\n",
            "Evaluation:  97% 61/63 [00:18<00:00,  3.28it/s]\u001b[A\n",
            "Evaluation:  98% 62/63 [00:18<00:00,  3.27it/s]\u001b[A\n",
            "Evaluation: 100% 63/63 [00:18<00:00,  3.32it/s]\n",
            "08/07/2024 14:03:10 - INFO - Blink -   Eval accuracy: 0.09300 (93/1000)\n",
            "08/07/2024 14:03:10 - INFO - Blink -   \n",
            "\n",
            "Epoch: 100% 1/1 [01:00<00:00, 60.83s/it]\n",
            "08/07/2024 14:03:10 - INFO - Blink -   The training took 1.0138350049654643 minutes\n",
            "\n",
            "08/07/2024 14:03:10 - INFO - Blink -   Best performance in epoch 0: 0.093\n",
            "self.n_gpu: 1\n",
            "self.device in BiEncoderRanker: cuda\n",
            "params[\"lowercase\"]: True\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model built\n",
            "model loaded\n",
            "moving model to cuda\n",
            "model moved to cuda\n",
            "self.n_gpu: 1\n",
            "self.device in BiEncoderRanker: cuda\n",
            "params[\"lowercase\"]: True\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model built\n",
            "model loaded\n",
            "moving model to cuda\n",
            "model moved to cuda\n",
            "237826it [00:01, 221357.44it/s]\n",
            "100% 237826/237826 [00:40<00:00, 5900.27it/s]\n",
            "tensor([[    2,  4543,  7465,  ...,     0,     0,     0],\n",
            "        [    2,  9019,    12,  ...,     0,     0,     0],\n",
            "        [    2,  4543,  9019,  ...,     0,     0,     0],\n",
            "        ...,\n",
            "        [    2, 15126,  1988,  ...,     0,     0,     0],\n",
            "        [    2,  3700,     1,  ...,     0,     0,     0],\n",
            "        [    2, 15126,  1988,  ...,     0,     0,     0]])\n",
            "self.n_gpu: 1\n",
            "self.device in BiEncoderRanker: cuda\n",
            "params[\"lowercase\"]: True\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model built\n",
            "model loaded\n",
            "moving model to cuda\n",
            "model moved to cuda\n",
            "08/07/2024 14:04:45 - INFO - Blink -   Loading pre-generated candidate pool from: \n",
            "08/07/2024 14:04:45 - INFO - Blink -   preprocessing/saved_cand_ids_snomedct-us-20140901-disease_edges_all-sapbert_re_tr.pt\n",
            "number of candidates in candidate_pool: 237826\n",
            "100% 4757/4757 [31:44<00:00,  2.50it/s]\n",
            "Saving in: models/SNOMEDCT-US-20140901-Disease_ent_enc_re_tr/SNOMEDCT-US-20140901-Disease_edges_all-sapbert_ent_enc_re_tr.t7\n",
            "tensor([-0.6992, -0.1403,  0.0474,  0.3139, -0.3139, -0.1518, -0.3242, -0.2332,\n",
            "         0.3976,  0.6963])\n",
            "INFO:deeponto:32g maximum memory allocated to JVM.\n",
            "INFO:deeponto:JVM started successfully.\n",
            "2024-08-07 14:36:44.554061: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2024-08-07 14:36:44.849863: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2024-08-07 14:36:44.925991: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2024-08-07 14:36:48.243664: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "INFO:numexpr.utils:NumExpr defaulting to 2 threads.\n",
            "Namespace(silent=False, debug=True, debug_max_lines=1000, data_parallel=True, no_cuda=False, top_k=50, seed=52313, zeshel=False, max_seq_length=256, max_context_length=32, max_cand_length=128, path_to_model='models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/pytorch_model.bin', bert_model='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', pull_from_layer=-1, lowercase=True, context_key='context', use_context=True, out_dim=1, add_linear=False, data_path='data/MedMentions-preprocessed+/Disease/st21pv_syn_attr-all-complexEdge-edges-final', output_path='models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug', use_triplet_loss_bi_enc=False, use_miner_bi_enc=False, use_ori_classification=False, use_NIL_classification=False, lambda_NIL=0.25, use_score_features=False, use_score_pooling=False, use_men_only_score_ft=False, use_extra_features=False, use_BM25=False, aggregating_factor=1, eval_batch_size=8, mode='train,valid', save_topk_result=True, encode_batch_size=8, use_synonyms=False, use_NIL_tag=False, use_NIL_desc=False, use_NIL_desc_tag=False, cand_pool_path='preprocessing/saved_cand_ids_snomedct-us-20140901-disease_edges_all-sapbert_re_tr.pt', cand_encode_path='models/SNOMEDCT-US-20140901-Disease_ent_enc_re_tr/SNOMEDCT-US-20140901-Disease_edges_all-sapbert_ent_enc_re_tr.t7', use_NIL_classification_infer=False, add_NIL_to_bi_enc_pred=True, NIL_ent_ind=64076, entity_dict_path='ontologies/SNOMEDCT-US-20140901-Disease-edges-all.jsonl', cand_pool_path_for_BM25=None, save_all_predictions=False, edge_cand_enrich=True, edge_catalogue_fn='ontologies/SNOMEDCT-US-20140901-Disease-edges-all.jsonl', top_k_cand_seed=25, use_leaf_edge_score=False, LEAF_EDGE_SCORE=1000, edge_ranking_by_score=True, use_fix_embs=False, debug_random_sample=False)\n",
            "self.n_gpu: 1\n",
            "self.device in BiEncoderRanker: cuda\n",
            "params[\"lowercase\"]: True\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model built\n",
            "model loaded\n",
            "moving model to cuda\n",
            "model moved to cuda\n",
            "INFO:Blink:Loading pre-generated candidate pool from: \n",
            "INFO:Blink:preprocessing/saved_cand_ids_snomedct-us-20140901-disease_edges_all-sapbert_re_tr.pt\n",
            "INFO:Blink:Loading pre-generated candidate encode path.\n",
            "local_id2wikipedia_id: 237826\n",
            "wikipedia_id2local_id: 237826\n",
            "id2synonyms: 237826\n",
            "  0% 999/887840 [00:00<00:30, 29044.74it/s]\n",
            "INFO:Blink:Read 1000 test samples.\n",
            "num of samples: 1000\n",
            "100% 1000/1000 [00:01<00:00, 557.42it/s]\n",
            "INFO:Blink:====Processed samples (first 10): ====\n",
            "INFO:Blink:Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "INFO:Blink:Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "INFO:Blink:Context ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "INFO:Blink:Contextual tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "INFO:Blink:Contextual ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis without mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 2979 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40179 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "INFO:Blink:Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "INFO:Blink:Context ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "INFO:Blink:Contextual tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "INFO:Blink:Contextual ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with intestinal manifestations ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 6139 10952 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40180 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "INFO:Blink:Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "INFO:Blink:Context ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "INFO:Blink:Contextual tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "INFO:Blink:Contextual ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40181 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "INFO:Blink:Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "INFO:Blink:Context ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "INFO:Blink:Contextual tokens : [CLS] dct ##n4 as a modifier of chronic pseudomonas aeruginosa infection in [unused0] cystic fibrosis [unused1] pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( cf ) patients is associated [SEP]\n",
            "INFO:Blink:Contextual ids : 2 22936 15580 1966 43 22883 1927 3724 13461 10830 2909 1922 1 11656 6769 1 13461 10830 12 3838 13 2909 1922 11656 6769 12 4920 13 2132 1977 2458 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis of the lung ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1927 1920 3614 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40182 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "INFO:Blink:Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "INFO:Blink:Context ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "INFO:Blink:Contextual tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "INFO:Blink:Contextual ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis without mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 2979 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40179 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "INFO:Blink:Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "INFO:Blink:Context ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "INFO:Blink:Contextual tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "INFO:Blink:Contextual ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with intestinal manifestations ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 6139 10952 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40180 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "INFO:Blink:Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "INFO:Blink:Context ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "INFO:Blink:Contextual tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "INFO:Blink:Contextual ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40181 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cystic fibrosis [unused1]\n",
            "INFO:Blink:Mention ids : 1 11656 6769 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "INFO:Blink:Context ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "INFO:Blink:Contextual tokens : [CLS] pseudomonas aeruginosa infection in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in [unused0] cystic fibrosis [unused1] ( cf ) patients is associated with worse long - term pulmonary disease [SEP]\n",
            "INFO:Blink:Contextual ids : 2 13461 10830 2909 1922 11656 6769 13461 10830 12 3838 13 2909 1922 1 11656 6769 1 12 4920 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis of the lung ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1927 1920 3614 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40182 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cf [unused1]\n",
            "INFO:Blink:Mention ids : 1 4920 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( [unused0] cf [unused1] ) patients is associated with worse long - term pulmonary disease and shorter survival [SEP]\n",
            "INFO:Blink:Context ids : 2 1922 11656 6769 13461 10830 12 3838 13 2909 1922 11656 6769 12 1 4920 1 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 1930 7310 3298 3\n",
            "INFO:Blink:Contextual tokens : [CLS] in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( [unused0] cf [unused1] ) patients is associated with worse long - term pulmonary disease and shorter survival [SEP]\n",
            "INFO:Blink:Contextual ids : 2 1922 11656 6769 13461 10830 12 3838 13 2909 1922 11656 6769 12 1 4920 1 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 1930 7310 3298 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis without mec ##onium ile ##us ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 2979 10685 15414 9940 1957 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40179 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cf [unused1]\n",
            "INFO:Blink:Mention ids : 1 4920 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 190 ##90 ##500 ##8\n",
            "INFO:Blink:Concept ID ori ids : 11981 5124 17877 1025 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( [unused0] cf [unused1] ) patients is associated with worse long - term pulmonary disease and shorter survival [SEP]\n",
            "INFO:Blink:Context ids : 2 1922 11656 6769 13461 10830 12 3838 13 2909 1922 11656 6769 12 1 4920 1 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 1930 7310 3298 3\n",
            "INFO:Blink:Contextual tokens : [CLS] in cystic fibrosis pseudomonas aeruginosa ( pa ) infection in cystic fibrosis ( [unused0] cf [unused1] ) patients is associated with worse long - term pulmonary disease and shorter survival [SEP]\n",
            "INFO:Blink:Contextual ids : 2 1922 11656 6769 13461 10830 12 3838 13 2909 1922 11656 6769 12 1 4920 1 13 2132 1977 2458 1956 9190 2815 17 2576 5352 2573 1930 7310 3298 3\n",
            "INFO:Blink:Label tokens : [CLS] inherited muc ##oci ##li ##ary clearance defect ( disorder ) [unused5] cystic fibrosis with intestinal manifestations ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 12911 3422 2240 12191 2141 7802 7374 12 4770 13 1 11656 6769 1956 6139 10952 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 40180 is in-KB\n",
            "label_text_vecs: tensor([[    2, 12911,  3422,  ...,     0,     0,     0],\n",
            "        [    2, 12911,  3422,  ...,     0,     0,     0],\n",
            "        [    2, 12911,  3422,  ...,     0,     0,     0],\n",
            "        ...,\n",
            "        [    2,  2535,  2849,  ...,     0,     0,     0],\n",
            "        [    2,  2535,  2849,  ...,     0,     0,     0],\n",
            "        [    2,  2535,  2849,  ...,     0,     0,     0]])\n",
            "INFO:Blink:Getting top 50 predictions.\n",
            "INFO:Blink:Enriched from top 25 predictions.\n",
            "  0% 0/125 [00:00<?, ?it/s]\n",
            "  0% 0/237826 [00:00<?, ?it/s]\u001b[A\n",
            "  4% 9784/237826 [00:00<00:02, 97832.63it/s]\u001b[A\n",
            "  9% 21641/237826 [00:00<00:01, 110021.35it/s]\u001b[A\n",
            " 14% 34039/237826 [00:00<00:01, 116189.28it/s]\u001b[A\n",
            " 19% 45861/237826 [00:00<00:01, 116984.20it/s]\u001b[A\n",
            " 24% 57559/237826 [00:00<00:01, 115625.50it/s]\u001b[A\n",
            " 29% 69299/237826 [00:00<00:01, 116219.26it/s]\u001b[A\n",
            " 34% 80923/237826 [00:00<00:01, 110974.65it/s]\u001b[A\n",
            " 39% 92065/237826 [00:00<00:01, 108964.53it/s]\u001b[A\n",
            " 43% 102993/237826 [00:00<00:01, 106213.96it/s]\u001b[A\n",
            " 48% 114137/237826 [00:01<00:01, 107715.42it/s]\u001b[A\n",
            " 53% 125474/237826 [00:01<00:01, 109388.71it/s]\u001b[A\n",
            " 57% 136436/237826 [00:01<00:00, 103754.13it/s]\u001b[A\n",
            " 62% 148030/237826 [00:01<00:00, 107249.79it/s]\u001b[A\n",
            " 67% 159002/237826 [00:01<00:00, 107965.27it/s]\u001b[A\n",
            " 72% 170428/237826 [00:01<00:00, 109808.38it/s]\u001b[A\n",
            " 76% 181448/237826 [00:01<00:00, 106814.47it/s]\u001b[A\n",
            " 81% 192171/237826 [00:01<00:00, 104833.06it/s]\u001b[A\n",
            " 85% 202686/237826 [00:01<00:00, 104116.89it/s]\u001b[A\n",
            " 90% 214387/237826 [00:01<00:00, 107863.72it/s]\u001b[A\n",
            " 95% 225200/237826 [00:02<00:00, 104787.70it/s]\u001b[A\n",
            "100% 237826/237826 [00:02<00:00, 108475.22it/s]\n",
            "list_inds_enriched_w_scores, sorted: [(40179, 228.56801), (40181, 228.24246), (40180, 228.12271), (72645, 227.48216), (40174, 227.20724), (72648, 226.57968), (72649, 226.3811), (40182, 225.51056), (40175, 223.99567), (40177, 223.57027), (40185, 217.47426), (40176, 216.97263), (72641, 215.89331), (220566, 215.60147), (72659, 215.17772), (72666, 214.41641), (72640, 214.38213), (40178, 214.32788), (223369, 213.54407), (223659, 213.35146), (223655, 213.16039), (72670, 212.78215), (72664, 212.67635), (72669, 212.03261), (77612, 211.39725), (223656, 209.22937), (72644, 208.61026), (72646, 208.24069), (223657, 208.17494), (72643, 208.14946), (72667, 206.94278), (111607, 206.92871), (223658, 206.10762), (72647, 205.75162), (72665, 205.74646), (72650, 204.69855), (72639, 203.86151), (223891, 203.45708), (40184, 202.89297), (55122, 202.7013), (237791, 202.7013), (72642, 202.33514), (177519, 202.19394), (55120, 201.08905), (177521, 200.99469), (248, 83.1364), (223370, 76.54911), (4450, 68.97248), (45953, 66.9971), (55182, 61.05818)]\n",
            "list_inds_enriched_w_scores, sorted: [(40179, 228.56801), (40181, 228.24246), (40180, 228.12271), (72645, 227.48216), (40174, 227.20724), (72648, 226.57968), (72649, 226.3811), (40182, 225.51056), (40175, 223.99567), (40177, 223.57027), (40185, 217.47426), (40176, 216.97263), (72641, 215.89331), (220566, 215.60147), (72659, 215.17772), (72666, 214.41641), (72640, 214.38213), (40178, 214.32788), (223369, 213.54407), (223659, 213.35146), (223655, 213.16039), (72670, 212.78215), (72664, 212.67635), (72669, 212.03261), (77612, 211.39725), (223656, 209.22937), (72644, 208.61026), (72646, 208.24069), (223657, 208.17494), (72643, 208.14946), (72667, 206.94278), (111607, 206.92871), (223658, 206.10762), (72647, 205.75162), (72665, 205.74646), (72650, 204.69855), (72639, 203.86151), (223891, 203.45708), (40184, 202.89297), (55122, 202.7013), (237791, 202.7013), (72642, 202.33514), (177519, 202.19394), (55120, 201.08905), (177521, 200.99469), (248, 83.1364), (223370, 76.54911), (4450, 68.97248), (45953, 66.9971), (55182, 61.05818)]\n",
            "list_inds_enriched_w_scores, sorted: [(40179, 228.56801), (40181, 228.24246), (40180, 228.12271), (72645, 227.48216), (40174, 227.20724), (72648, 226.57968), (72649, 226.3811), (40182, 225.51056), (40175, 223.99567), (40177, 223.57027), (40185, 217.47426), (40176, 216.97263), (72641, 215.89331), (220566, 215.60147), (72659, 215.17772), (72666, 214.41641), (72640, 214.38213), (40178, 214.32788), (223369, 213.54407), (223659, 213.35146), (223655, 213.16039), (72670, 212.78215), (72664, 212.67635), (72669, 212.03261), (77612, 211.39725), (223656, 209.22937), (72644, 208.61026), (72646, 208.24069), (223657, 208.17494), (72643, 208.14946), (72667, 206.94278), (111607, 206.92871), (223658, 206.10762), (72647, 205.75162), (72665, 205.74646), (72650, 204.69855), (72639, 203.86151), (223891, 203.45708), (40184, 202.89297), (55122, 202.7013), (237791, 202.7013), (72642, 202.33514), (177519, 202.19394), (55120, 201.08905), (177521, 200.99469), (248, 83.1364), (223370, 76.54911), (4450, 68.97248), (45953, 66.9971), (55182, 61.05818)]\n",
            "list_inds_enriched_w_scores, sorted: [(40179, 228.56801), (40181, 228.24246), (40180, 228.12271), (72645, 227.48216), (40174, 227.20724), (72648, 226.57968), (72649, 226.3811), (40182, 225.51056), (40175, 223.99567), (40177, 223.57027), (40185, 217.47426), (40176, 216.97263), (72641, 215.89331), (220566, 215.60147), (72659, 215.17772), (72666, 214.41641), (72640, 214.38213), (40178, 214.32788), (223369, 213.54407), (223659, 213.35146), (223655, 213.16039), (72670, 212.78215), (72664, 212.67635), (72669, 212.03261), (77612, 211.39725), (223656, 209.22937), (72644, 208.61026), (72646, 208.24069), (223657, 208.17494), (72643, 208.14946), (72667, 206.94278), (111607, 206.92871), (223658, 206.10762), (72647, 205.75162), (72665, 205.74646), (72650, 204.69855), (72639, 203.86151), (223891, 203.45708), (40184, 202.89297), (55122, 202.7013), (237791, 202.7013), (72642, 202.33514), (177519, 202.19394), (55120, 201.08905), (177521, 200.99469), (248, 83.1364), (223370, 76.54911), (4450, 68.97248), (45953, 66.9971), (55182, 61.05818)]\n",
            "list_inds_enriched_w_scores, sorted: [(72645, 230.69049), (40180, 229.93813), (72648, 229.83774), (40174, 229.41113), (72649, 228.57668), (40179, 228.41211), (40181, 228.20119), (40177, 227.59583), (40182, 227.1579), (40175, 227.14369), (40176, 222.79729), (40185, 221.27919), (40178, 219.93732), (220566, 218.86774), (223369, 218.45209), (77612, 218.02509), (72641, 217.1734), (72659, 216.68314), (72640, 216.41502), (72666, 215.98633), (223659, 215.41257), (72664, 215.3151), (223655, 215.22122), (72670, 215.05745), (72669, 215.00037), (111607, 211.99252), (72643, 211.92319), (223656, 211.74545), (72644, 210.68999), (223657, 210.54932), (72667, 209.84293), (72646, 209.66226), (40184, 209.60687), (72665, 209.5154), (223658, 209.29114), (55120, 209.22136), (55122, 208.93623), (237791, 208.93623), (177519, 208.312), (72647, 208.27069), (72639, 208.01166), (223891, 207.76831), (55172, 207.6327), (72642, 207.44504), (72650, 206.70688), (248, 85.97899), (223370, 81.60027), (4450, 73.49018), (45953, 68.93336), (55182, 64.66751)]\n",
            "list_inds_enriched_w_scores, sorted: [(72645, 230.69049), (40180, 229.93813), (72648, 229.83774), (40174, 229.41113), (72649, 228.57668), (40179, 228.41211), (40181, 228.20119), (40177, 227.59583), (40182, 227.1579), (40175, 227.14369), (40176, 222.79729), (40185, 221.27919), (40178, 219.93732), (220566, 218.86774), (223369, 218.45209), (77612, 218.02509), (72641, 217.1734), (72659, 216.68314), (72640, 216.41502), (72666, 215.98633), (223659, 215.41257), (72664, 215.3151), (223655, 215.22122), (72670, 215.05745), (72669, 215.00037), (111607, 211.99252), (72643, 211.92319), (223656, 211.74545), (72644, 210.68999), (223657, 210.54932), (72667, 209.84293), (72646, 209.66226), (40184, 209.60687), (72665, 209.5154), (223658, 209.29114), (55120, 209.22136), (55122, 208.93623), (237791, 208.93623), (177519, 208.312), (72647, 208.27069), (72639, 208.01166), (223891, 207.76831), (55172, 207.6327), (72642, 207.44504), (72650, 206.70688), (248, 85.97899), (223370, 81.60027), (4450, 73.49018), (45953, 68.93336), (55182, 64.66751)]\n",
            "list_inds_enriched_w_scores, sorted: [(72645, 230.69049), (40180, 229.93813), (72648, 229.83774), (40174, 229.41113), (72649, 228.57668), (40179, 228.41211), (40181, 228.20119), (40177, 227.59583), (40182, 227.1579), (40175, 227.14369), (40176, 222.79729), (40185, 221.27919), (40178, 219.93732), (220566, 218.86774), (223369, 218.45209), (77612, 218.02509), (72641, 217.1734), (72659, 216.68314), (72640, 216.41502), (72666, 215.98633), (223659, 215.41257), (72664, 215.3151), (223655, 215.22122), (72670, 215.05745), (72669, 215.00037), (111607, 211.99252), (72643, 211.92319), (223656, 211.74545), (72644, 210.68999), (223657, 210.54932), (72667, 209.84293), (72646, 209.66226), (40184, 209.60687), (72665, 209.5154), (223658, 209.29114), (55120, 209.22136), (55122, 208.93623), (237791, 208.93623), (177519, 208.312), (72647, 208.27069), (72639, 208.01166), (223891, 207.76831), (55172, 207.6327), (72642, 207.44504), (72650, 206.70688), (248, 85.97899), (223370, 81.60027), (4450, 73.49018), (45953, 68.93336), (55182, 64.66751)]\n",
            "list_inds_enriched_w_scores, sorted: [(72645, 230.69049), (40180, 229.93813), (72648, 229.83774), (40174, 229.41113), (72649, 228.57668), (40179, 228.41211), (40181, 228.20119), (40177, 227.59583), (40182, 227.1579), (40175, 227.14369), (40176, 222.79729), (40185, 221.27919), (40178, 219.93732), (220566, 218.86774), (223369, 218.45209), (77612, 218.02509), (72641, 217.1734), (72659, 216.68314), (72640, 216.41502), (72666, 215.98633), (223659, 215.41257), (72664, 215.3151), (223655, 215.22122), (72670, 215.05745), (72669, 215.00037), (111607, 211.99252), (72643, 211.92319), (223656, 211.74545), (72644, 210.68999), (223657, 210.54932), (72667, 209.84293), (72646, 209.66226), (40184, 209.60687), (72665, 209.5154), (223658, 209.29114), (55120, 209.22136), (55122, 208.93623), (237791, 208.93623), (177519, 208.312), (72647, 208.27069), (72639, 208.01166), (223891, 207.76831), (55172, 207.6327), (72642, 207.44504), (72650, 206.70688), (248, 85.97899), (223370, 81.60027), (4450, 73.49018), (45953, 68.93336), (55182, 64.66751)]\n",
            "  1% 1/125 [00:02<05:33,  2.69s/it]list_inds_enriched_w_scores, sorted: [(40180, 238.2767), (72645, 237.89174), (40174, 237.65376), (40179, 237.289), (40181, 237.1248), (72648, 236.34137), (72649, 235.99207), (40182, 235.58241), (40175, 230.64542), (40177, 230.33301), (72641, 225.98093), (40176, 225.37518), (72659, 224.67477), (72666, 224.07242), (72640, 224.04213), (223659, 223.69427), (40185, 223.23267), (223655, 222.84444), (40178, 222.60294), (72664, 222.3706), (72670, 221.88129), (220566, 221.69366), (72669, 221.54211), (223656, 220.89761), (223369, 220.22049), (72644, 219.90909), (223657, 218.85956), (72667, 218.45615), (72646, 218.44783), (223658, 218.05835), (72643, 217.95117), (77612, 217.4476), (72665, 217.02463), (111607, 216.99545), (72647, 216.92111), (72650, 214.69159), (72639, 213.361), (72642, 211.83928), (40184, 211.13957), (223891, 210.14647), (55120, 208.52992), (55122, 208.486), (237791, 208.486), (177519, 208.34387), (177521, 206.66617), (248, 82.147415), (223370, 78.87257), (4450, 70.260056), (45953, 67.5547), (55182, 62.541428)]\n",
            "list_inds_enriched_w_scores, sorted: [(40180, 238.2767), (72645, 237.89174), (40174, 237.65376), (40179, 237.289), (40181, 237.1248), (72648, 236.34137), (72649, 235.99207), (40182, 235.58241), (40175, 230.64542), (40177, 230.33301), (72641, 225.98093), (40176, 225.37518), (72659, 224.67477), (72666, 224.07242), (72640, 224.04213), (223659, 223.69427), (40185, 223.23267), (223655, 222.84444), (40178, 222.60294), (72664, 222.3706), (72670, 221.88129), (220566, 221.69366), (72669, 221.54211), (223656, 220.89761), (223369, 220.22049), (72644, 219.90909), (223657, 218.85956), (72667, 218.45615), (72646, 218.44783), (223658, 218.05835), (72643, 217.95117), (77612, 217.4476), (72665, 217.02463), (111607, 216.99545), (72647, 216.92111), (72650, 214.69159), (72639, 213.361), (72642, 211.83928), (40184, 211.13957), (223891, 210.14647), (55120, 208.52992), (55122, 208.486), (237791, 208.486), (177519, 208.34387), (177521, 206.66617), (248, 82.147415), (223370, 78.87257), (4450, 70.260056), (45953, 67.5547), (55182, 62.541428)]\n",
            "100% 125/125 [00:07<00:00, 17.18it/s]\n",
            "nn_data['labels']: tensor([ 0,  2,  1,  7,  5,  1,  6,  8,  3,  0,  4,  7,  6, 34, 35, 39, 17,  9,\n",
            "         8,  7, 21, 37, 40, 27, 41, 42, 28,  0, 11, 18, 32, 25, 12, 44, 47, 31,\n",
            "        46, 15, 29, 38,  3, 45,  5, 36, 48, 49, 10,  1,  2,  0,  3, 36, 24,  5,\n",
            "        10,  2, 18, 22, 19, 12, 25, 41,  8, 37, 39,  9,  7, 44, 13, 30, 35,  6,\n",
            "        23, 27, 21, 11, 17, 14, 16, 15, 26, 29,  1, 40, 28,  4,  3,  0,  2,  1,\n",
            "         5,  1,  2,  0,  5, 33, 49, 47, 20, 26,  2,  1,  2,  0,  5,  0,  2,  1,\n",
            "         5,  0,  3,  1,  7,  0,  2,  1,  5,  0,  3,  1,  7,  0,  2,  1,  7, 20,\n",
            "        20, 36,  6,  7, 17, 10, 27, 13, 45, 26, 21,  3, 18, 23, 39, 29, 28, 12,\n",
            "        24,  8,  5, 14,  5, 23, 10, 16, 26, 19, 43, 28, 13,  8, 24, 39, 29, 27,\n",
            "        17, 12, 25, 22,  0, 20,  3,  2,  4, 20, 16, 41, 25, 45, 37, 33, 17,  6,\n",
            "        31, 24, 15, 27,  8, 11,  0,  9, 10, 35, 28, 23, 46, 17, 19, 24,  7, 33,\n",
            "         6, 43, 26, 13, 27, 38, 40, 35, 30, 34, 37, 10,  5, 45, 22, 32,  4, 47,\n",
            "         0, 18, 48, 31, 41, 16, 25,  3,  7, 30, 40, 14,  6, 42, 27, 25, 33,  5,\n",
            "         9, 38, 45, 35, 29, 37, 12, 39, 38, 37, 32, 22, 41, 18, 46, 48,  3, 27,\n",
            "        20, 34, 17, 10, 36, 40, 25, 35, 11, 22, 31, 33, 16, 21, 38, 32,  2, 13,\n",
            "        37, 30,  5,  6, 47,  0, 26,  4, 45,  9, 49,  8, 24, 43, 17,  3,  2,  3,\n",
            "         0,  4,  1,  6,  1,  3,  0, 17,  4, 23,  9, 12, 36, 29, 16,  3,  6, 38,\n",
            "        14, 24,  1, 41,  5,  0, 45, 31, 42, 10, 19, 35, 15, 18,  7,  8,  2, 11,\n",
            "        36, 26, 29, 47,  5, 30, 13,  3, 26, 23, 43,  2,  9, 27, 32, 48])\n",
            "INFO:Blink:num of nn_data: 340\n",
            "INFO:Blink:biencoder recall@k: 0.34 (340/1000)\n",
            "data saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top50_candidates/train.t7\n",
            "self.n_gpu: 1\n",
            "self.device in BiEncoderRanker: cuda\n",
            "params[\"lowercase\"]: True\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model built\n",
            "model loaded\n",
            "moving model to cuda\n",
            "model moved to cuda\n",
            "INFO:Blink:Loading pre-generated candidate pool from: \n",
            "INFO:Blink:preprocessing/saved_cand_ids_snomedct-us-20140901-disease_edges_all-sapbert_re_tr.pt\n",
            "INFO:Blink:Loading pre-generated candidate encode path.\n",
            "local_id2wikipedia_id: 237826\n",
            "wikipedia_id2local_id: 237826\n",
            "id2synonyms: 237826\n",
            "  0% 999/383457 [00:00<00:03, 103564.34it/s]\n",
            "INFO:Blink:Read 1000 test samples.\n",
            "num of samples: 1000\n",
            "100% 1000/1000 [00:15<00:00, 63.63it/s]\n",
            "INFO:Blink:====Processed samples (first 10): ====\n",
            "INFO:Blink:Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "INFO:Blink:Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 475 ##05 ##003\n",
            "INFO:Blink:Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] stock ##holm syndrome ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 5592 4770 12 4770 13 1 8260 23608 4741 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 189459 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "INFO:Blink:Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 475 ##05 ##003\n",
            "INFO:Blink:Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] acute post - trauma stress state ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 5592 4770 12 4770 13 1 3939 2753 17 7176 3071 3360 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 189460 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "INFO:Blink:Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 475 ##05 ##003\n",
            "INFO:Blink:Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] acute posttraumatic stress disorder following military combat ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 5592 4770 12 4770 13 1 3939 20999 3071 4770 2894 15720 18565 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 189461 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "INFO:Blink:Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 475 ##05 ##003\n",
            "INFO:Blink:Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] chronic post - traumatic stress disorder ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 5592 4770 12 4770 13 1 3724 2753 17 10242 3071 4770 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 189462 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "INFO:Blink:Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 475 ##05 ##003\n",
            "INFO:Blink:Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] posttraumatic stress disorder , delayed onset ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 5592 4770 12 4770 13 1 20999 3071 4770 16 7083 4727 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 189463 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] posttraumatic stress symptoms [unused1]\n",
            "INFO:Blink:Mention ids : 1 20999 3071 3601 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 475 ##05 ##003\n",
            "INFO:Blink:Concept ID ori ids : 28005 15264 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Context ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Contextual tokens : [CLS] related to impaired family functioning and marital instability . parental depressive and [unused0] posttraumatic stress symptoms [unused1] were associated with impairments in social emotional adjustment in young children , increased [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2712 1942 5761 3416 7295 1930 13245 9557 18 7213 7745 1930 1 20999 3071 3601 1 1985 2458 1956 12454 1922 3686 6687 7970 1922 4201 3069 16 2502 3\n",
            "INFO:Blink:Label tokens : [CLS] anxiety disorder ( disorder ) [unused5] complex posttraumatic stress disorder ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 5592 4770 12 4770 13 1 2796 20999 3071 4770 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 189464 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] nosocomial pneumonia [unused1]\n",
            "INFO:Blink:Mention ids : 1 22905 7868 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 425 ##46 ##400 ##7\n",
            "INFO:Blink:Concept ID ori ids : 24592 7772 10893 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] [unused0] nosocomial pneumonia [unused1] caused by methicillin - resistant staphylococcus aureus treated with line ##zo ##li ##d or vancomycin : a secondary economic analysis of resource use from a spanish [SEP]\n",
            "INFO:Blink:Context ids : 2 1 22905 7868 1 4189 2007 23025 17 4910 13452 8849 2868 1956 2823 13642 12191 1007 2014 16100 30 43 4067 7029 2333 1927 7965 2485 2037 43 13289 3\n",
            "INFO:Blink:Contextual tokens : [CLS] [unused0] nosocomial pneumonia [unused1] caused by methicillin - resistant staphylococcus aureus treated with line ##zo ##li ##d or vancomycin : a secondary economic analysis of resource use from a spanish [SEP]\n",
            "INFO:Blink:Contextual ids : 2 1 22905 7868 1 4189 2007 23025 17 4910 13452 8849 2868 1956 2823 13642 12191 1007 2014 16100 30 43 4067 7029 2333 1927 7965 2485 2037 43 13289 3\n",
            "INFO:Blink:Label tokens : [CLS] infective pneumonia ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 20905 7868 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 72468 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] nosocomial pneumonia [unused1]\n",
            "INFO:Blink:Mention ids : 1 22905 7868 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 425 ##46 ##400 ##7\n",
            "INFO:Blink:Concept ID ori ids : 24592 7772 10893 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] [unused0] nosocomial pneumonia [unused1] caused by methicillin - resistant staphylococcus aureus treated with line ##zo ##li ##d or vancomycin : a secondary economic analysis of resource use from a spanish [SEP]\n",
            "INFO:Blink:Context ids : 2 1 22905 7868 1 4189 2007 23025 17 4910 13452 8849 2868 1956 2823 13642 12191 1007 2014 16100 30 43 4067 7029 2333 1927 7965 2485 2037 43 13289 3\n",
            "INFO:Blink:Contextual tokens : [CLS] [unused0] nosocomial pneumonia [unused1] caused by methicillin - resistant staphylococcus aureus treated with line ##zo ##li ##d or vancomycin : a secondary economic analysis of resource use from a spanish [SEP]\n",
            "INFO:Blink:Contextual ids : 2 1 22905 7868 1 4189 2007 23025 17 4910 13452 8849 2868 1956 2823 13642 12191 1007 2014 16100 30 43 4067 7029 2333 1927 7965 2485 2037 43 13289 3\n",
            "INFO:Blink:Label tokens : [CLS] nosocomial infectious disease ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 22905 7465 2573 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 177009 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] nosocomial pneumonia [unused1]\n",
            "INFO:Blink:Mention ids : 1 22905 7868 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 425 ##46 ##400 ##7\n",
            "INFO:Blink:Concept ID ori ids : 24592 7772 10893 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] assess healthcare resource utilization ( hc ##ru ) and the costs of treating [unused0] nosocomial pneumonia [unused1] ( np ) produced by methicillin - resistant staphylococcus aureus ( mrsa ) [SEP]\n",
            "INFO:Blink:Context ids : 2 2634 5841 7965 8422 12 4387 3176 13 1930 1920 5982 1927 8247 1 22905 7868 1 12 5007 13 4152 2007 23025 17 4910 13452 8849 12 12836 13 3\n",
            "INFO:Blink:Contextual tokens : [CLS] assess healthcare resource utilization ( hc ##ru ) and the costs of treating [unused0] nosocomial pneumonia [unused1] ( np ) produced by methicillin - resistant staphylococcus aureus ( mrsa ) [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2634 5841 7965 8422 12 4387 3176 13 1930 1920 5982 1927 8247 1 22905 7868 1 12 5007 13 4152 2007 23025 17 4910 13452 8849 12 12836 13 3\n",
            "INFO:Blink:Label tokens : [CLS] infective pneumonia ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 20905 7868 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 72468 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] nosocomial pneumonia [unused1]\n",
            "INFO:Blink:Mention ids : 1 22905 7868 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 425 ##46 ##400 ##7\n",
            "INFO:Blink:Concept ID ori ids : 24592 7772 10893 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] assess healthcare resource utilization ( hc ##ru ) and the costs of treating [unused0] nosocomial pneumonia [unused1] ( np ) produced by methicillin - resistant staphylococcus aureus ( mrsa ) [SEP]\n",
            "INFO:Blink:Context ids : 2 2634 5841 7965 8422 12 4387 3176 13 1930 1920 5982 1927 8247 1 22905 7868 1 12 5007 13 4152 2007 23025 17 4910 13452 8849 12 12836 13 3\n",
            "INFO:Blink:Contextual tokens : [CLS] assess healthcare resource utilization ( hc ##ru ) and the costs of treating [unused0] nosocomial pneumonia [unused1] ( np ) produced by methicillin - resistant staphylococcus aureus ( mrsa ) [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2634 5841 7965 8422 12 4387 3176 13 1930 1920 5982 1927 8247 1 22905 7868 1 12 5007 13 4152 2007 23025 17 4910 13452 8849 12 12836 13 3\n",
            "INFO:Blink:Label tokens : [CLS] nosocomial infectious disease ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 22905 7465 2573 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 177009 is in-KB\n",
            "label_text_vecs: tensor([[   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        [   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        [   2, 5592, 4770,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0],\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0],\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0]])\n",
            "INFO:Blink:Getting top 50 predictions.\n",
            "INFO:Blink:Enriched from top 25 predictions.\n",
            "  0% 0/125 [00:00<?, ?it/s]\n",
            "  0% 0/237826 [00:00<?, ?it/s]\u001b[A\n",
            "  0% 515/237826 [00:00<02:28, 1599.16it/s]\u001b[A\n",
            "  1% 2455/237826 [00:00<00:59, 3928.42it/s]\u001b[A\n",
            "  1% 3559/237826 [00:00<01:01, 3809.34it/s]\u001b[A\n",
            "  2% 4636/237826 [00:01<01:01, 3773.39it/s]\u001b[A\n",
            "  3% 6200/237826 [00:01<00:55, 4137.13it/s]\u001b[A\n",
            "  3% 6610/237826 [00:01<01:10, 3275.53it/s]\u001b[A\n",
            "  3% 8003/237826 [00:02<01:02, 3698.49it/s]\u001b[A\n",
            "  4% 8368/237826 [00:02<01:21, 2806.01it/s]\u001b[A\n",
            "  4% 10256/237826 [00:02<00:48, 4690.63it/s]\u001b[A\n",
            "  5% 10932/237826 [00:02<01:00, 3745.32it/s]\u001b[A\n",
            "  5% 12466/237826 [00:03<00:58, 3843.26it/s]\u001b[A\n",
            "  6% 14035/237826 [00:03<00:59, 3745.99it/s]\u001b[A\n",
            "  7% 15946/237826 [00:04<00:41, 5332.67it/s]\u001b[A\n",
            "  7% 16744/237826 [00:04<00:52, 4241.56it/s]\u001b[A\n",
            "  7% 17448/237826 [00:04<01:07, 3257.77it/s]\u001b[A\n",
            "  8% 19034/237826 [00:05<01:06, 3284.28it/s]\u001b[A\n",
            "  9% 20804/237826 [00:05<00:46, 4661.74it/s]\u001b[A\n",
            "  9% 21563/237826 [00:05<00:58, 3706.66it/s]\u001b[A\n",
            " 10% 23093/237826 [00:06<00:58, 3679.91it/s]\u001b[A\n",
            " 10% 24824/237826 [00:06<00:57, 3699.20it/s]\u001b[A\n",
            " 11% 26610/237826 [00:06<00:42, 4972.94it/s]\u001b[A\n",
            " 12% 27365/237826 [00:07<00:55, 3774.85it/s]\u001b[A\n",
            " 12% 27946/237826 [00:07<01:05, 3190.76it/s]\u001b[A\n",
            " 12% 28406/237826 [00:08<01:38, 2120.60it/s]\u001b[A\n",
            " 12% 28750/237826 [00:08<01:50, 1897.32it/s]\u001b[A\n",
            " 12% 29026/237826 [00:08<02:07, 1635.57it/s]\u001b[A\n",
            " 12% 29434/237826 [00:08<02:15, 1543.50it/s]\u001b[A\n",
            " 13% 30597/237826 [00:09<01:36, 2155.87it/s]\u001b[A\n",
            " 13% 31608/237826 [00:09<01:25, 2405.95it/s]\u001b[A\n",
            " 13% 31872/237826 [00:09<01:44, 1966.47it/s]\u001b[A\n",
            " 14% 32137/237826 [00:10<02:05, 1643.02it/s]\u001b[A\n",
            " 14% 32310/237826 [00:10<02:29, 1379.12it/s]\u001b[A\n",
            " 14% 32450/237826 [00:10<03:13, 1063.86it/s]\u001b[A\n",
            " 14% 32561/237826 [00:11<03:51, 887.85it/s] \u001b[A\n",
            " 14% 32719/237826 [00:11<04:32, 753.08it/s]\u001b[A\n",
            " 14% 32974/237826 [00:11<04:27, 767.13it/s]\u001b[A\n",
            " 15% 34600/237826 [00:12<01:39, 2034.68it/s]\u001b[A\n",
            " 15% 35647/237826 [00:12<01:26, 2344.69it/s]\u001b[A\n",
            " 15% 35877/237826 [00:12<01:47, 1881.61it/s]\u001b[A\n",
            " 15% 36435/237826 [00:13<01:52, 1791.71it/s]\u001b[A\n",
            " 16% 37489/237826 [00:13<01:29, 2244.43it/s]\u001b[A\n",
            " 16% 38583/237826 [00:13<01:17, 2562.55it/s]\u001b[A\n",
            " 17% 39983/237826 [00:14<01:06, 2984.01it/s]\u001b[A\n",
            " 17% 40279/237826 [00:14<01:22, 2402.43it/s]\u001b[A\n",
            " 17% 40514/237826 [00:14<01:44, 1884.54it/s]\u001b[A\n",
            " 17% 40700/237826 [00:15<02:13, 1479.16it/s]\u001b[A\n",
            " 17% 40847/237826 [00:15<02:38, 1244.05it/s]\u001b[A\n",
            " 17% 40966/237826 [00:15<03:19, 985.40it/s] \u001b[A\n",
            " 18% 42084/237826 [00:16<01:56, 1683.30it/s]\u001b[A\n",
            " 18% 43127/237826 [00:16<01:31, 2119.93it/s]\u001b[A\n",
            " 18% 43842/237826 [00:16<01:32, 2099.10it/s]\u001b[A\n",
            " 19% 44048/237826 [00:17<01:53, 1704.36it/s]\u001b[A\n",
            " 19% 44464/237826 [00:17<02:12, 1461.68it/s]\u001b[A\n",
            " 19% 45716/237826 [00:17<01:42, 1865.72it/s]\u001b[A\n",
            " 20% 47713/237826 [00:18<01:13, 2580.42it/s]\u001b[A\n",
            " 20% 48265/237826 [00:18<01:24, 2241.03it/s]\u001b[A\n",
            " 21% 49568/237826 [00:19<01:16, 2467.45it/s]\u001b[A\n",
            " 21% 50978/237826 [00:19<00:54, 3458.63it/s]\u001b[A\n",
            " 22% 51489/237826 [00:19<01:09, 2699.64it/s]\u001b[A\n",
            " 22% 51884/237826 [00:20<01:29, 2079.71it/s]\u001b[A\n",
            " 23% 53598/237826 [00:20<01:12, 2555.82it/s]\u001b[A\n",
            " 23% 55572/237826 [00:21<00:55, 3268.76it/s]\u001b[A\n",
            " 24% 57314/237826 [00:21<00:40, 4506.71it/s]\u001b[A\n",
            " 24% 57998/237826 [00:21<00:51, 3489.18it/s]\u001b[A\n",
            " 25% 59642/237826 [00:22<00:46, 3859.79it/s]\u001b[A\n",
            " 26% 60727/237826 [00:22<00:48, 3685.38it/s]\u001b[A\n",
            " 26% 61801/237826 [00:22<00:50, 3519.28it/s]\u001b[A\n",
            " 26% 62883/237826 [00:23<00:51, 3420.95it/s]\u001b[A\n",
            " 27% 63268/237826 [00:23<01:02, 2786.54it/s]\u001b[A\n",
            " 27% 63905/237826 [00:23<01:12, 2415.43it/s]\u001b[A\n",
            " 27% 65055/237826 [00:24<01:03, 2702.28it/s]\u001b[A\n",
            " 28% 66058/237826 [00:24<01:02, 2728.92it/s]\u001b[A\n",
            " 28% 66343/237826 [00:24<01:17, 2208.91it/s]\u001b[A\n",
            " 28% 67130/237826 [00:25<01:16, 2240.76it/s]\u001b[A\n",
            " 29% 68418/237826 [00:25<01:06, 2565.34it/s]\u001b[A\n",
            " 29% 68686/237826 [00:25<01:20, 2090.42it/s]\u001b[A\n",
            " 29% 69646/237826 [00:26<01:15, 2234.14it/s]\u001b[A\n",
            " 29% 69866/237826 [00:26<01:31, 1825.82it/s]\u001b[A\n",
            " 30% 70185/237826 [00:26<01:51, 1497.54it/s]\u001b[A\n",
            " 30% 70975/237826 [00:27<01:39, 1677.94it/s]\u001b[A\n",
            " 31% 72554/237826 [00:27<01:08, 2396.40it/s]\u001b[A\n",
            " 31% 74107/237826 [00:28<00:56, 2874.17it/s]\u001b[A\n",
            " 32% 75163/237826 [00:28<00:55, 2934.83it/s]\u001b[A\n",
            " 32% 76203/237826 [00:28<00:54, 2948.75it/s]\u001b[A\n",
            " 32% 76496/237826 [00:29<01:06, 2426.01it/s]\u001b[A\n",
            " 33% 77767/237826 [00:29<01:00, 2656.01it/s]\u001b[A\n",
            " 33% 78791/237826 [00:29<00:58, 2717.92it/s]\u001b[A\n",
            " 34% 80069/237826 [00:30<00:55, 2856.32it/s]\u001b[A\n",
            " 34% 80873/237826 [00:30<01:01, 2538.90it/s]\u001b[A\n",
            " 34% 81922/237826 [00:31<00:58, 2664.77it/s]\u001b[A\n",
            " 35% 83162/237826 [00:31<00:56, 2753.11it/s]\u001b[A\n",
            " 36% 84559/237826 [00:31<00:53, 2839.33it/s]\u001b[A\n",
            " 36% 85473/237826 [00:32<00:56, 2685.42it/s]\u001b[A\n",
            " 36% 86581/237826 [00:32<00:43, 3467.02it/s]\u001b[A\n",
            " 37% 87085/237826 [00:32<00:56, 2668.52it/s]\u001b[A\n",
            " 37% 88565/237826 [00:33<00:54, 2756.11it/s]\u001b[A\n",
            " 38% 89908/237826 [00:33<00:38, 3800.85it/s]\u001b[A\n",
            " 38% 90563/237826 [00:33<00:49, 2985.36it/s]\u001b[A\n",
            " 39% 92451/237826 [00:34<00:31, 4620.77it/s]\u001b[A\n",
            " 39% 93246/237826 [00:34<00:40, 3541.81it/s]\u001b[A\n",
            " 39% 93859/237826 [00:34<00:53, 2675.28it/s]\u001b[A\n",
            " 40% 94763/237826 [00:35<00:43, 3322.71it/s]\u001b[A\n",
            " 40% 95346/237826 [00:35<00:54, 2615.89it/s]\u001b[A\n",
            " 40% 96159/237826 [00:35<00:43, 3262.49it/s]\u001b[A\n",
            " 41% 96719/237826 [00:36<01:16, 1837.13it/s]\u001b[A\n",
            " 41% 97251/237826 [00:36<01:23, 1677.77it/s]\u001b[A\n",
            " 41% 97580/237826 [00:37<01:37, 1444.40it/s]\u001b[A\n",
            " 41% 97836/237826 [00:37<01:49, 1273.67it/s]\u001b[A\n",
            " 41% 98123/237826 [00:37<02:08, 1089.05it/s]\u001b[A\n",
            " 41% 98286/237826 [00:38<02:25, 957.02it/s] \u001b[A\n",
            " 42% 99605/237826 [00:38<01:20, 1726.10it/s]\u001b[A\n",
            " 42% 100157/237826 [00:38<01:26, 1588.14it/s]\u001b[A\n",
            " 42% 100705/237826 [00:39<01:30, 1520.14it/s]\u001b[A\n",
            " 42% 100869/237826 [00:40<01:49, 1252.00it/s]\u001b[A\n",
            " 42% 100999/237826 [00:40<02:17, 995.21it/s] \u001b[A\n",
            " 43% 102867/237826 [00:40<01:02, 2152.31it/s]\u001b[A\n",
            " 44% 104174/237826 [00:40<00:54, 2455.89it/s]\u001b[A\n",
            " 44% 105765/237826 [00:41<00:45, 2930.25it/s]\u001b[A\n",
            " 45% 106896/237826 [00:41<00:45, 2855.78it/s]\u001b[A\n",
            " 45% 108161/237826 [00:42<00:44, 2903.64it/s]\u001b[A\n",
            " 46% 108702/237826 [00:42<00:51, 2493.31it/s]\u001b[A\n",
            " 46% 109998/237826 [00:42<00:48, 2658.80it/s]\u001b[A\n",
            " 47% 111116/237826 [00:43<00:47, 2667.39it/s]\u001b[A\n",
            " 47% 112743/237826 [00:43<00:41, 3020.92it/s]\u001b[A\n",
            " 48% 113283/237826 [00:44<00:49, 2524.48it/s]\u001b[A\n",
            " 48% 113533/237826 [00:44<00:59, 2093.44it/s]\u001b[A\n",
            " 48% 115172/237826 [00:44<00:45, 2696.80it/s]\u001b[A\n",
            " 49% 116075/237826 [00:45<00:44, 2709.74it/s]\u001b[A\n",
            " 49% 116618/237826 [00:45<00:53, 2245.58it/s]\u001b[A\n",
            " 50% 118092/237826 [00:46<00:47, 2529.49it/s]\u001b[A\n",
            " 50% 119582/237826 [00:46<00:45, 2592.81it/s]\u001b[A\n",
            " 51% 121539/237826 [00:47<00:28, 4051.47it/s]\u001b[A\n",
            " 51% 122305/237826 [00:47<00:35, 3214.96it/s]\u001b[A\n",
            " 52% 123717/237826 [00:47<00:37, 3037.83it/s]\u001b[A\n",
            " 53% 125244/237826 [00:48<00:27, 4161.08it/s]\u001b[A\n",
            " 53% 126038/237826 [00:48<00:35, 3178.58it/s]\u001b[A\n",
            " 53% 126643/237826 [00:48<00:32, 3427.03it/s]\u001b[A\n",
            " 53% 127227/237826 [00:48<00:43, 2535.09it/s]\u001b[A\n",
            " 54% 128748/237826 [00:49<00:28, 3861.69it/s]\u001b[A\n",
            " 54% 129458/237826 [00:49<00:37, 2901.76it/s]\u001b[A\n",
            " 55% 131202/237826 [00:50<00:23, 4554.83it/s]\u001b[A\n",
            " 56% 132099/237826 [00:50<00:30, 3474.51it/s]\u001b[A\n",
            " 56% 132787/237826 [00:50<00:49, 2142.81it/s]\u001b[A\n",
            " 56% 133295/237826 [00:51<00:56, 1859.65it/s]\u001b[A\n",
            " 57% 134843/237826 [00:51<00:43, 2345.78it/s]\u001b[A\n",
            " 57% 135395/237826 [00:52<00:49, 2060.18it/s]\u001b[A\n",
            " 57% 136746/237826 [00:52<00:42, 2355.84it/s]\u001b[A\n",
            " 58% 138100/237826 [00:52<00:38, 2583.59it/s]\u001b[A\n",
            " 58% 138407/237826 [00:53<00:49, 2015.67it/s]\u001b[A\n",
            " 58% 138643/237826 [00:53<00:59, 1671.78it/s]\u001b[A\n",
            " 58% 138946/237826 [00:54<01:11, 1377.60it/s]\u001b[A\n",
            " 58% 139097/237826 [00:54<01:29, 1102.78it/s]\u001b[A\n",
            " 59% 140411/237826 [00:55<00:57, 1697.99it/s]\u001b[A\n",
            " 59% 140583/237826 [00:55<01:12, 1347.00it/s]\u001b[A\n",
            " 59% 140991/237826 [00:55<01:17, 1241.55it/s]\u001b[A\n",
            " 60% 141570/237826 [00:56<01:13, 1308.75it/s]\u001b[A\n",
            " 60% 141701/237826 [00:56<01:34, 1015.73it/s]\u001b[A\n",
            " 60% 141803/237826 [00:56<01:58, 812.13it/s] \u001b[A\n",
            " 60% 142258/237826 [00:57<01:45, 909.43it/s]\u001b[A\n",
            " 60% 142348/237826 [00:58<02:11, 723.78it/s]\u001b[A\n",
            " 60% 142419/237826 [00:58<03:33, 446.81it/s]\u001b[A\n",
            " 60% 143155/237826 [00:58<01:59, 792.01it/s]\u001b[A\n",
            " 60% 143242/237826 [00:59<02:25, 650.20it/s]\u001b[A\n",
            " 60% 143732/237826 [00:59<02:00, 783.59it/s]\u001b[A\n",
            " 60% 143813/237826 [01:00<02:24, 650.25it/s]\u001b[A\n",
            " 60% 143877/237826 [01:00<03:25, 457.87it/s]\u001b[A\n",
            " 61% 144503/237826 [01:00<01:37, 961.49it/s]\u001b[A\n",
            " 61% 144725/237826 [01:01<01:58, 785.05it/s]\u001b[A\n",
            " 61% 146214/237826 [01:01<01:01, 1493.10it/s]\u001b[A\n",
            " 62% 147662/237826 [01:02<00:34, 2616.98it/s]\u001b[A\n",
            " 62% 148184/237826 [01:02<00:42, 2120.04it/s]\u001b[A\n",
            " 63% 148759/237826 [01:02<00:53, 1664.32it/s]\u001b[A\n",
            " 63% 150322/237826 [01:03<00:29, 2920.69it/s]\u001b[A\n",
            " 63% 151005/237826 [01:03<00:35, 2431.91it/s]\u001b[A\n",
            " 64% 151530/237826 [01:03<00:46, 1846.95it/s]\u001b[A\n",
            " 64% 153084/237826 [01:04<00:27, 3100.48it/s]\u001b[A\n",
            " 65% 153810/237826 [01:04<00:33, 2510.79it/s]\u001b[A\n",
            " 65% 154365/237826 [01:04<00:40, 2048.93it/s]\u001b[A\n",
            " 65% 154975/237826 [01:05<00:45, 1819.45it/s]\u001b[A\n",
            " 65% 155313/237826 [01:05<00:53, 1537.42it/s]\u001b[A\n",
            " 66% 156871/237826 [01:06<00:37, 2139.12it/s]\u001b[A\n",
            " 67% 158411/237826 [01:06<00:31, 2539.66it/s]\u001b[A\n",
            " 67% 160504/237826 [01:07<00:24, 3169.02it/s]\u001b[A\n",
            " 68% 162133/237826 [01:07<00:23, 3285.90it/s]\u001b[A\n",
            " 68% 162483/237826 [01:08<00:28, 2682.30it/s]\u001b[A\n",
            " 69% 163947/237826 [01:08<00:25, 2853.66it/s]\u001b[A\n",
            " 69% 165254/237826 [01:08<00:25, 2862.12it/s]\u001b[A\n",
            " 70% 167040/237826 [01:09<00:22, 3209.95it/s]\u001b[A\n",
            " 71% 168322/237826 [01:09<00:22, 3089.32it/s]\u001b[A\n",
            " 71% 169445/237826 [01:10<00:23, 2892.97it/s]\u001b[A\n",
            " 72% 170709/237826 [01:10<00:23, 2877.44it/s]\u001b[A\n",
            " 72% 170998/237826 [01:11<00:28, 2334.67it/s]\u001b[A\n",
            " 72% 171273/237826 [01:11<00:36, 1800.05it/s]\u001b[A\n",
            " 72% 171450/237826 [01:11<00:45, 1454.43it/s]\u001b[A\n",
            " 72% 171862/237826 [01:12<00:51, 1275.86it/s]\u001b[A\n",
            " 72% 171986/237826 [01:12<01:04, 1013.75it/s]\u001b[A\n",
            " 73% 172732/237826 [01:13<00:54, 1202.92it/s]\u001b[A\n",
            " 73% 172848/237826 [01:13<01:06, 973.41it/s] \u001b[A\n",
            " 73% 173304/237826 [01:14<01:06, 977.03it/s]\u001b[A\n",
            " 73% 173892/237826 [01:14<00:59, 1083.29it/s]\u001b[A\n",
            " 73% 174764/237826 [01:15<00:35, 1760.62it/s]\u001b[A\n",
            " 74% 175077/237826 [01:15<00:45, 1366.49it/s]\u001b[A\n",
            " 74% 176448/237826 [01:15<00:23, 2646.29it/s]\u001b[A\n",
            " 74% 177024/237826 [01:15<00:29, 2049.90it/s]\u001b[A\n",
            " 75% 177463/237826 [01:16<00:39, 1526.79it/s]\u001b[A\n",
            " 75% 178528/237826 [01:16<00:24, 2382.51it/s]\u001b[A\n",
            " 75% 179071/237826 [01:16<00:31, 1893.34it/s]\u001b[A\n",
            " 75% 179486/237826 [01:17<00:40, 1451.51it/s]\u001b[A\n",
            " 76% 180092/237826 [01:17<00:30, 1871.42it/s]\u001b[A\n",
            " 76% 180487/237826 [01:17<00:41, 1391.30it/s]\u001b[A\n",
            " 76% 181664/237826 [01:18<00:24, 2335.92it/s]\u001b[A\n",
            " 77% 182136/237826 [01:18<00:31, 1745.79it/s]\u001b[A\n",
            " 77% 182494/237826 [01:18<00:36, 1499.42it/s]\u001b[A\n",
            " 77% 183280/237826 [01:19<00:35, 1538.93it/s]\u001b[A\n",
            " 77% 183523/237826 [01:19<00:42, 1286.74it/s]\u001b[A\n",
            " 77% 184041/237826 [01:20<00:43, 1229.75it/s]\u001b[A\n",
            " 77% 184208/237826 [01:20<00:54, 989.38it/s] \u001b[A\n",
            " 78% 184550/237826 [01:21<00:56, 935.26it/s]\u001b[A\n",
            " 78% 185064/237826 [01:21<00:51, 1017.57it/s]\u001b[A\n",
            " 78% 185181/237826 [01:21<01:04, 810.16it/s] \u001b[A\n",
            " 78% 185572/237826 [01:22<01:04, 813.93it/s]\u001b[A\n",
            " 79% 187115/237826 [01:22<00:31, 1591.30it/s]\u001b[A\n",
            " 79% 188395/237826 [01:23<00:25, 1942.63it/s]\u001b[A\n",
            " 79% 188588/237826 [01:23<00:31, 1564.42it/s]\u001b[A\n",
            " 79% 188948/237826 [01:24<00:37, 1308.36it/s]\u001b[A\n",
            " 80% 189805/237826 [01:25<00:32, 1477.64it/s]\u001b[A\n",
            " 80% 189951/237826 [01:25<00:51, 931.84it/s] \u001b[A\n",
            " 80% 190425/237826 [01:25<00:49, 959.14it/s]\u001b[A\n",
            " 80% 190532/237826 [01:26<00:58, 805.57it/s]\u001b[A\n",
            " 80% 191324/237826 [01:26<00:43, 1065.34it/s]\u001b[A\n",
            " 80% 191433/237826 [01:27<00:53, 871.02it/s] \u001b[A\n",
            " 81% 191921/237826 [01:27<00:50, 917.71it/s]\u001b[A\n",
            " 81% 192011/237826 [01:28<01:02, 735.82it/s]\u001b[A\n",
            " 81% 192800/237826 [01:28<00:45, 987.34it/s]\u001b[A\n",
            " 81% 193421/237826 [01:28<00:31, 1424.83it/s]\u001b[A\n",
            " 81% 193723/237826 [01:29<00:39, 1120.26it/s]\u001b[A\n",
            " 82% 195822/237826 [01:29<00:13, 3050.56it/s]\u001b[A\n",
            " 83% 196586/237826 [01:29<00:17, 2403.00it/s]\u001b[A\n",
            " 83% 197166/237826 [01:30<00:14, 2753.43it/s]\u001b[A\n",
            " 83% 197744/237826 [01:30<00:19, 2069.58it/s]\u001b[A\n",
            " 84% 199216/237826 [01:30<00:11, 3427.21it/s]\u001b[A\n",
            " 84% 199952/237826 [01:31<00:14, 2533.02it/s]\u001b[A\n",
            " 84% 200510/237826 [01:31<00:19, 1879.46it/s]\u001b[A\n",
            " 85% 201345/237826 [01:31<00:14, 2476.71it/s]\u001b[A\n",
            " 85% 202219/237826 [01:31<00:11, 3194.79it/s]\u001b[A\n",
            " 85% 203129/237826 [01:31<00:08, 4000.12it/s]\u001b[A\n",
            " 86% 203829/237826 [01:32<00:12, 2686.73it/s]\u001b[A\n",
            " 87% 205851/237826 [01:32<00:06, 4844.79it/s]\u001b[A\n",
            " 87% 206796/237826 [01:32<00:09, 3397.83it/s]\u001b[A\n",
            " 87% 207998/237826 [01:33<00:09, 3072.62it/s]\u001b[A\n",
            " 88% 208590/237826 [01:33<00:11, 2445.57it/s]\u001b[A\n",
            " 88% 209898/237826 [01:34<00:10, 2549.51it/s]\u001b[A\n",
            " 89% 210497/237826 [01:34<00:12, 2175.06it/s]\u001b[A\n",
            " 89% 210813/237826 [01:35<00:14, 1826.48it/s]\u001b[A\n",
            " 89% 212133/237826 [01:35<00:12, 2138.24it/s]\u001b[A\n",
            " 90% 213700/237826 [01:36<00:09, 2466.50it/s]\u001b[A\n",
            " 90% 213973/237826 [01:36<00:12, 1909.85it/s]\u001b[A\n",
            " 90% 214182/237826 [01:37<00:15, 1545.35it/s]\u001b[A\n",
            " 90% 214640/237826 [01:37<00:17, 1362.07it/s]\u001b[A\n",
            " 90% 214783/237826 [01:37<00:21, 1090.29it/s]\u001b[A\n",
            " 91% 215532/237826 [01:38<00:17, 1247.93it/s]\u001b[A\n",
            " 91% 216855/237826 [01:38<00:12, 1718.08it/s]\u001b[A\n",
            " 91% 217476/237826 [01:39<00:12, 1613.62it/s]\u001b[A\n",
            " 92% 217636/237826 [01:39<00:15, 1307.69it/s]\u001b[A\n",
            " 92% 217761/237826 [01:40<00:20, 960.42it/s] \u001b[A\n",
            " 92% 218239/237826 [01:40<00:19, 979.94it/s]\u001b[A\n",
            " 92% 218337/237826 [01:41<00:25, 757.13it/s]\u001b[A\n",
            " 92% 218544/237826 [01:41<00:29, 661.75it/s]\u001b[A\n",
            " 92% 219165/237826 [01:42<00:21, 866.12it/s]\u001b[A\n",
            " 93% 220334/237826 [01:42<00:13, 1327.11it/s]\u001b[A\n",
            " 93% 220463/237826 [01:42<00:16, 1021.41it/s]\u001b[A\n",
            " 93% 221692/237826 [01:43<00:08, 1931.24it/s]\u001b[A\n",
            " 93% 222066/237826 [01:43<00:10, 1486.98it/s]\u001b[A\n",
            " 94% 223087/237826 [01:44<00:06, 2316.90it/s]\u001b[A\n",
            " 94% 223590/237826 [01:44<00:08, 1725.25it/s]\u001b[A\n",
            " 94% 224445/237826 [01:44<00:05, 2410.54it/s]\u001b[A\n",
            " 95% 224967/237826 [01:44<00:06, 1851.32it/s]\u001b[A\n",
            " 95% 226111/237826 [01:44<00:04, 2787.13it/s]\u001b[A\n",
            " 95% 226651/237826 [01:45<00:05, 1935.28it/s]\u001b[A\n",
            " 96% 227802/237826 [01:46<00:03, 2901.95it/s]\u001b[A\n",
            " 96% 228415/237826 [01:46<00:04, 2141.39it/s]\u001b[A\n",
            " 96% 228879/237826 [01:46<00:05, 1580.84it/s]\u001b[A\n",
            " 96% 229463/237826 [01:47<00:04, 1956.35it/s]\u001b[A\n",
            " 97% 229875/237826 [01:47<00:04, 1627.82it/s]\u001b[A\n",
            " 97% 231220/237826 [01:47<00:03, 2063.70it/s]\u001b[A\n",
            " 97% 231784/237826 [01:48<00:03, 1773.98it/s]\u001b[A\n",
            " 98% 233165/237826 [01:48<00:02, 2057.08it/s]\u001b[A\n",
            " 98% 234004/237826 [01:49<00:01, 1934.12it/s]\u001b[A\n",
            " 99% 235122/237826 [01:49<00:01, 2040.35it/s]\u001b[A\n",
            " 99% 235350/237826 [01:50<00:01, 1639.00it/s]\u001b[A\n",
            " 99% 235702/237826 [01:50<00:01, 1346.85it/s]\u001b[A\n",
            " 99% 235847/237826 [01:51<00:01, 1087.24it/s]\u001b[A\n",
            " 99% 236525/237826 [01:51<00:01, 1172.02it/s]\u001b[A\n",
            "100% 237826/237826 [01:52<00:00, 2122.26it/s]\n",
            "list_inds_enriched_w_scores, sorted: [(34585, 206.55821), (34595, 204.5958), (34582, 204.32503), (34586, 203.35674), (34563, 202.08955), (106750, 201.56215), (41371, 201.55183), (34597, 201.22499), (34564, 201.08159), (222398, 200.5041), (36369, 200.40881), (34593, 200.30167), (210815, 200.19403), (106749, 200.1312), (41576, 200.07939), (34576, 200.0676), (34588, 199.97131), (195830, 199.89282), (34596, 199.71039), (197442, 199.34384), (208138, 199.30531), (46481, 199.22299), (210816, 199.11911), (46477, 199.0426), (106748, 199.02963), (2554, 199.01295), (144854, 198.97337), (34603, 198.9085), (34560, 198.67928), (41358, 198.61032), (178367, 198.22948), (236791, 198.22948), (34558, 198.1414), (48146, 198.10033), (34587, 198.0161), (41452, 198.01201), (172435, 197.9025), (34594, 197.73358), (158905, 197.71063), (34574, 197.6782), (34589, 197.592), (215864, 197.54651), (197443, 197.37453), (208139, 193.57481), (32896, 189.7868), (33212, 187.57916), (36370, 186.82387), (34599, 185.96443), (41321, 183.36395), (248, 150.13351)]\n",
            "list_inds_enriched_w_scores, sorted: [(34585, 206.55821), (34595, 204.5958), (34582, 204.32503), (34586, 203.35674), (34563, 202.08955), (106750, 201.56215), (41371, 201.55183), (34597, 201.22499), (34564, 201.08159), (222398, 200.5041), (36369, 200.40881), (34593, 200.30167), (210815, 200.19403), (106749, 200.1312), (41576, 200.07939), (34576, 200.0676), (34588, 199.97131), (195830, 199.89282), (34596, 199.71039), (197442, 199.34384), (208138, 199.30531), (46481, 199.22299), (210816, 199.11911), (46477, 199.0426), (106748, 199.02963), (2554, 199.01295), (144854, 198.97337), (34603, 198.9085), (34560, 198.67928), (41358, 198.61032), (178367, 198.22948), (236791, 198.22948), (34558, 198.1414), (48146, 198.10033), (34587, 198.0161), (41452, 198.01201), (172435, 197.9025), (34594, 197.73358), (158905, 197.71063), (34574, 197.6782), (34589, 197.592), (215864, 197.54651), (197443, 197.37453), (208139, 193.57481), (32896, 189.7868), (33212, 187.57916), (36370, 186.82387), (34599, 185.96443), (41321, 183.36395), (248, 150.13351)]\n",
            "list_inds_enriched_w_scores, sorted: [(34585, 206.55821), (34595, 204.5958), (34582, 204.32503), (34586, 203.35674), (34563, 202.08955), (106750, 201.56215), (41371, 201.55183), (34597, 201.22499), (34564, 201.08159), (222398, 200.5041), (36369, 200.40881), (34593, 200.30167), (210815, 200.19403), (106749, 200.1312), (41576, 200.07939), (34576, 200.0676), (34588, 199.97131), (195830, 199.89282), (34596, 199.71039), (197442, 199.34384), (208138, 199.30531), (46481, 199.22299), (210816, 199.11911), (46477, 199.0426), (106748, 199.02963), (2554, 199.01295), (144854, 198.97337), (34603, 198.9085), (34560, 198.67928), (41358, 198.61032), (178367, 198.22948), (236791, 198.22948), (34558, 198.1414), (48146, 198.10033), (34587, 198.0161), (41452, 198.01201), (172435, 197.9025), (34594, 197.73358), (158905, 197.71063), (34574, 197.6782), (34589, 197.592), (215864, 197.54651), (197443, 197.37453), (208139, 193.57481), (32896, 189.7868), (33212, 187.57916), (36370, 186.82387), (34599, 185.96443), (41321, 183.36395), (248, 150.13351)]\n",
            "list_inds_enriched_w_scores, sorted: [(34585, 206.55821), (34595, 204.5958), (34582, 204.32503), (34586, 203.35674), (34563, 202.08955), (106750, 201.56215), (41371, 201.55183), (34597, 201.22499), (34564, 201.08159), (222398, 200.5041), (36369, 200.40881), (34593, 200.30167), (210815, 200.19403), (106749, 200.1312), (41576, 200.07939), (34576, 200.0676), (34588, 199.97131), (195830, 199.89282), (34596, 199.71039), (197442, 199.34384), (208138, 199.30531), (46481, 199.22299), (210816, 199.11911), (46477, 199.0426), (106748, 199.02963), (2554, 199.01295), (144854, 198.97337), (34603, 198.9085), (34560, 198.67928), (41358, 198.61032), (178367, 198.22948), (236791, 198.22948), (34558, 198.1414), (48146, 198.10033), (34587, 198.0161), (41452, 198.01201), (172435, 197.9025), (34594, 197.73358), (158905, 197.71063), (34574, 197.6782), (34589, 197.592), (215864, 197.54651), (197443, 197.37453), (208139, 193.57481), (32896, 189.7868), (33212, 187.57916), (36370, 186.82387), (34599, 185.96443), (41321, 183.36395), (248, 150.13351)]\n",
            "list_inds_enriched_w_scores, sorted: [(34585, 206.55821), (34595, 204.5958), (34582, 204.32503), (34586, 203.35674), (34563, 202.08955), (106750, 201.56215), (41371, 201.55183), (34597, 201.22499), (34564, 201.08159), (222398, 200.5041), (36369, 200.40881), (34593, 200.30167), (210815, 200.19403), (106749, 200.1312), (41576, 200.07939), (34576, 200.0676), (34588, 199.97131), (195830, 199.89282), (34596, 199.71039), (197442, 199.34384), (208138, 199.30531), (46481, 199.22299), (210816, 199.11911), (46477, 199.0426), (106748, 199.02963), (2554, 199.01295), (144854, 198.97337), (34603, 198.9085), (34560, 198.67928), (41358, 198.61032), (178367, 198.22948), (236791, 198.22948), (34558, 198.1414), (48146, 198.10033), (34587, 198.0161), (41452, 198.01201), (172435, 197.9025), (34594, 197.73358), (158905, 197.71063), (34574, 197.6782), (34589, 197.592), (215864, 197.54651), (197443, 197.37453), (208139, 193.57481), (32896, 189.7868), (33212, 187.57916), (36370, 186.82387), (34599, 185.96443), (41321, 183.36395), (248, 150.13351)]\n",
            "list_inds_enriched_w_scores, sorted: [(34585, 206.55821), (34595, 204.5958), (34582, 204.32503), (34586, 203.35674), (34563, 202.08955), (106750, 201.56215), (41371, 201.55183), (34597, 201.22499), (34564, 201.08159), (222398, 200.5041), (36369, 200.40881), (34593, 200.30167), (210815, 200.19403), (106749, 200.1312), (41576, 200.07939), (34576, 200.0676), (34588, 199.97131), (195830, 199.89282), (34596, 199.71039), (197442, 199.34384), (208138, 199.30531), (46481, 199.22299), (210816, 199.11911), (46477, 199.0426), (106748, 199.02963), (2554, 199.01295), (144854, 198.97337), (34603, 198.9085), (34560, 198.67928), (41358, 198.61032), (178367, 198.22948), (236791, 198.22948), (34558, 198.1414), (48146, 198.10033), (34587, 198.0161), (41452, 198.01201), (172435, 197.9025), (34594, 197.73358), (158905, 197.71063), (34574, 197.6782), (34589, 197.592), (215864, 197.54651), (197443, 197.37453), (208139, 193.57481), (32896, 189.7868), (33212, 187.57916), (36370, 186.82387), (34599, 185.96443), (41321, 183.36395), (248, 150.13351)]\n",
            "list_inds_enriched_w_scores, sorted: [(181446, 210.06749), (169144, 206.71397), (67113, 206.52263), (175044, 205.77252), (67112, 203.29025), (54916, 203.02103), (100072, 201.64066), (54917, 201.38966), (67115, 200.79088), (67110, 200.67099), (67114, 200.2651), (54914, 200.16052), (108968, 200.07413), (169610, 200.07202), (175043, 199.54669), (169609, 199.04504), (100074, 197.91545), (175465, 197.59929), (54926, 197.27618), (175042, 196.77779), (54915, 196.4357), (67116, 196.36862), (72476, 196.07976), (175154, 196.03932), (159758, 195.51247), (196280, 195.33693), (54918, 195.05954), (169611, 194.64091), (193787, 193.55217), (108963, 193.18365), (54927, 192.60056), (169613, 189.79619), (44872, 189.1505), (63589, 186.32117), (44868, 185.70575), (72468, 185.52444), (193817, 184.36124), (103470, 182.21346), (34742, 179.80792), (14300, 178.75891), (10130, 178.24747), (36773, 174.08153), (10158, 172.43608), (3406, 168.69487), (111726, 167.67969), (103171, 165.75449), (35846, 161.97849), (141681, 161.78764), (44871, 157.86435), (847, 155.95114), (28951, 153.8268), (105225, 136.98752), (68012, 133.56734)]\n",
            "list_inds_enriched_w_scores, sorted: [(181446, 210.06749), (169144, 206.71397), (67113, 206.52263), (175044, 205.77252), (67112, 203.29025), (54916, 203.02103), (100072, 201.64066), (54917, 201.38966), (67115, 200.79088), (67110, 200.67099), (67114, 200.2651), (54914, 200.16052), (108968, 200.07413), (169610, 200.07202), (175043, 199.54669), (169609, 199.04504), (100074, 197.91545), (175465, 197.59929), (54926, 197.27618), (175042, 196.77779), (54915, 196.4357), (67116, 196.36862), (72476, 196.07976), (175154, 196.03932), (159758, 195.51247), (196280, 195.33693), (54918, 195.05954), (169611, 194.64091), (193787, 193.55217), (108963, 193.18365), (54927, 192.60056), (169613, 189.79619), (44872, 189.1505), (63589, 186.32117), (44868, 185.70575), (72468, 185.52444), (193817, 184.36124), (103470, 182.21346), (34742, 179.80792), (14300, 178.75891), (10130, 178.24747), (36773, 174.08153), (10158, 172.43608), (3406, 168.69487), (111726, 167.67969), (103171, 165.75449), (35846, 161.97849), (141681, 161.78764), (44871, 157.86435), (847, 155.95114), (28951, 153.8268), (105225, 136.98752), (68012, 133.56734)]\n",
            "  1% 1/125 [01:54<3:55:39, 114.03s/it]list_inds_enriched_w_scores, sorted: [(169598, 226.79568), (40978, 223.99124), (169611, 223.2891), (144023, 223.17154), (169607, 222.92969), (169609, 222.67325), (5361, 222.64871), (237749, 222.64398), (5364, 222.64398), (144006, 222.33029), (169589, 222.1536), (169608, 221.99548), (1066, 221.10364), (169610, 221.0007), (237748, 220.3385), (40977, 220.18596), (169601, 219.33983), (1059, 218.91934), (193815, 218.66989), (144014, 218.54471), (159757, 218.50357), (213329, 218.38428), (192658, 218.2844), (193790, 217.84973), (72528, 217.63551), (72440, 217.5376), (144025, 217.46744), (108976, 214.66808), (193820, 214.00919), (44872, 213.3101), (193845, 213.12796), (72468, 211.8758), (14300, 210.93027), (34742, 204.6875), (44574, 202.42177), (2786, 201.05959), (847, 199.47614), (45159, 197.43149), (63589, 186.79779), (169613, 186.08115), (177009, 182.47414), (36773, 159.99533), (10158, 155.27101), (3406, 151.67703), (105225, 133.6164), (2856, 129.2853), (30077, 122.62872), (248, 114.76696), (29066, 112.57063), (236179, 102.35861)]\n",
            "list_inds_enriched_w_scores, sorted: [(169598, 226.79568), (40978, 223.99124), (169611, 223.2891), (144023, 223.17154), (169607, 222.92969), (169609, 222.67325), (5361, 222.64871), (237749, 222.64398), (5364, 222.64398), (144006, 222.33029), (169589, 222.1536), (169608, 221.99548), (1066, 221.10364), (169610, 221.0007), (237748, 220.3385), (40977, 220.18596), (169601, 219.33983), (1059, 218.91934), (193815, 218.66989), (144014, 218.54471), (159757, 218.50357), (213329, 218.38428), (192658, 218.2844), (193790, 217.84973), (72528, 217.63551), (72440, 217.5376), (144025, 217.46744), (108976, 214.66808), (193820, 214.00919), (44872, 213.3101), (193845, 213.12796), (72468, 211.8758), (14300, 210.93027), (34742, 204.6875), (44574, 202.42177), (2786, 201.05959), (847, 199.47614), (45159, 197.43149), (63589, 186.79779), (169613, 186.08115), (177009, 182.47414), (36773, 159.99533), (10158, 155.27101), (3406, 151.67703), (105225, 133.6164), (2856, 129.2853), (30077, 122.62872), (248, 114.76696), (29066, 112.57063), (236179, 102.35861)]\n",
            "100% 125/125 [02:26<00:00,  1.17s/it]\n",
            "nn_data['labels']: tensor([35, 31, 40, 36, 44, 34,  8, 12, 39, 25, 10, 29, 33,  7,  8, 13, 10, 12,\n",
            "        26,  9, 26,  0,  7, 19,  3, 32, 23, 11,  2, 37, 18, 12, 44, 33, 13, 22,\n",
            "        40, 20, 11, 45, 15, 10, 26,  9,  2, 43, 47,  2, 35, 46, 33, 32, 29, 28,\n",
            "        39, 35, 15, 41, 34, 25,  1, 29, 24, 28, 13, 14, 11, 31, 16, 22, 35, 35,\n",
            "        35, 35, 35, 35, 28, 16, 16, 11, 11])\n",
            "INFO:Blink:num of nn_data: 81\n",
            "INFO:Blink:biencoder recall@k: 0.08 (81/1000)\n",
            "data saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top50_candidates/valid.t7\n",
            "Namespace(silent=False, debug=False, debug_max_lines=200, data_parallel=True, no_cuda=False, top_k=50, seed=52313, zeshel=False, max_seq_length=160, max_context_length=32, max_cand_length=128, path_to_model=None, bert_model='microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext', pull_from_layer=-1, lowercase=True, context_key='context', use_context=False, out_dim=1, add_linear=True, data_path='models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top50_candidates', output_path='models/crossencoder/mm+Disease-2017AA/original-NIL-top50-pubmedbert', use_triplet_loss_bi_enc=False, use_miner_bi_enc=False, use_ori_classification=True, use_NIL_classification=False, lambda_NIL=0.25, use_score_features=False, use_score_pooling=False, use_men_only_score_ft=False, use_extra_features=False, use_BM25=False, aggregating_factor=20, evaluate=False, output_eval_file=None, limit_by_max_lines=False, max_number_of_train_size=300000, train_batch_size=1, max_grad_norm=1.0, learning_rate=3e-05, num_train_epochs=4, limit_by_train_steps=False, max_num_train_steps=20000, print_interval=100, eval_interval=2000, save_interval=1, save_model_epoch_parts=True, warmup_proportion=0.1, gradient_accumulation_steps=1, type_optimization='all_encoder_layers', shuffle=False, optimize_NIL=False, use_preprocessed_data=False, use_full_training_data=False, fix_seeds=True, eval_batch_size=1, mode='valid', save_topk_result=False, encode_batch_size=8, use_synonyms=False, use_NIL_tag=False, use_NIL_desc=False, use_NIL_desc_tag=False, cand_pool_path=None, cand_encode_path=None, use_NIL_classification_infer=False, add_NIL_to_bi_enc_pred=False, NIL_ent_ind=64076)\n",
            "08/07/2024 14:40:36 - INFO - Blink -   use_ori_classification\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "tokenizer_config.json: 100% 28.0/28.0 [00:00<00:00, 193kB/s]\n",
            "config.json: 100% 385/385 [00:00<00:00, 3.28MB/s]\n",
            "vocab.txt: 100% 226k/226k [00:00<00:00, 1.74MB/s]\n",
            "pytorch_model.bin: 100% 440M/440M [00:02<00:00, 183MB/s]\n",
            "Some weights of the model checkpoint at microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n",
            "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "max_seq_length: 160\n",
            "context_input torch.Size([340, 50, 159])\n",
            "label_input: torch.Size([340]) tensor([ 0,  2,  1,  7,  5,  1,  6,  8,  3,  0,  4,  7,  6, 34, 35, 39, 17,  9,\n",
            "         8,  7, 21, 37, 40, 27, 41, 42, 28,  0, 11, 18, 32, 25, 12, 44, 47, 31,\n",
            "        46, 15, 29, 38,  3, 45,  5, 36, 48, 49, 10,  1,  2,  0,  3, 36, 24,  5,\n",
            "        10,  2, 18, 22, 19, 12, 25, 41,  8, 37, 39,  9,  7, 44, 13, 30, 35,  6,\n",
            "        23, 27, 21, 11, 17, 14, 16, 15, 26, 29,  1, 40, 28,  4,  3,  0,  2,  1,\n",
            "         5,  1,  2,  0,  5, 33, 49, 47, 20, 26,  2,  1,  2,  0,  5,  0,  2,  1,\n",
            "         5,  0,  3,  1,  7,  0,  2,  1,  5,  0,  3,  1,  7,  0,  2,  1,  7, 20,\n",
            "        20, 36,  6,  7, 17, 10, 27, 13, 45, 26, 21,  3, 18, 23, 39, 29, 28, 12,\n",
            "        24,  8,  5, 14,  5, 23, 10, 16, 26, 19, 43, 28, 13,  8, 24, 39, 29, 27,\n",
            "        17, 12, 25, 22,  0, 20,  3,  2,  4, 20, 16, 41, 25, 45, 37, 33, 17,  6,\n",
            "        31, 24, 15, 27,  8, 11,  0,  9, 10, 35, 28, 23, 46, 17, 19, 24,  7, 33,\n",
            "         6, 43, 26, 13, 27, 38, 40, 35, 30, 34, 37, 10,  5, 45, 22, 32,  4, 47,\n",
            "         0, 18, 48, 31, 41, 16, 25,  3,  7, 30, 40, 14,  6, 42, 27, 25, 33,  5,\n",
            "         9, 38, 45, 35, 29, 37, 12, 39, 38, 37, 32, 22, 41, 18, 46, 48,  3, 27,\n",
            "        20, 34, 17, 10, 36, 40, 25, 35, 11, 22, 31, 33, 16, 21, 38, 32,  2, 13,\n",
            "        37, 30,  5,  6, 47,  0, 26,  4, 45,  9, 49,  8, 24, 43, 17,  3,  2,  3,\n",
            "         0,  4,  1,  6,  1,  3,  0, 17,  4, 23,  9, 12, 36, 29, 16,  3,  6, 38,\n",
            "        14, 24,  1, 41,  5,  0, 45, 31, 42, 10, 19, 35, 15, 18,  7,  8,  2, 11,\n",
            "        36, 26, 29, 47,  5, 30, 13,  3, 26, 23, 43,  2,  9, 27, 32, 48])\n",
            "doc_ind_aggregated_strs:\n",
            " 0|1|2|3\n",
            "4|5|6|7\n",
            "8|9|10|11\n",
            "12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46\n",
            "47|48|49|50\n",
            "51|52|53|54|55|56|57|58|59|60|61|62|63|64|65|66|67|68|69|70|71|72|73|74|75|76|77|78|79|80|81|82|83|84|85|86\n",
            "87|88|89|90\n",
            "91|92|93|94\n",
            "95|96|97|98|99|100\n",
            "101|102|103|104\n",
            "105|106|107|108\n",
            "109|110|111|112\n",
            "113|114|115|116\n",
            "117|118|119|120\n",
            "121|122|123|124\n",
            "125\n",
            "126\n",
            "127\n",
            "128|129|130|131|132|133|134|135|136|137|138|139|140|141|142|143|144|145|146|147\n",
            "148|149|150|151|152|153|154|155|156|157|158|159|160|161|162|163|164|165|166|167\n",
            "168\n",
            "169|170|171|172|173|174|175|176|177|178|179|180|181|182|183|184|185|186|187|188\n",
            "189\n",
            "190|191|192|193|194|195|196|197|198|199|200|201|202|203|204|205|206|207|208|209|210|211|212|213|214|215|216|217|218|219|220|221|222|223\n",
            "224|225|226|227|228|229|230|231|232|233|234|235|236|237|238|239|240|241\n",
            "242\n",
            "243\n",
            "244\n",
            "245\n",
            "246|247|248|249|250|251|252|253|254|255|256|257|258|259|260|261|262|263|264|265|266|267|268|269|270|271|272|273|274|275|276|277|278|279|280|281|282|283\n",
            "284\n",
            "285|286\n",
            "287|288\n",
            "289|290\n",
            "291|292\n",
            "293|294\n",
            "295|296|297|298|299|300|301|302|303|304|305|306|307|308|309|310|311|312|313|314|315|316|317|318|319|320|321|322|323\n",
            "324|325|326\n",
            "327|328|329|330|331|332|333|334|335|336|337|338|339\n",
            "context_input: 340\n",
            "dict_context_to_label_inputs: 39\n",
            "label_single-label[0]: [0, 2, 1, 7]\n",
            "label_multi-label[0]: tensor([1., 1., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n",
            "label_multi_hot_is_complete: tensor([True, True, True, True, True, True, True, True, True, True, True, True,\n",
            "        True, True, True, True, True, True, True, True, True, True, True, True,\n",
            "        True, True, True, True, True, True, True, True, True, True, True, True,\n",
            "        True, True, True])\n",
            "context_input[0]: tensor([[    2, 22936, 15580,  ...,     0,     0,     0],\n",
            "        [    2, 22936, 15580,  ...,     0,     0,     0],\n",
            "        [    2, 22936, 15580,  ...,     0,     0,     0],\n",
            "        ...,\n",
            "        [    2, 22936, 15580,  ...,     0,     0,     0],\n",
            "        [    2, 22936, 15580,  ...,     0,     0,     0],\n",
            "        [    2, 22936, 15580,  ...,     0,     0,     0]])\n",
            "label_input[0]: tensor([1., 1., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n",
            "label_is_NIL_input[0]: tensor(False)\n",
            "max_seq_length: 160\n",
            "doc_ind_aggregated_strs:\n",
            " 0\n",
            "1|2\n",
            "3|4\n",
            "5|6|7|8|9|10|11\n",
            "12\n",
            "13|14|15\n",
            "16|17|18|19\n",
            "20|21|22|23|24|25\n",
            "26|27|28\n",
            "29|30|31|32|33|34|35|36|37|38|39|40\n",
            "41\n",
            "42\n",
            "43\n",
            "44\n",
            "45\n",
            "46\n",
            "47|48|49\n",
            "50|51|52|53|54\n",
            "55|56|57|58|59|60|61|62|63|64|65|66|67|68\n",
            "69\n",
            "70\n",
            "71\n",
            "72\n",
            "73\n",
            "74\n",
            "75\n",
            "76\n",
            "77\n",
            "78\n",
            "79\n",
            "80\n",
            "context_input: 81\n",
            "dict_context_to_label_inputs: 31\n",
            "label_single-label[0]: [35]\n",
            "label_multi-label[0]: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n",
            "label_multi_hot_is_complete: tensor([True, True, True, True, True, True, True, True, True, True, True, True,\n",
            "        True, True, True, True, True, True, True, True, True, True, True, True,\n",
            "        True, True, True, True, True, True, True])\n",
            "Evaluation:   0% 0/31 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
            "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
            "Evaluation: 100% 31/31 [00:13<00:00,  2.24it/s]\n",
            "num_labels: 50\n",
            "num in insertion_all: 0 num_ave: 0.0\n",
            "num in insertion_any: 0 num_ave: 0.0\n",
            "\n",
            "[MACRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.3902\n",
            "[MICRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.4171\n",
            "rec_at_1: 0.0373\n",
            "prec_at_1: 0.0968\n",
            "rec_at_3: 0.0422\n",
            "prec_at_3: 0.0538\n",
            "rec_at_5: 0.1413\n",
            "prec_at_5: 0.0581\n",
            "rec_at_10: 0.2204\n",
            "prec_at_10: 0.0548\n",
            "rec_at_50: 1.0000\n",
            "prec_at_50: 0.0523\n",
            "ins_at_1_all: 0.0323\n",
            "ins_at_1_any: 0.0968\n",
            "ins_at_3_all: 0.0323\n",
            "ins_at_3_any: 0.0968\n",
            "ins_at_5_all: 0.1290\n",
            "ins_at_5_any: 0.1935\n",
            "ins_at_10_all: 0.1613\n",
            "ins_at_10_any: 0.3548\n",
            "ins_at_50_all: 1.0000\n",
            "ins_at_50_any: 1.0000\n",
            "ins_all: 0.0000\n",
            "ins_any: 0.0000\n",
            "\n",
            "08/07/2024 14:41:03 - INFO - Blink -   Starting training\n",
            "08/07/2024 14:41:03 - INFO - Blink -   device: cuda n_gpu: 1, distributed training: False\n",
            "The following parameters will be optimized WITH decay:\n",
            "module.encoder.bert_model.encoder.layer.0.attention.self.query.weight , module.encoder.bert_model.encoder.layer.0.attention.self.key.weight , module.encoder.bert_model.encoder.layer.0.attention.self.value.weight , module.encoder.bert_model.encoder.layer.0.attention.output.dense.weight , module.encoder.bert_model.encoder.layer.0.attention.output.LayerNorm.weight , ...and 92 more\n",
            "The following parameters will be optimized WITHOUT decay:\n",
            "module.encoder.bert_model.encoder.layer.0.attention.self.query.bias , module.encoder.bert_model.encoder.layer.0.attention.self.key.bias , module.encoder.bert_model.encoder.layer.0.attention.self.value.bias , module.encoder.bert_model.encoder.layer.0.attention.output.dense.bias , module.encoder.bert_model.encoder.layer.0.attention.output.LayerNorm.bias , ...and 92 more\n",
            "08/07/2024 14:41:03 - INFO - Blink -    Num optimization steps = 156\n",
            "08/07/2024 14:41:03 - INFO - Blink -    Num warmup steps = 15\n",
            "Epoch:   0% 0/4 [00:00<?, ?it/s]\n",
            "Batch:   0% 0/39 [00:00<?, ?it/s]\u001b[A/usr/local/lib/python3.10/dist-packages/pytorch_transformers/optimization.py:166: UserWarning: This overload of add_ is deprecated:\n",
            "\tadd_(Number alpha, Tensor other)\n",
            "Consider using one of the following signatures instead:\n",
            "\tadd_(Tensor other, *, Number alpha) (Triggered internally at  ../torch/csrc/utils/python_arg_parser.cpp:1055.)\n",
            "  exp_avg.mul_(beta1).add_(1.0 - beta1, grad)\n",
            "\n",
            "Batch:   3% 1/39 [00:01<00:47,  1.25s/it]\u001b[A\n",
            "Batch:   5% 2/39 [00:02<00:45,  1.24s/it]\u001b[A\n",
            "Batch:   8% 3/39 [00:03<00:44,  1.24s/it]\u001b[A\n",
            "Batch:  10% 4/39 [00:04<00:43,  1.25s/it]\u001b[A\n",
            "Batch:  13% 5/39 [00:06<00:42,  1.25s/it]\u001b[A\n",
            "Batch:  15% 6/39 [00:07<00:41,  1.25s/it]\u001b[A\n",
            "Batch:  18% 7/39 [00:08<00:40,  1.26s/it]\u001b[A\n",
            "Batch:  21% 8/39 [00:10<00:39,  1.26s/it]\u001b[A\n",
            "Batch:  23% 9/39 [00:11<00:37,  1.26s/it]\u001b[A\n",
            "Batch:  26% 10/39 [00:12<00:36,  1.26s/it]\u001b[A\n",
            "Batch:  28% 11/39 [00:13<00:35,  1.26s/it]\u001b[A\n",
            "Batch:  31% 12/39 [00:15<00:34,  1.26s/it]\u001b[A\n",
            "Batch:  33% 13/39 [00:16<00:32,  1.27s/it]\u001b[A\n",
            "Batch:  36% 14/39 [00:17<00:31,  1.27s/it]\u001b[A\n",
            "Batch:  38% 15/39 [00:18<00:30,  1.27s/it]\u001b[A\n",
            "Batch:  41% 16/39 [00:20<00:29,  1.27s/it]\u001b[A\n",
            "Batch:  44% 17/39 [00:21<00:27,  1.27s/it]\u001b[A\n",
            "Batch:  46% 18/39 [00:22<00:26,  1.27s/it]\u001b[A\n",
            "Batch:  49% 19/39 [00:23<00:25,  1.27s/it]\u001b[A\n",
            "Batch:  51% 20/39 [00:25<00:24,  1.27s/it]\u001b[A\n",
            "Batch:  54% 21/39 [00:26<00:22,  1.27s/it]\u001b[A\n",
            "Batch:  56% 22/39 [00:27<00:21,  1.27s/it]\u001b[A\n",
            "Batch:  59% 23/39 [00:29<00:20,  1.27s/it]\u001b[A\n",
            "Batch:  62% 24/39 [00:30<00:19,  1.27s/it]\u001b[A\n",
            "Batch:  64% 25/39 [00:31<00:17,  1.27s/it]\u001b[A\n",
            "Batch:  67% 26/39 [00:32<00:16,  1.28s/it]\u001b[A\n",
            "Batch:  69% 27/39 [00:34<00:15,  1.27s/it]\u001b[A\n",
            "Batch:  72% 28/39 [00:35<00:14,  1.28s/it]\u001b[A\n",
            "Batch:  74% 29/39 [00:36<00:12,  1.28s/it]\u001b[A\n",
            "Batch:  77% 30/39 [00:38<00:11,  1.28s/it]\u001b[A\n",
            "Batch:  79% 31/39 [00:39<00:10,  1.28s/it]\u001b[A\n",
            "Batch:  82% 32/39 [00:40<00:08,  1.28s/it]\u001b[A\n",
            "Batch:  85% 33/39 [00:41<00:07,  1.28s/it]\u001b[A\n",
            "Batch:  87% 34/39 [00:43<00:06,  1.28s/it]\u001b[A\n",
            "Batch:  90% 35/39 [00:44<00:05,  1.28s/it]\u001b[A\n",
            "Batch:  92% 36/39 [00:45<00:03,  1.28s/it]\u001b[A\n",
            "Batch:  95% 37/39 [00:46<00:02,  1.28s/it]\u001b[A\n",
            "Batch:  97% 38/39 [00:48<00:01,  1.28s/it]\u001b[A\n",
            "Batch: 100% 39/39 [00:49<00:00,  1.27s/it]\n",
            "08/07/2024 14:41:53 - INFO - Blink -   ***** Saving fine - tuned model *****\n",
            "\n",
            "Evaluation:   0% 0/31 [00:00<?, ?it/s]\u001b[A/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
            "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
            "\n",
            "Evaluation:   3% 1/31 [00:00<00:13,  2.27it/s]\u001b[A\n",
            "Evaluation:   6% 2/31 [00:00<00:13,  2.12it/s]\u001b[A\n",
            "Evaluation:  10% 3/31 [00:01<00:12,  2.17it/s]\u001b[A\n",
            "Evaluation:  13% 4/31 [00:01<00:12,  2.16it/s]\u001b[A\n",
            "Evaluation:  16% 5/31 [00:02<00:12,  2.15it/s]\u001b[A\n",
            "Evaluation:  19% 6/31 [00:02<00:11,  2.15it/s]\u001b[A\n",
            "Evaluation:  23% 7/31 [00:03<00:11,  2.13it/s]\u001b[A\n",
            "Evaluation:  26% 8/31 [00:03<00:10,  2.14it/s]\u001b[A\n",
            "Evaluation:  29% 9/31 [00:04<00:10,  2.12it/s]\u001b[A\n",
            "Evaluation:  32% 10/31 [00:04<00:09,  2.14it/s]\u001b[A\n",
            "Evaluation:  35% 11/31 [00:05<00:09,  2.12it/s]\u001b[A\n",
            "Evaluation:  39% 12/31 [00:05<00:08,  2.13it/s]\u001b[A\n",
            "Evaluation:  42% 13/31 [00:06<00:08,  2.12it/s]\u001b[A\n",
            "Evaluation:  45% 14/31 [00:06<00:08,  2.12it/s]\u001b[A\n",
            "Evaluation:  48% 15/31 [00:07<00:07,  2.12it/s]\u001b[A\n",
            "Evaluation:  52% 16/31 [00:07<00:07,  2.13it/s]\u001b[A\n",
            "Evaluation:  55% 17/31 [00:07<00:06,  2.12it/s]\u001b[A\n",
            "Evaluation:  58% 18/31 [00:08<00:06,  2.12it/s]\u001b[A\n",
            "Evaluation:  61% 19/31 [00:08<00:05,  2.12it/s]\u001b[A\n",
            "Evaluation:  65% 20/31 [00:09<00:05,  2.11it/s]\u001b[A\n",
            "Evaluation:  68% 21/31 [00:09<00:04,  2.11it/s]\u001b[A\n",
            "Evaluation:  71% 22/31 [00:10<00:04,  2.10it/s]\u001b[A\n",
            "Evaluation:  74% 23/31 [00:10<00:03,  2.10it/s]\u001b[A\n",
            "Evaluation:  77% 24/31 [00:11<00:03,  2.10it/s]\u001b[A\n",
            "Evaluation:  81% 25/31 [00:11<00:02,  2.10it/s]\u001b[A\n",
            "Evaluation:  84% 26/31 [00:12<00:02,  2.10it/s]\u001b[A\n",
            "Evaluation:  87% 27/31 [00:12<00:01,  2.09it/s]\u001b[A\n",
            "Evaluation:  90% 28/31 [00:13<00:01,  2.09it/s]\u001b[A\n",
            "Evaluation:  94% 29/31 [00:13<00:00,  2.08it/s]\u001b[A\n",
            "Evaluation:  97% 30/31 [00:14<00:00,  2.09it/s]\u001b[A\n",
            "Evaluation: 100% 31/31 [00:14<00:00,  2.11it/s]\n",
            "num_labels: 50\n",
            "/content/LM-ontology-concept-placement/blink/candidate_ranking/multi_label_evaluation.py:233: RuntimeWarning: invalid value encountered in scalar divide\n",
            "  return intersect_size(yhatmic, ymic, 0) / yhatmic.sum(axis=0)\n",
            "num in insertion_all: 0 num_ave: 0.0\n",
            "num in insertion_any: 0 num_ave: 0.0\n",
            "\n",
            "[MACRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.4835\n",
            "[MICRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, nan, 0.0000, nan, 0.5221\n",
            "rec_at_1: 0.0023\n",
            "prec_at_1: 0.0323\n",
            "rec_at_3: 0.0422\n",
            "prec_at_3: 0.0430\n",
            "rec_at_5: 0.0641\n",
            "prec_at_5: 0.0516\n",
            "rec_at_10: 0.1733\n",
            "prec_at_10: 0.0677\n",
            "rec_at_50: 1.0000\n",
            "prec_at_50: 0.0523\n",
            "ins_at_1_all: 0.0000\n",
            "ins_at_1_any: 0.0323\n",
            "ins_at_3_all: 0.0323\n",
            "ins_at_3_any: 0.0968\n",
            "ins_at_5_all: 0.0323\n",
            "ins_at_5_any: 0.1613\n",
            "ins_at_10_all: 0.0645\n",
            "ins_at_10_any: 0.3548\n",
            "ins_at_50_all: 1.0000\n",
            "ins_at_50_any: 1.0000\n",
            "ins_all: 0.0000\n",
            "ins_any: 0.0000\n",
            "\n",
            "08/07/2024 14:42:09 - INFO - Blink -   \n",
            "\n",
            "Epoch:  25% 1/4 [01:05<03:17, 65.97s/it]\n",
            "Batch:   0% 0/39 [00:00<?, ?it/s]\u001b[A\n",
            "Batch:   3% 1/39 [00:01<00:49,  1.29s/it]\u001b[A\n",
            "Batch:   5% 2/39 [00:02<00:47,  1.30s/it]\u001b[A\n",
            "Batch:   8% 3/39 [00:03<00:46,  1.30s/it]\u001b[A\n",
            "Batch:  10% 4/39 [00:05<00:45,  1.30s/it]\u001b[A\n",
            "Batch:  13% 5/39 [00:06<00:44,  1.31s/it]\u001b[A\n",
            "Batch:  15% 6/39 [00:07<00:43,  1.31s/it]\u001b[A\n",
            "Batch:  18% 7/39 [00:09<00:41,  1.31s/it]\u001b[A\n",
            "Batch:  21% 8/39 [00:10<00:40,  1.31s/it]\u001b[A\n",
            "Batch:  23% 9/39 [00:11<00:39,  1.32s/it]\u001b[A\n",
            "Batch:  26% 10/39 [00:13<00:38,  1.33s/it]\u001b[A\n",
            "Batch:  28% 11/39 [00:14<00:37,  1.32s/it]\u001b[A\n",
            "Batch:  31% 12/39 [00:15<00:35,  1.32s/it]\u001b[A\n",
            "Batch:  33% 13/39 [00:17<00:34,  1.33s/it]\u001b[A\n",
            "Batch:  36% 14/39 [00:18<00:33,  1.33s/it]\u001b[A\n",
            "Batch:  38% 15/39 [00:19<00:31,  1.33s/it]\u001b[A\n",
            "Batch:  41% 16/39 [00:21<00:30,  1.33s/it]\u001b[A\n",
            "Batch:  44% 17/39 [00:22<00:29,  1.33s/it]\u001b[A\n",
            "Batch:  46% 18/39 [00:23<00:27,  1.33s/it]\u001b[A\n",
            "Batch:  49% 19/39 [00:25<00:26,  1.34s/it]\u001b[A\n",
            "Batch:  51% 20/39 [00:26<00:25,  1.34s/it]\u001b[A\n",
            "Batch:  54% 21/39 [00:27<00:24,  1.34s/it]\u001b[A\n",
            "Batch:  56% 22/39 [00:29<00:22,  1.34s/it]\u001b[A\n",
            "Batch:  59% 23/39 [00:30<00:21,  1.34s/it]\u001b[A\n",
            "Batch:  62% 24/39 [00:31<00:20,  1.34s/it]\u001b[A\n",
            "Batch:  64% 25/39 [00:33<00:18,  1.34s/it]\u001b[A\n",
            "Batch:  67% 26/39 [00:34<00:17,  1.34s/it]\u001b[A\n",
            "Batch:  69% 27/39 [00:35<00:16,  1.34s/it]\u001b[A\n",
            "Batch:  72% 28/39 [00:37<00:14,  1.34s/it]\u001b[A\n",
            "Batch:  74% 29/39 [00:38<00:13,  1.34s/it]\u001b[A\n",
            "Batch:  77% 30/39 [00:39<00:12,  1.34s/it]\u001b[A\n",
            "Batch:  79% 31/39 [00:41<00:10,  1.34s/it]\u001b[A\n",
            "Batch:  82% 32/39 [00:42<00:09,  1.34s/it]\u001b[A\n",
            "Batch:  85% 33/39 [00:43<00:08,  1.34s/it]\u001b[A\n",
            "Batch:  87% 34/39 [00:45<00:06,  1.34s/it]\u001b[A\n",
            "Batch:  90% 35/39 [00:46<00:05,  1.34s/it]\u001b[A\n",
            "Batch:  92% 36/39 [00:47<00:04,  1.34s/it]\u001b[A\n",
            "Batch:  95% 37/39 [00:49<00:02,  1.34s/it]\u001b[A\n",
            "Batch:  97% 38/39 [00:50<00:01,  1.34s/it]\u001b[A\n",
            "Batch: 100% 39/39 [00:51<00:00,  1.33s/it]\n",
            "08/07/2024 14:43:01 - INFO - Blink -   ***** Saving fine - tuned model *****\n",
            "\n",
            "Evaluation:   0% 0/31 [00:00<?, ?it/s]\u001b[A/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
            "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
            "\n",
            "Evaluation:   3% 1/31 [00:00<00:15,  1.90it/s]\u001b[A\n",
            "Evaluation:   6% 2/31 [00:01<00:14,  2.00it/s]\u001b[A\n",
            "Evaluation:  10% 3/31 [00:01<00:14,  1.99it/s]\u001b[A\n",
            "Evaluation:  13% 4/31 [00:02<00:13,  2.01it/s]\u001b[A\n",
            "Evaluation:  16% 5/31 [00:02<00:12,  2.00it/s]\u001b[A\n",
            "Evaluation:  19% 6/31 [00:02<00:12,  2.02it/s]\u001b[A\n",
            "Evaluation:  23% 7/31 [00:03<00:11,  2.01it/s]\u001b[A\n",
            "Evaluation:  26% 8/31 [00:03<00:11,  2.02it/s]\u001b[A\n",
            "Evaluation:  29% 9/31 [00:04<00:10,  2.01it/s]\u001b[A\n",
            "Evaluation:  32% 10/31 [00:04<00:10,  2.02it/s]\u001b[A\n",
            "Evaluation:  35% 11/31 [00:05<00:09,  2.01it/s]\u001b[A\n",
            "Evaluation:  39% 12/31 [00:05<00:09,  2.02it/s]\u001b[A\n",
            "Evaluation:  42% 13/31 [00:06<00:08,  2.01it/s]\u001b[A\n",
            "Evaluation:  45% 14/31 [00:06<00:08,  2.02it/s]\u001b[A\n",
            "Evaluation:  48% 15/31 [00:07<00:07,  2.01it/s]\u001b[A\n",
            "Evaluation:  52% 16/31 [00:07<00:07,  2.02it/s]\u001b[A\n",
            "Evaluation:  55% 17/31 [00:08<00:06,  2.01it/s]\u001b[A\n",
            "Evaluation:  58% 18/31 [00:08<00:06,  2.01it/s]\u001b[A\n",
            "Evaluation:  61% 19/31 [00:09<00:05,  2.02it/s]\u001b[A\n",
            "Evaluation:  65% 20/31 [00:09<00:05,  2.01it/s]\u001b[A\n",
            "Evaluation:  68% 21/31 [00:10<00:04,  2.02it/s]\u001b[A\n",
            "Evaluation:  71% 22/31 [00:10<00:04,  2.01it/s]\u001b[A\n",
            "Evaluation:  74% 23/31 [00:11<00:03,  2.02it/s]\u001b[A\n",
            "Evaluation:  77% 24/31 [00:11<00:03,  2.01it/s]\u001b[A\n",
            "Evaluation:  81% 25/31 [00:12<00:02,  2.02it/s]\u001b[A\n",
            "Evaluation:  84% 26/31 [00:12<00:02,  2.01it/s]\u001b[A\n",
            "Evaluation:  87% 27/31 [00:13<00:01,  2.01it/s]\u001b[A\n",
            "Evaluation:  90% 28/31 [00:13<00:01,  2.01it/s]\u001b[A\n",
            "Evaluation:  94% 29/31 [00:14<00:00,  2.00it/s]\u001b[A\n",
            "Evaluation:  97% 30/31 [00:14<00:00,  2.00it/s]\u001b[A\n",
            "Evaluation: 100% 31/31 [00:15<00:00,  2.01it/s]\n",
            "num_labels: 50\n",
            "num in insertion_all: 0 num_ave: 0.0\n",
            "num in insertion_any: 0 num_ave: 0.0\n",
            "\n",
            "[MACRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.5501\n",
            "[MICRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.5643\n",
            "rec_at_1: 0.0373\n",
            "prec_at_1: 0.0968\n",
            "rec_at_3: 0.0580\n",
            "prec_at_3: 0.0860\n",
            "rec_at_5: 0.1108\n",
            "prec_at_5: 0.0903\n",
            "rec_at_10: 0.1667\n",
            "prec_at_10: 0.0677\n",
            "rec_at_50: 1.0000\n",
            "prec_at_50: 0.0523\n",
            "ins_at_1_all: 0.0323\n",
            "ins_at_1_any: 0.0968\n",
            "ins_at_3_all: 0.0323\n",
            "ins_at_3_any: 0.1290\n",
            "ins_at_5_all: 0.0645\n",
            "ins_at_5_any: 0.1935\n",
            "ins_at_10_all: 0.0968\n",
            "ins_at_10_any: 0.2258\n",
            "ins_at_50_all: 1.0000\n",
            "ins_at_50_any: 1.0000\n",
            "ins_all: 0.0000\n",
            "ins_any: 0.0000\n",
            "\n",
            "08/07/2024 14:43:19 - INFO - Blink -   \n",
            "\n",
            "Epoch:  50% 2/4 [02:15<02:16, 68.01s/it]\n",
            "Batch:   0% 0/39 [00:00<?, ?it/s]\u001b[A\n",
            "Batch:   3% 1/39 [00:01<00:50,  1.33s/it]\u001b[A\n",
            "Batch:   5% 2/39 [00:02<00:49,  1.33s/it]\u001b[A\n",
            "Batch:   8% 3/39 [00:04<00:48,  1.34s/it]\u001b[A\n",
            "Batch:  10% 4/39 [00:05<00:46,  1.34s/it]\u001b[A\n",
            "Batch:  13% 5/39 [00:06<00:45,  1.34s/it]\u001b[A\n",
            "Batch:  15% 6/39 [00:08<00:44,  1.34s/it]\u001b[A\n",
            "Batch:  18% 7/39 [00:09<00:42,  1.34s/it]\u001b[A\n",
            "Batch:  21% 8/39 [00:10<00:41,  1.35s/it]\u001b[A\n",
            "Batch:  23% 9/39 [00:12<00:40,  1.34s/it]\u001b[A\n",
            "Batch:  26% 10/39 [00:13<00:38,  1.34s/it]\u001b[A\n",
            "Batch:  28% 11/39 [00:14<00:37,  1.34s/it]\u001b[A\n",
            "Batch:  31% 12/39 [00:16<00:36,  1.34s/it]\u001b[A\n",
            "Batch:  33% 13/39 [00:17<00:35,  1.35s/it]\u001b[A\n",
            "Batch:  36% 14/39 [00:18<00:33,  1.35s/it]\u001b[A\n",
            "Batch:  38% 15/39 [00:20<00:32,  1.35s/it]\u001b[A\n",
            "Batch:  41% 16/39 [00:21<00:31,  1.35s/it]\u001b[A\n",
            "Batch:  44% 17/39 [00:22<00:29,  1.35s/it]\u001b[A\n",
            "Batch:  46% 18/39 [00:24<00:28,  1.35s/it]\u001b[A\n",
            "Batch:  49% 19/39 [00:25<00:26,  1.35s/it]\u001b[A\n",
            "Batch:  51% 20/39 [00:26<00:25,  1.35s/it]\u001b[A\n",
            "Batch:  54% 21/39 [00:28<00:24,  1.35s/it]\u001b[A\n",
            "Batch:  56% 22/39 [00:29<00:22,  1.35s/it]\u001b[A\n",
            "Batch:  59% 23/39 [00:30<00:21,  1.35s/it]\u001b[A\n",
            "Batch:  62% 24/39 [00:32<00:20,  1.35s/it]\u001b[A\n",
            "Batch:  64% 25/39 [00:33<00:18,  1.35s/it]\u001b[A\n",
            "Batch:  67% 26/39 [00:34<00:17,  1.35s/it]\u001b[A\n",
            "Batch:  69% 27/39 [00:36<00:16,  1.35s/it]\u001b[A\n",
            "Batch:  72% 28/39 [00:37<00:14,  1.35s/it]\u001b[A\n",
            "Batch:  74% 29/39 [00:39<00:13,  1.35s/it]\u001b[A\n",
            "Batch:  77% 30/39 [00:40<00:12,  1.35s/it]\u001b[A\n",
            "Batch:  79% 31/39 [00:41<00:10,  1.35s/it]\u001b[A\n",
            "Batch:  82% 32/39 [00:43<00:09,  1.35s/it]\u001b[A\n",
            "Batch:  85% 33/39 [00:44<00:08,  1.35s/it]\u001b[A\n",
            "Batch:  87% 34/39 [00:45<00:06,  1.35s/it]\u001b[A\n",
            "Batch:  90% 35/39 [00:47<00:05,  1.35s/it]\u001b[A\n",
            "Batch:  92% 36/39 [00:48<00:04,  1.35s/it]\u001b[A\n",
            "Batch:  95% 37/39 [00:49<00:02,  1.35s/it]\u001b[A\n",
            "Batch:  97% 38/39 [00:51<00:01,  1.35s/it]\u001b[A\n",
            "Batch: 100% 39/39 [00:52<00:00,  1.35s/it]\n",
            "08/07/2024 14:44:11 - INFO - Blink -   ***** Saving fine - tuned model *****\n",
            "\n",
            "Evaluation:   0% 0/31 [00:00<?, ?it/s]\u001b[A/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
            "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
            "\n",
            "Evaluation:   3% 1/31 [00:00<00:14,  2.14it/s]\u001b[A\n",
            "Evaluation:   6% 2/31 [00:00<00:14,  1.99it/s]\u001b[A\n",
            "Evaluation:  10% 3/31 [00:01<00:13,  2.03it/s]\u001b[A\n",
            "Evaluation:  13% 4/31 [00:01<00:13,  1.99it/s]\u001b[A\n",
            "Evaluation:  16% 5/31 [00:02<00:12,  2.01it/s]\u001b[A\n",
            "Evaluation:  19% 6/31 [00:02<00:12,  2.00it/s]\u001b[A\n",
            "Evaluation:  23% 7/31 [00:03<00:11,  2.01it/s]\u001b[A\n",
            "Evaluation:  26% 8/31 [00:03<00:11,  1.99it/s]\u001b[A\n",
            "Evaluation:  29% 9/31 [00:04<00:10,  2.01it/s]\u001b[A\n",
            "Evaluation:  32% 10/31 [00:04<00:10,  2.00it/s]\u001b[A\n",
            "Evaluation:  35% 11/31 [00:05<00:09,  2.02it/s]\u001b[A\n",
            "Evaluation:  39% 12/31 [00:05<00:09,  2.00it/s]\u001b[A\n",
            "Evaluation:  42% 13/31 [00:06<00:08,  2.01it/s]\u001b[A\n",
            "Evaluation:  45% 14/31 [00:06<00:08,  2.00it/s]\u001b[A\n",
            "Evaluation:  48% 15/31 [00:07<00:07,  2.01it/s]\u001b[A\n",
            "Evaluation:  52% 16/31 [00:07<00:07,  2.00it/s]\u001b[A\n",
            "Evaluation:  55% 17/31 [00:08<00:06,  2.00it/s]\u001b[A\n",
            "Evaluation:  58% 18/31 [00:08<00:06,  2.00it/s]\u001b[A\n",
            "Evaluation:  61% 19/31 [00:09<00:06,  2.00it/s]\u001b[A\n",
            "Evaluation:  65% 20/31 [00:09<00:05,  1.99it/s]\u001b[A\n",
            "Evaluation:  68% 21/31 [00:10<00:04,  2.01it/s]\u001b[A\n",
            "Evaluation:  71% 22/31 [00:10<00:04,  2.00it/s]\u001b[A\n",
            "Evaluation:  74% 23/31 [00:11<00:04,  2.00it/s]\u001b[A\n",
            "Evaluation:  77% 24/31 [00:11<00:03,  1.99it/s]\u001b[A\n",
            "Evaluation:  81% 25/31 [00:12<00:02,  2.01it/s]\u001b[A\n",
            "Evaluation:  84% 26/31 [00:12<00:02,  2.00it/s]\u001b[A\n",
            "Evaluation:  87% 27/31 [00:13<00:01,  2.00it/s]\u001b[A\n",
            "Evaluation:  90% 28/31 [00:13<00:01,  2.00it/s]\u001b[A\n",
            "Evaluation:  94% 29/31 [00:14<00:00,  2.00it/s]\u001b[A\n",
            "Evaluation:  97% 30/31 [00:14<00:00,  2.00it/s]\u001b[A\n",
            "Evaluation: 100% 31/31 [00:15<00:00,  2.00it/s]\n",
            "num_labels: 50\n",
            "num in insertion_all: 0 num_ave: 0.0\n",
            "num in insertion_any: 0 num_ave: 0.0\n",
            "\n",
            "[MACRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.6255\n",
            "[MICRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.6037\n",
            "rec_at_1: 0.0349\n",
            "prec_at_1: 0.0645\n",
            "rec_at_3: 0.0898\n",
            "prec_at_3: 0.0753\n",
            "rec_at_5: 0.1039\n",
            "prec_at_5: 0.0710\n",
            "rec_at_10: 0.2017\n",
            "prec_at_10: 0.0774\n",
            "rec_at_50: 1.0000\n",
            "prec_at_50: 0.0523\n",
            "ins_at_1_all: 0.0323\n",
            "ins_at_1_any: 0.0645\n",
            "ins_at_3_all: 0.0645\n",
            "ins_at_3_any: 0.1613\n",
            "ins_at_5_all: 0.0645\n",
            "ins_at_5_any: 0.1935\n",
            "ins_at_10_all: 0.0968\n",
            "ins_at_10_any: 0.3548\n",
            "ins_at_50_all: 1.0000\n",
            "ins_at_50_any: 1.0000\n",
            "ins_all: 0.0000\n",
            "ins_any: 0.0000\n",
            "\n",
            "08/07/2024 14:44:29 - INFO - Blink -   \n",
            "\n",
            "Epoch:  75% 3/4 [03:25<01:08, 68.84s/it]\n",
            "Batch:   0% 0/39 [00:00<?, ?it/s]\u001b[A\n",
            "Batch:   3% 1/39 [00:01<00:50,  1.33s/it]\u001b[A\n",
            "Batch:   5% 2/39 [00:02<00:49,  1.34s/it]\u001b[A\n",
            "Batch:   8% 3/39 [00:04<00:48,  1.34s/it]\u001b[A\n",
            "Batch:  10% 4/39 [00:05<00:47,  1.34s/it]\u001b[A\n",
            "Batch:  13% 5/39 [00:06<00:45,  1.34s/it]\u001b[A\n",
            "Batch:  15% 6/39 [00:08<00:44,  1.34s/it]\u001b[A\n",
            "Batch:  18% 7/39 [00:09<00:42,  1.34s/it]\u001b[A\n",
            "Batch:  21% 8/39 [00:10<00:41,  1.35s/it]\u001b[A\n",
            "Batch:  23% 9/39 [00:12<00:40,  1.35s/it]\u001b[A\n",
            "Batch:  26% 10/39 [00:13<00:39,  1.35s/it]\u001b[A\n",
            "Batch:  28% 11/39 [00:14<00:37,  1.35s/it]\u001b[A\n",
            "Batch:  31% 12/39 [00:16<00:36,  1.35s/it]\u001b[A\n",
            "Batch:  33% 13/39 [00:17<00:35,  1.35s/it]\u001b[A\n",
            "Batch:  36% 14/39 [00:18<00:33,  1.35s/it]\u001b[A\n",
            "Batch:  38% 15/39 [00:20<00:32,  1.35s/it]\u001b[A\n",
            "Batch:  41% 16/39 [00:21<00:30,  1.35s/it]\u001b[A\n",
            "Batch:  44% 17/39 [00:22<00:29,  1.35s/it]\u001b[A\n",
            "Batch:  46% 18/39 [00:24<00:28,  1.35s/it]\u001b[A\n",
            "Batch:  49% 19/39 [00:25<00:27,  1.35s/it]\u001b[A\n",
            "Batch:  51% 20/39 [00:26<00:25,  1.35s/it]\u001b[A\n",
            "Batch:  54% 21/39 [00:28<00:24,  1.35s/it]\u001b[A\n",
            "Batch:  56% 22/39 [00:29<00:22,  1.35s/it]\u001b[A\n",
            "Batch:  59% 23/39 [00:30<00:21,  1.35s/it]\u001b[A\n",
            "Batch:  62% 24/39 [00:32<00:20,  1.35s/it]\u001b[A\n",
            "Batch:  64% 25/39 [00:33<00:18,  1.35s/it]\u001b[A\n",
            "Batch:  67% 26/39 [00:35<00:17,  1.35s/it]\u001b[A\n",
            "Batch:  69% 27/39 [00:36<00:16,  1.35s/it]\u001b[A\n",
            "Batch:  72% 28/39 [00:37<00:14,  1.35s/it]\u001b[A\n",
            "Batch:  74% 29/39 [00:39<00:13,  1.35s/it]\u001b[A\n",
            "Batch:  77% 30/39 [00:40<00:12,  1.35s/it]\u001b[A\n",
            "Batch:  79% 31/39 [00:41<00:10,  1.35s/it]\u001b[A\n",
            "Batch:  82% 32/39 [00:43<00:09,  1.35s/it]\u001b[A\n",
            "Batch:  85% 33/39 [00:44<00:08,  1.35s/it]\u001b[A\n",
            "Batch:  87% 34/39 [00:45<00:06,  1.35s/it]\u001b[A\n",
            "Batch:  90% 35/39 [00:47<00:05,  1.35s/it]\u001b[A\n",
            "Batch:  92% 36/39 [00:48<00:04,  1.35s/it]\u001b[A\n",
            "Batch:  95% 37/39 [00:49<00:02,  1.35s/it]\u001b[A\n",
            "Batch:  97% 38/39 [00:51<00:01,  1.35s/it]\u001b[A\n",
            "Batch: 100% 39/39 [00:52<00:00,  1.35s/it]\n",
            "08/07/2024 14:45:21 - INFO - Blink -   ***** Saving fine - tuned model *****\n",
            "\n",
            "Evaluation:   0% 0/31 [00:00<?, ?it/s]\u001b[A/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
            "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
            "\n",
            "Evaluation:   3% 1/31 [00:00<00:14,  2.14it/s]\u001b[A\n",
            "Evaluation:   6% 2/31 [00:00<00:14,  2.01it/s]\u001b[A\n",
            "Evaluation:  10% 3/31 [00:01<00:13,  2.04it/s]\u001b[A\n",
            "Evaluation:  13% 4/31 [00:01<00:13,  2.00it/s]\u001b[A\n",
            "Evaluation:  16% 5/31 [00:02<00:12,  2.01it/s]\u001b[A\n",
            "Evaluation:  19% 6/31 [00:02<00:12,  2.00it/s]\u001b[A\n",
            "Evaluation:  23% 7/31 [00:03<00:11,  2.01it/s]\u001b[A\n",
            "Evaluation:  26% 8/31 [00:03<00:11,  2.00it/s]\u001b[A\n",
            "Evaluation:  29% 9/31 [00:04<00:10,  2.01it/s]\u001b[A\n",
            "Evaluation:  32% 10/31 [00:04<00:10,  2.00it/s]\u001b[A\n",
            "Evaluation:  35% 11/31 [00:05<00:09,  2.00it/s]\u001b[A\n",
            "Evaluation:  39% 12/31 [00:05<00:09,  1.99it/s]\u001b[A\n",
            "Evaluation:  42% 13/31 [00:06<00:08,  2.00it/s]\u001b[A\n",
            "Evaluation:  45% 14/31 [00:06<00:08,  2.00it/s]\u001b[A\n",
            "Evaluation:  48% 15/31 [00:07<00:07,  2.00it/s]\u001b[A\n",
            "Evaluation:  52% 16/31 [00:07<00:07,  1.99it/s]\u001b[A\n",
            "Evaluation:  55% 17/31 [00:08<00:06,  2.01it/s]\u001b[A\n",
            "Evaluation:  58% 18/31 [00:08<00:06,  2.00it/s]\u001b[A\n",
            "Evaluation:  61% 19/31 [00:09<00:06,  2.00it/s]\u001b[A\n",
            "Evaluation:  65% 20/31 [00:09<00:05,  2.00it/s]\u001b[A\n",
            "Evaluation:  68% 21/31 [00:10<00:05,  2.00it/s]\u001b[A\n",
            "Evaluation:  71% 22/31 [00:10<00:04,  1.99it/s]\u001b[A\n",
            "Evaluation:  74% 23/31 [00:11<00:04,  1.99it/s]\u001b[A\n",
            "Evaluation:  77% 24/31 [00:11<00:03,  1.99it/s]\u001b[A\n",
            "Evaluation:  81% 25/31 [00:12<00:03,  1.99it/s]\u001b[A\n",
            "Evaluation:  84% 26/31 [00:12<00:02,  1.99it/s]\u001b[A\n",
            "Evaluation:  87% 27/31 [00:13<00:02,  2.00it/s]\u001b[A\n",
            "Evaluation:  90% 28/31 [00:14<00:01,  1.99it/s]\u001b[A\n",
            "Evaluation:  94% 29/31 [00:14<00:01,  1.99it/s]\u001b[A\n",
            "Evaluation:  97% 30/31 [00:15<00:00,  1.99it/s]\u001b[A\n",
            "Evaluation: 100% 31/31 [00:15<00:00,  2.00it/s]\n",
            "num_labels: 50\n",
            "num in insertion_all: 0 num_ave: 0.0\n",
            "num in insertion_any: 0 num_ave: 0.0\n",
            "\n",
            "[MACRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.6064\n",
            "[MICRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.5897\n",
            "rec_at_1: 0.0373\n",
            "prec_at_1: 0.0968\n",
            "rec_at_3: 0.0472\n",
            "prec_at_3: 0.0645\n",
            "rec_at_5: 0.0721\n",
            "prec_at_5: 0.0710\n",
            "rec_at_10: 0.1876\n",
            "prec_at_10: 0.0742\n",
            "rec_at_50: 1.0000\n",
            "prec_at_50: 0.0523\n",
            "ins_at_1_all: 0.0323\n",
            "ins_at_1_any: 0.0968\n",
            "ins_at_3_all: 0.0323\n",
            "ins_at_3_any: 0.1290\n",
            "ins_at_5_all: 0.0323\n",
            "ins_at_5_any: 0.1935\n",
            "ins_at_10_all: 0.0968\n",
            "ins_at_10_any: 0.3548\n",
            "ins_at_50_all: 1.0000\n",
            "ins_at_50_any: 1.0000\n",
            "ins_all: 0.0000\n",
            "ins_any: 0.0000\n",
            "\n",
            "08/07/2024 14:45:39 - INFO - Blink -   \n",
            "\n",
            "Epoch: 100% 4/4 [04:35<00:00, 68.78s/it]\n",
            "08/07/2024 14:45:39 - INFO - Blink -   The training took 4.585304951667785 minutes\n",
            "\n",
            "08/07/2024 14:45:39 - INFO - Blink -   Best performance in epoch: 0\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "Some weights of the model checkpoint at microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n",
            "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "INFO:deeponto:32g maximum memory allocated to JVM.\n",
            "INFO:deeponto:JVM started successfully.\n",
            "2024-08-07 14:45:53.537985: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2024-08-07 14:45:53.568286: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2024-08-07 14:45:53.576810: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2024-08-07 14:45:56.243065: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "INFO:numexpr.utils:NumExpr defaulting to 2 threads.\n",
            "INFO:datasets:PyTorch version 1.11.0 available.\n",
            "INFO:datasets:Polars version 0.20.2 available.\n",
            "INFO:datasets:TensorFlow version 2.17.0 available.\n",
            "INFO:datasets:JAX version 0.4.26 available.\n",
            "INFO:faiss.loader:Loading faiss with AVX512 support.\n",
            "INFO:faiss.loader:Successfully loaded faiss with AVX512 support.\n",
            "entity_catalogue_postfix: -edges-all\n",
            "params_dict_str[0]: {'silent': False, 'debug': False, 'debug_max_lines': 200, 'data_parallel': True, 'no_cuda': False, 'top_k': 50, 'seed': 52313, 'zeshel': False, 'max_seq_length': 160, 'max_context_length': 32, 'max_cand_length': 128, 'path_to_model': None, 'bert_model': 'microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext', 'pull_from_layer': -1, 'lowercase': True, 'context_key': 'context', 'use_context': False, 'out_dim': 1, 'add_linear': True, 'data_path': 'models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top50_candidates', 'output_path': 'models/crossencoder/mm+Disease-2017AA/original-NIL-top50-pubmedbert', 'use_triplet_loss_bi_enc': False, 'use_miner_bi_enc': False, 'use_ori_classification': True, 'use_NIL_classification': False, 'lambda_NIL': 0.25, 'use_score_features': False, 'use_score_pooling': False, 'use_men_only_score_ft': False, 'use_extra_features': False, 'use_BM25': False, 'aggregating_factor': 20, 'evaluate': False, 'output_eval_file': None, 'limit_by_max_lines': False, 'max_number_of_train_size': 300000, 'train_batch_size': 1, 'max_grad_norm': 1.0, 'learning_rate': 3e-05, 'num_train_epochs': 4, 'limit_by_train_steps': False, 'max_num_train_steps': 20000, 'print_interval': 100, 'eval_interval': 2000, 'save_interval': 1, 'save_model_epoch_parts': True, 'warmup_proportion': 0.1, 'gradient_accumulation_steps': 1, 'type_optimization': 'all_encoder_layers', 'shuffle': False, 'optimize_NIL': False, 'use_preprocessed_data': False, 'use_full_training_data': False, 'fix_seeds': True, 'eval_batch_size': 1, 'mode': 'valid', 'save_topk_result': False, 'encode_batch_size': 8, 'use_synonyms': False, 'use_NIL_tag': False, 'use_NIL_desc': False, 'use_NIL_desc_tag': False, 'cand_pool_path': None, 'cand_encode_path': None, 'use_NIL_classification_infer': False, 'add_NIL_to_bi_enc_pred': False, 'NIL_ent_ind': 64076}\n",
            "args: Namespace(faiss_index=None, index_path=None, test_entities=None, test_mentions=None, interactive=False, biencoder_model='models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/pytorch_model.bin', biencoder_config='models/biencoder_custom_base.json', entity_catalogue='ontologies/SNOMEDCT-US-20140901-Disease-edges-all.jsonl', entity_ids='preprocessing/saved_cand_ids_snomedct-us-20140901-disease_edges_all-sapbert_re_tr.pt', entity_encoding='models/SNOMEDCT-US-20140901-Disease_ent_enc_re_tr/SNOMEDCT-US-20140901-Disease_edges_all-sapbert_ent_enc_re_tr.t7', crossencoder_model='models/crossencoder/mm+Disease-2017AA/original-NIL-top50-pubmedbert/pytorch_model.bin', crossencoder_config='models/crossencoder_custom_base.json', output_path='output', fast=False, candidate_path='models/candidates', prediction_path='models/crossencoder/mm+Disease-2017AA/original-NIL-top50-pubmedbert', use_ori_classification=True, use_NIL_classification=False, use_NIL_classification_infer=False, lambda_NIL=0.25, use_score_features=False, use_score_pooling=False, use_men_only_score_ft=False, use_extra_features=False, data='mm+Disease-2017AA', onto_name='SNOMEDCT-US-20140901-Disease', onto_ver='', snomed_subset='Disease', ontology_fn='ontologies/SNOMEDCT-US-20140901-Disease-final.owl', iri_prefix='http://snomed.info/id/', onto_complex_concept_setting='-all-complexEdge', lowercase=True, biencoder_bert_model='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', biencoder_model_name='mm+Disease2017AA-tl-sapbert-NIL-bs16-debug', biencoder_model_size='base', use_context=True, max_cand_length='128', eval_batch_size='8', use_synonyms=False, NIL_enc_mark='_edges_all-sapbert', top_k=50, top_k_cand_seed=25, edge_cand_enrich=True, edge_ranking_by_score=True, use_leaf_edge_score=False, LEAF_EDGE_SCORE=1000, aggregating_factor=1, crossencoder_bert_model='microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext', cross_model_setting='original-NIL-top50-pubmedbert', cross_model_size='base', set_NIL_as_cand=False, use_NIL_tag=False, use_NIL_desc=False, use_NIL_desc_tag=False, with_NIL_infer=False, th_NIL_bi_enc=0, th_NIL_cross_enc=0, use_BM25=False, save_cand=False, cand_only=False, marking='_edges_all-sapbert_top50-pubmedbert', debug=True, no_cuda=False, NIL_concept='SCTID-less', measure_wp=False)\n",
            "INFO:Blink:loading biencoder model\n",
            "biencoder bert model: cambridgeltl/SapBERT-from-PubMedBERT-fulltext\n",
            "self.n_gpu: 1\n",
            "self.device in BiEncoderRanker: cuda\n",
            "params[\"lowercase\"]: True\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model built\n",
            "model loaded\n",
            "moving model to cuda\n",
            "model moved to cuda\n",
            "INFO:Blink:loading crossencoder model\n",
            "crossencoder bert model: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext\n",
            "Some weights of the model checkpoint at microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias', 'cls.seq_relationship.weight']\n",
            "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "INFO:Blink:loading candidate entities\n",
            "INFO:Blink:mm-valid-in-KB-sample\n",
            "INFO:Blink:test dataset mode\n",
            "INFO:Blink:100/100 samples considered\n",
            "keep_all: False\n",
            "INFO:Blink:preparing data for biencoder\n",
            "args.use_NIL_tag: False\n",
            "args.use_NIL_desc: False\n",
            "args.use_NIL_desc_tag: False\n",
            "num of samples: 100\n",
            "label_text_vecs: tensor([[   2, 2535, 2849,  ...,    0,    0,    0],\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0],\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0],\n",
            "        [   2, 2573,   12,  ...,    0,    0,    0],\n",
            "        [   2, 2535, 2849,  ...,    0,    0,    0]])\n",
            "100% 237826/237826 [00:01<00:00, 162223.92it/s]\n",
            "INFO:Blink:run biencoder\n",
            "100% 13/13 [00:02<00:00,  4.81it/s]\n",
            "labels: 100 1\n",
            "nns: 100 50\n",
            "all_scores_sorted: 100 50\n",
            "all_scores: 100 237826\n",
            "INFO:Blink:Enriching edge candidates\n",
            "INFO:Blink:args.top_k_cand_seed: 25\n",
            "100% 100/100 [00:00<00:00, 1540.51it/s]\n",
            "nns: 100 50\n",
            "labels: 100 [206340] <class 'numpy.ndarray'>\n",
            "nns: 100 [236181   1861 236180  82882  82889  15294  82883 198233   1860 153033\n",
            "  82880 216418 216760 107821  89899 216508 216753 234856  82492 196277\n",
            "  82487 158953 153020 214014 198282  14982 152987 153042  82491 153035\n",
            " 153023 153043 153022 224463  82886 153048  82493 153037 153027 153017\n",
            " 153044 153049 153031 153039 153038  15227 107876  82887 153021 153034]\n",
            "missed case (ind 0, within first 5): ('404684003', '264491001') not in [('[EX.](<http://snomed.info/id/609096000> [EX.](<http://snomed.info/id/47429007> <http://snomed.info/id/238521005>))', 'SCTID_NULL'), ('105715005', 'SCTID_NULL'), ('SCTID_THING', '[EX.](<http://snomed.info/id/609096000> [EX.](<http://snomed.info/id/47429007> <http://snomed.info/id/238521005>))'), ('238521005', '58306008'), ('238521005', 'SCTID_NULL'), ('36465006', '123932008'), ('238521005', '58419006'), ('58306008', 'SCTID_NULL'), ('64572001', '105715005'), ('275390005', '275389001'), ('238521005', '95349002'), ('64572001', '241897004'), ('64572001', '241893000'), ('275390005', '36465006'), ('242838009', 'SCTID_NULL'), ('64572001', '275390005'), ('64572001', '241783002'), ('95349002', 'SCTID_NULL'), ('87628006', '238415008'), ('371582002', '238414007'), ('87628006', '238413001'), ('38729007', 'SCTID_NULL'), ('275390005', '123932008'), ('60225005', 'SCTID_NULL'), ('58419006', 'SCTID_NULL'), ('275390005', 'SCTID_NULL'), ('86068002', '123932008'), ('275390005', '123933003'), ('87628006', '238414007'), ('275390005', '123934009'), ('275390005', '123905003'), ('275390005', '123927007'), ('275390005', '359680004'), ('64572001', '238413001'), ('53597009', '58306008'), ('275390005', '123926003'), ('87628006', '402936008'), ('275390005', '123917001'), ('275390005', '123915009'), ('275390005', '123899003'), ('275390005', '123886001'), ('275390005', '123916005'), ('275390005', '123938007'), ('275390005', '123907006'), ('275390005', '123937002'), ('36465006', 'SCTID_NULL'), ('75478009', '36465006'), ('53597009', '58419006'), ('275390005', '123895009'), ('275390005', '123914008')]\n",
            "missed case (ind 1, within first 5): ('404684003', '93574001') not in [('[EX.](<609096000> [EX.](<255234002> <34014006>))', '402899000'), ('[EX.](<609096000> [EX.](<47429007> <87628006>))', '402968000'), ('64572001', '402150002'), ('[EX.](<609096000> [EX.](<255234002> <40733004>))', '403272001'), ('[EX.](<609096000> [EX.](<363705008> <238812004>))', '40070004'), ('10461000', '47027001'), ('49701002', '47027001'), ('[EX.](<609096000> [EX.](<47429007> <87628006>))', '402970009'), ('10461000', '428471001'), ('64572001', '71571008'), ('123946008', '238839007'), ('[EX.](<609096000> [EX.](<363705008> <238812004>))', '36420002'), ('10461000', '42089007'), ('10461000', '19108007'), ('10461000', '33155001'), ('609337004', 'SCTID_NULL'), ('10461000', '44352004'), ('123946008', '238847007'), ('262550002', '47027001'), ('87628006', '238413001'), ('363137000', '238834002'), ('[EX.](<609096000> [EX.](<42752001> <40733004>))', '414492009'), ('10461000', '25726008'), ('10461000', '76584003'), ('[EX.](<http://snomed.info/id/609096000> [EX.](<http://snomed.info/id/47429007> <http://snomed.info/id/238521005>))', 'SCTID_NULL'), ('87628006', '238415008'), ('87628006', '402936008'), ('262550002', '428471001'), ('262550002', '33155001'), ('262550002', '44352004'), ('10461000', 'SCTID_NULL'), ('262550002', '19108007'), ('262550002', '25726008'), ('87628006', '238414007'), ('262550002', '42089007'), ('262550002', 'SCTID_NULL'), ('363137000', 'SCTID_NULL'), ('[EX.](<609096000> [EX.](<255234002> <40733004>))', '55768006'), ('[EX.](<609096000> [EX.](<47429007> <87628006>))', '402969008'), ('123946008', 'SCTID_NULL'), ('[EX.](<609096000> [EX.](<255234002> <34014006>))', 'SCTID_NULL'), ('64572001', '238413001'), ('276433004', 'SCTID_NULL'), ('[EX.](<609096000> [EX.](<42752001> <40733004>))', 'SCTID_NULL'), ('[EX.](<609096000> [EX.](<255234002> <40733004>))', 'SCTID_NULL'), ('87628006', 'SCTID_NULL'), ('49701002', 'SCTID_NULL'), ('32895009', 'SCTID_NULL'), ('64557000', 'SCTID_NULL'), ('409985002', 'SCTID_NULL')]\n",
            "missed case (ind 2, within first 5): ('404684003', '93048002') not in [('111842002', 'SCTID_NULL'), ('409904000', '111842002'), ('77876004', 'SCTID_NULL'), ('78355003', '111842002'), ('302231008', '90974009'), ('64572001', '254211001'), ('128945009', '90974009'), ('41116009', '82074004'), ('90974009', 'SCTID_NULL'), ('47375003', '71299003'), ('73787001', '77876004'), ('302231008', '71299003'), ('416372004', 'SCTID_NULL'), ('406625002', 'SCTID_NULL'), ('23513009', '77876004'), ('41116009', '93465000'), ('41116009', '13243006'), ('406622004', 'SCTID_NULL'), ('417627005', '416372004'), ('5841002', 'SCTID_NULL'), ('406621006', 'SCTID_NULL'), ('41116009', '53243005'), ('41116009', '23191004'), ('24313004', '68814008'), ('41116009', '240688009'), ('10443009', '71299003'), ('41116009', '90911007'), ('302231008', 'SCTID_NULL'), ('24313004', 'SCTID_NULL'), ('47375003', 'SCTID_NULL'), ('398557001', 'SCTID_NULL'), ('41116009', 'SCTID_NULL'), ('78355003', 'SCTID_NULL'), ('409904000', 'SCTID_NULL'), ('17322007', 'SCTID_NULL'), ('416574002', 'SCTID_NULL'), ('123898006', '68814008'), ('87628006', 'SCTID_NULL'), ('64572001', 'SCTID_NULL'), ('128945009', 'SCTID_NULL'), ('34014006', 'SCTID_NULL'), ('40733004', 'SCTID_NULL'), ('371583007', 'SCTID_NULL'), ('10443009', 'SCTID_NULL'), ('73787001', 'SCTID_NULL'), ('123898006', 'SCTID_NULL'), ('275390005', 'SCTID_NULL'), ('59258008', 'SCTID_NULL'), ('23513009', 'SCTID_NULL'), ('47969004', 'SCTID_NULL')]\n",
            "missed case (ind 4, within first 5): ('404684003', '93329009') not in [('75934005', '14210003'), ('64572001', '363090004'), ('32895009', '286920009'), ('75934005', '61961002'), ('75934005', '16784003'), ('23585005', '70199000'), ('237911005', '37548006'), ('75934005', '23585005'), ('237911005', '52692001'), ('75934005', '44779003'), ('75934005', '190948002'), ('[EX.](<609096000> [EX.](<42752001> <267437005>))', '420146005'), ('363090004', '416676005'), ('190680002', 'SCTID_NULL'), ('235903001', '6075009'), ('75934005', '42930003'), ('237911005', '190737001'), ('235903001', '41527003'), ('235903001', '40191005'), ('238017009', '190794006'), ('363090004', 'SCTID_NULL'), ('237911005', '64235006'), ('75934005', '78548001'), ('235903001', '29291001'), ('237911005', '124258007'), ('235903001', '11179002'), ('235903001', '37666005'), ('75934005', '423148005'), ('75934005', '4887000'), ('44779003', 'SCTID_NULL'), ('237911005', 'SCTID_NULL'), ('235903001', '7265005'), ('235903001', '66937008'), ('86095007', '70199000'), ('44779003', '64235006'), ('235856003', '6075009'), ('75934005', '416676005'), ('75934005', '64235006'), ('75934005', '21529005'), ('75934005', '6075009'), ('75934005', '129456006'), ('75934005', '27718001'), ('75934005', '61764000'), ('64572001', '78548001'), ('64572001', '61961002'), ('23585005', 'SCTID_NULL'), ('75934005', '63329001'), ('75934005', '273700000'), ('75934005', '67845009'), ('75934005', '124680001')]\n",
            "biencoder accuracy: 0.0000\n",
            "biencoder recall@50: 0.0500\n",
            "  0% 0/100 [00:00<?, ?it/s]context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'dispersal', 'of', 'grape', 'rot', '##s', '[unused1]'], 'mention_ids': [1, 15239, 1927, 18673, 5337, 1036, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', '.', 'despite', 'this', ',', 'their', 'possible', 'role', 'in', 'disease', 'facilitation', 'and', '[unused0]', 'dispersal', 'of', 'grape', 'rot', '##s', '[unused1]', 'has', 'not', 'been', 'explored', '.', 'we', 'tested', 'the', 'hypothesis', 'that', 'the', 'paper', '[SEP]'], 'ids': [2, 18, 4281, 2052, 16, 2310, 3216, 2770, 1922, 2573, 17477, 1930, 1, 15239, 1927, 18673, 5337, 1036, 1, 2258, 2084, 2252, 7255, 18, 2038, 3620, 1920, 4676, 1988, 1920, 4648, 3], 'contextual_tokens': ['[CLS]', '.', 'despite', 'this', ',', 'their', 'possible', 'role', 'in', 'disease', 'facilitation', 'and', '[unused0]', 'dispersal', 'of', 'grape', 'rot', '##s', '[unused1]', 'has', 'not', 'been', 'explored', '.', 'we', 'tested', 'the', 'hypothesis', 'that', 'the', 'paper', '[SEP]'], 'contextual_ids': [2, 18, 4281, 2052, 16, 2310, 3216, 2770, 1922, 2573, 17477, 1930, 1, 15239, 1927, 18673, 5337, 1036, 1, 2258, 2084, 2252, 7255, 18, 2038, 3620, 1920, 4676, 1988, 1920, 4648, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'disease', '[unused1]'], 'mention_ids': [1, 2573, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'of', 'fruit', 'flies', 'in', 'facilitating', 'symptoms', 'and', 'vector', '##ing', 'the', 'microorganisms', 'of', 'this', '[unused0]', 'disease', '[unused1]', 'complex', '.', 'like', 'fruit', 'flies', ',', 'social', 'wasp', '##s', 'are', 'abundant', 'in', 'vin', '##ey', '[SEP]'], 'ids': [2, 1927, 9021, 10130, 1922, 12440, 3601, 1930, 4580, 1941, 1920, 11679, 1927, 2052, 1, 2573, 1, 2796, 18, 3189, 9021, 10130, 16, 3686, 21830, 1036, 2032, 7437, 1922, 11132, 10676, 3], 'contextual_tokens': ['[CLS]', 'of', 'fruit', 'flies', 'in', 'facilitating', 'symptoms', 'and', 'vector', '##ing', 'the', 'microorganisms', 'of', 'this', '[unused0]', 'disease', '[unused1]', 'complex', '.', 'like', 'fruit', 'flies', ',', 'social', 'wasp', '##s', 'are', 'abundant', 'in', 'vin', '##ey', '[SEP]'], 'contextual_ids': [2, 1927, 9021, 10130, 1922, 12440, 3601, 1930, 4580, 1941, 1920, 11679, 1927, 2052, 1, 2573, 1, 2796, 18, 3189, 9021, 10130, 16, 3686, 21830, 1036, 2032, 7437, 1922, 11132, 10676, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'diseased', '[unused1]'], 'mention_ids': [1, 16663, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'yers', '##inia', 'ru', '##ck', '##eri', 'isolates', 'recovered', 'from', '[unused0]', 'diseased', '[unused1]', 'atlantic', 'salmon', '(', 'sal', '##mo', 'sal', '##ar', ')', 'in', 'scotland', 'are', 'more', 'diverse', 'than', 'those', 'from', 'rainbow', 'trout', '(', '[SEP]'], 'ids': [2, 29323, 14102, 7212, 12940, 18842, 5645, 8294, 2037, 1, 16663, 1, 17717, 9590, 12, 3370, 12492, 3370, 1933, 13, 1922, 24534, 2032, 2253, 6445, 2254, 2690, 2037, 28945, 23071, 12, 3], 'contextual_tokens': ['[CLS]', 'yers', '##inia', 'ru', '##ck', '##eri', 'isolates', 'recovered', 'from', '[unused0]', 'diseased', '[unused1]', 'atlantic', 'salmon', '(', 'sal', '##mo', 'sal', '##ar', ')', 'in', 'scotland', 'are', 'more', 'diverse', 'than', 'those', 'from', 'rainbow', 'trout', '(', '[SEP]'], 'contextual_ids': [2, 29323, 14102, 7212, 12940, 18842, 5645, 8294, 2037, 1, 16663, 1, 17717, 9590, 12, 3370, 12492, 3370, 1933, 13, 1922, 24534, 2032, 2253, 6445, 2254, 2690, 2037, 28945, 23071, 12, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'hypoxia', '[unused1]'], 'mention_ids': [1, 7264, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'formula', ':', 'see', 'text', ']', ',', 'or', 'aerobic', 'scope', 'between', 'fish', 'raised', 'in', '[unused0]', 'hypoxia', '[unused1]', 'or', 'normoxia', '.', 'there', 'was', 'some', 'evidence', 'that', 'hypoxia', 'tolerance', 'was', 'lower', '(', 'higher', '[SEP]'], 'ids': [2, 9378, 30, 3365, 5316, 39, 16, 2014, 11251, 11440, 2192, 5724, 8930, 1922, 1, 7264, 1, 2014, 23912, 18, 2276, 1982, 2673, 3095, 1988, 7264, 6555, 1982, 2826, 12, 2580, 3], 'contextual_tokens': ['[CLS]', 'formula', ':', 'see', 'text', ']', ',', 'or', 'aerobic', 'scope', 'between', 'fish', 'raised', 'in', '[unused0]', 'hypoxia', '[unused1]', 'or', 'normoxia', '.', 'there', 'was', 'some', 'evidence', 'that', 'hypoxia', 'tolerance', 'was', 'lower', '(', 'higher', '[SEP]'], 'contextual_ids': [2, 9378, 30, 3365, 5316, 39, 16, 2014, 11251, 11440, 2192, 5724, 8930, 1922, 1, 7264, 1, 2014, 23912, 18, 2276, 1982, 2673, 3095, 1988, 7264, 6555, 1982, 2826, 12, 2580, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'disease', '[unused1]'], 'mention_ids': [1, 2573, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'sequencing', 'approaches', 'to', 'define', 'the', 'role', 'of', 'the', 'autophagy', 'lysosomal', 'pathway', 'in', 'human', '[unused0]', 'disease', '[unused1]', ':', 'the', 'example', 'of', 'lys', '##ople', '##x', 'next', '-', 'generation', 'sequencing', '(', 'ngs', ')', '[SEP]'], 'ids': [2, 4754, 4895, 1942, 7425, 1920, 2770, 1927, 1920, 6661, 12097, 3374, 1922, 2616, 1, 2573, 1, 30, 1920, 3534, 1927, 3629, 4279, 1039, 4560, 17, 4841, 4754, 12, 14960, 13, 3], 'contextual_tokens': ['[CLS]', 'sequencing', 'approaches', 'to', 'define', 'the', 'role', 'of', 'the', 'autophagy', 'lysosomal', 'pathway', 'in', 'human', '[unused0]', 'disease', '[unused1]', ':', 'the', 'example', 'of', 'lys', '##ople', '##x', 'next', '-', 'generation', 'sequencing', '(', 'ngs', ')', '[SEP]'], 'contextual_ids': [2, 4754, 4895, 1942, 7425, 1920, 2770, 1927, 1920, 6661, 12097, 3374, 1922, 2616, 1, 2573, 1, 30, 1920, 3534, 1927, 3629, 4279, 1039, 4560, 17, 4841, 4754, 12, 14960, 13, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'disorders', '[unused1]'], 'mention_ids': [1, 4414, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'insertion', '/', 'deletion', '(', 'i', '/', 'd', ')', 'polymorphism', 'on', 'obesity', 'and', 'related', '[unused0]', 'disorders', '[unused1]', 'was', 'studied', 'in', 'several', 'populations', ',', 'because', 'of', 'its', 'effect', 'on', 'ace', 'activity', '.', '[SEP]'], 'ids': [2, 7413, 19, 5541, 12, 51, 19, 46, 13, 8218, 1990, 5379, 1930, 2712, 1, 4414, 1, 1982, 3942, 1922, 2980, 4098, 16, 2882, 1927, 2496, 2495, 1990, 8298, 2455, 18, 3], 'contextual_tokens': ['[CLS]', 'insertion', '/', 'deletion', '(', 'i', '/', 'd', ')', 'polymorphism', 'on', 'obesity', 'and', 'related', '[unused0]', 'disorders', '[unused1]', 'was', 'studied', 'in', 'several', 'populations', ',', 'because', 'of', 'its', 'effect', 'on', 'ace', 'activity', '.', '[SEP]'], 'contextual_ids': [2, 7413, 19, 5541, 12, 51, 19, 46, 13, 8218, 1990, 5379, 1930, 2712, 1, 4414, 1, 1982, 3942, 1922, 2980, 4098, 16, 2882, 1927, 2496, 2495, 1990, 8298, 2455, 18, 3]}\n",
            "100% 100/100 [00:00<00:00, 494.55it/s]\n",
            "topk in prepare_crossencoder_candidates: 50\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'something', 'that', 'is', 'role', 'group', '(', 'attribute', ')', 'some', 'something', 'that', 'is', 'associated', 'with', '(', 'attribute', ')', 'some', 'phyto', '##ph', '##oto', '##der', '##ma', '##titis', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 15126, 1988, 1977, 2770, 2210, 12, 16008, 13, 2673, 15126, 1988, 1977, 2458, 1956, 12, 16008, 13, 2673, 25512, 2102, 11099, 2179, 3368, 11648, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'infectious', 'disease', 'due', 'to', 'plant', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 7465, 2573, 2810, 1942, 4496, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'top', '[unused5]', 'something', 'that', 'is', 'role', 'group', '(', 'attribute', ')', 'some', 'something', 'that', 'is', 'associated', 'with', '(', 'attribute', ')', 'some', 'phyto', '##ph', '##oto', '##der', '##ma', '##titis', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 3700, 1, 15126, 1988, 1977, 2770, 2210, 12, 16008, 13, 2673, 15126, 1988, 1977, 2458, 1956, 12, 16008, 13, 2673, 25512, 2102, 11099, 2179, 3368, 11648, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'something', 'that', 'is', 'role', 'group', '(', 'attribute', ')', 'some', 'something', 'that', 'after', '(', 'attribute', ')', 'some', 'viral', 'disease', '(', 'disorder', ')', '[unused5]', 'ane', '##to', '##der', '##ma', 'following', 'varicella', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 15126, 1988, 1977, 2770, 2210, 12, 16008, 13, 2673, 15126, 1988, 2256, 12, 16008, 13, 2673, 4324, 2573, 12, 4770, 13, 1, 9860, 4575, 2179, 3368, 2894, 29438, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'something', 'that', 'is', 'role', 'group', '(', 'attribute', ')', 'some', 'something', 'that', 'is', 'associated', 'with', '(', 'attribute', ')', 'some', 'bacterial', 'infectious', 'disease', '(', 'disorder', ')', '[unused5]', 'erythema', 'nod', '##osum', 'due', 'to', 'bacterial', 'infection', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 15126, 1988, 1977, 2770, 2210, 12, 16008, 13, 2673, 15126, 1988, 1977, 2458, 1956, 12, 16008, 13, 2673, 4543, 7465, 2573, 12, 4770, 13, 1, 23349, 3889, 14740, 2810, 1942, 4543, 2909, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'disease', '(', 'disorder', ')', '[unused5]', 'insect', 'bite', 'reaction', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 2573, 12, 4770, 13, 1, 11422, 21095, 3227, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'salmon', 'poisoning', 'disease', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 9590, 18722, 2573, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'disease', 'due', 'to', 'ricketts', '##ial', '##es', '(', 'disorder', ')', '[unused5]', 'salmon', 'poisoning', 'disease', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 2573, 2810, 1942, 24479, 2086, 1918, 12, 4770, 13, 1, 9590, 18722, 2573, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'herpesvirus', 'disease', 'of', 'salmon', '##ids', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 25718, 2573, 1927, 9590, 3753, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "label_input_list: [-1 -1 -1 39 -1 -1 -1 -1 -1 -1 -1 -1 -1 44 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1  6 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 29 -1 -1 -1 -1 -1 -1 -1 -1  4 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1]\n",
            "label_input: tensor([-1, -1, -1, 39, -1, -1, -1, -1, -1, -1, -1, -1, -1, 44, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  6, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, 29, -1, -1, -1, -1, -1, -1, -1, -1,  4, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1])\n",
            "tensor_is_NIL_labels: tensor([False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False])\n",
            "label_input[0]: tensor(-1)\n",
            "samples[0]: {'context_left': 'polymicrobial disease of vineyards that causes millions of dollars in lost revenue per year due to decreased quality of grapes and resultant wine. the disease is associated with damaged berries infected with a community of acetic acid bacteria, yeasts, and filamentous fungi that results in rotting berries with high amounts of undesirable volatile acidity. many insect species cause the initial grape berry damage that can lead to this disease, but most studies have focused on the role of fruit flies in facilitating symptoms and vectoring the microorganisms of this disease complex. like fruit flies, social wasps are abundant in vineyards where they feed on ripe berries and cause significant damage, while also dispersing yeasts involved in wine fermentation. despite this, their possible role in disease facilitation and ', 'mention': 'dispersal of grape rots', 'context_right': ' has not been explored. we tested the hypothesis that the paper wasp polistes dominulus could facilitate grape sour rot in the absence of other insect vectors. using marker gene sequencing we characterized the bacterial and fungal community of wild-caught adults. we used a sterilized foraging arena to determine if these wasps transfer viable microorganisms when foraging. we then tested if wasps harboring their native microbial community, or those inoculated with sour rot, had an effect on grape sour rot incidence and severity using a laboratory foraging arena. we found that all wasps harbor some portion of the sour rot microbial community and that they have the ability to transfer viable microorganisms when foraging. foraging by inoculated and uninoculated wasps led to an increase in berry rot disease', 'label_concept_UMLS': 'C0012634', 'label_concept': 206340, 'label_concept_ori': '64572001', 'entity_label_id': 53968, 'entity_label': '', 'entity_label_title': 'disease (disorder)', 'parent_concept': '404684003', 'child_concept': '264491001', 'parent': 'clinical finding (finding)', 'child': 'mullerian remnant (disorder)', 'edge_label_id': 206340, 'label_concept_str': ('404684003', '264491001'), 'row_ind_men': 0}\n",
            "max_seq_length: 160\n",
            "doc_ind_aggregated_strs:\n",
            " 0\n",
            "1\n",
            "2|75|92\n",
            "3\n",
            "4\n",
            "5\n",
            "6\n",
            "7\n",
            "8\n",
            "9|50\n",
            "10|95\n",
            "11\n",
            "12\n",
            "13\n",
            "14\n",
            "15\n",
            "16|57\n",
            "17\n",
            "18\n",
            "19\n",
            "20\n",
            "21\n",
            "22\n",
            "23\n",
            "24\n",
            "25|51\n",
            "26\n",
            "27\n",
            "28|82\n",
            "29\n",
            "30\n",
            "31|89\n",
            "32\n",
            "33|61\n",
            "34|53\n",
            "35\n",
            "36\n",
            "37\n",
            "38\n",
            "39\n",
            "40\n",
            "41\n",
            "42|44|68\n",
            "43\n",
            "45|48\n",
            "46|73\n",
            "47\n",
            "49\n",
            "52\n",
            "54\n",
            "55\n",
            "56\n",
            "58\n",
            "59\n",
            "60\n",
            "62\n",
            "63\n",
            "64\n",
            "65\n",
            "66\n",
            "67\n",
            "69|83\n",
            "70\n",
            "71\n",
            "72\n",
            "74\n",
            "76|96\n",
            "77\n",
            "78\n",
            "79\n",
            "80\n",
            "81\n",
            "84\n",
            "85\n",
            "86\n",
            "87\n",
            "88|90\n",
            "91\n",
            "93\n",
            "94\n",
            "97\n",
            "98\n",
            "99\n",
            "context_input: 100\n",
            "dict_context_to_label_inputs: 83\n",
            "label_single-label[0]: [-1]\n",
            "label_multi-label[0]: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0.])\n",
            "label_multi_hot_is_complete: tensor([False, False, False,  True, False, False, False, False, False, False,\n",
            "        False, False, False,  True, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False,  True, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False,  True, False,\n",
            "        False, False, False, False, False,  True, False, False, False, False,\n",
            "        False, False, False])\n",
            "label_input[0]-multi-label: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0.])\n",
            "evaluate crossencoder on all candidate labels\n",
            "Evaluation:   0% 0/83 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
            "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
            "Evaluation: 100% 83/83 [00:40<00:00,  2.07it/s]\n",
            "num_labels: 50\n",
            "/content/LM-ontology-concept-placement/blink/candidate_ranking/multi_label_evaluation.py:233: RuntimeWarning: invalid value encountered in scalar divide\n",
            "  return intersect_size(yhatmic, ymic, 0) / yhatmic.sum(axis=0)\n",
            "/content/LM-ontology-concept-placement/blink/candidate_ranking/multi_label_evaluation.py:203: RuntimeWarning: invalid value encountered in divide\n",
            "  vals.append(num_true_in_top_k / float(denom))\n",
            "num in insertion_all: 0 num_ave: 0.0\n",
            "num in insertion_any: 0 num_ave: 0.0\n",
            "\n",
            "[MACRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.7756\n",
            "[MICRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, nan, 0.0000, nan, 0.7689\n",
            "rec_at_1: 0.0120\n",
            "prec_at_1: 0.0120\n",
            "rec_at_3: 0.0120\n",
            "prec_at_3: 0.0040\n",
            "rec_at_5: 0.0241\n",
            "prec_at_5: 0.0048\n",
            "rec_at_10: 0.0361\n",
            "prec_at_10: 0.0036\n",
            "rec_at_50: 0.0602\n",
            "prec_at_50: 0.0012\n",
            "ins_at_1_all: 0.0120\n",
            "ins_at_1_any: 0.0120\n",
            "ins_at_3_all: 0.0120\n",
            "ins_at_3_any: 0.0120\n",
            "ins_at_5_all: 0.0241\n",
            "ins_at_5_any: 0.0241\n",
            "ins_at_10_all: 0.0361\n",
            "ins_at_10_any: 0.0361\n",
            "ins_at_50_all: 0.0602\n",
            "ins_at_50_any: 0.0602\n",
            "ins_all: 0.0000\n",
            "ins_any: 0.0000\n",
            "\n",
            "labels_aggregated: [[206340], [206851], [205807, 205771, 206683], [159415], [206800], [205129], [206031], [204973], [205171], [205374, 205864], [206849, 206029], [204941], [205365], [105384], [216602], [206316], [206324, 205245], [205337], [205515], [206416], [206449], [205512], [205745], [217131], [205836], [205698, 205797], [151948], [205426], [205204, 206290], [205130], [213960], [206622, 205629], [18438], [205567, 205188], [205196, 206616], [205347], [206791], [206352], [216671], [151992], [205615], [206344], [206035, 206104, 206637], [205343], [204997, 205581], [206565, 205031], [205572], [59636], [205197], [152696], [206171], [204818], [205245], [205315], [206288], [205815], [205752], [205419], [204993], [206415], [206184], [205730, 205354], [216404], [206366], [18445], [206057], [204834, 204828], [216379], [82299], [151957], [205295], [206624], [30023], [206221], [220191], [66596], [206046, 206348], [205977], [206438], [33296], [205543], [169275], [205551]]\n",
            "predictions[0]: [('something that is Role group (attribute) some something that is Associated with (attribute) some Phytophotodermatitis (disorder)', 'NULL'), ('infectious disease due to plant (disorder)', 'NULL'), ('TOP', 'something that is Role group (attribute) some something that is Associated with (attribute) some Phytophotodermatitis (disorder)'), ('phytophotodermatitis (disorder)', 'acute phytophotodermatitis (disorder)'), ('phytophotodermatitis (disorder)', 'NULL'), ('toxic effect from eating berries and/or other plants (disorder)', 'family oxalidaceae poisoning (disorder)'), ('phytophotodermatitis (disorder)', 'chronic phytophotodermatitis (disorder)'), ('acute phytophotodermatitis (disorder)', 'NULL'), ('disease (disorder)', 'infectious disease due to plant (disorder)'), ('plants - toxic effect (disorder)', 'berries - toxic effect (disorder)'), ('phytophotodermatitis (disorder)', 'phytophotodermatitis due to topical exposure to furocoumarin (disorder)'), ('disease (disorder)', 'gardening product causing toxic effect (disorder)'), ('disease (disorder)', 'rust remover causing toxic effect (disorder)'), ('plants - toxic effect (disorder)', 'toxic effect from eating berries and/or other plants (disorder)'), ('self poisoning by plants or parts of plants (disorder)', 'NULL'), ('disease (disorder)', 'plants - toxic effect (disorder)'), ('disease (disorder)', 'agrochemical or pesticide causing toxic effect (disorder)'), ('phytophotodermatitis due to topical exposure to furocoumarin (disorder)', 'NULL'), ('bacterial infectious disease (disorder)', 'cutaneous botryomycosis (disorder)'), ('disease due to gram-positive bacteria (disorder)', 'staphylococcal botryomycosis (disorder)'), ('bacterial infectious disease (disorder)', 'botryomycosis (disorder)'), ('wheat weevil disease (disorder)', 'NULL'), ('plants - toxic effect (disorder)', 'family oxalidaceae poisoning (disorder)'), ('agrostemma species poisoning (disorder)', 'NULL'), ('chronic phytophotodermatitis (disorder)', 'NULL'), ('plants - toxic effect (disorder)', 'NULL'), ('toxic effect of noxious substance eaten as food (disorder)', 'family oxalidaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family phytolaccaceae poisoning (disorder)'), ('bacterial infectious disease (disorder)', 'staphylococcal botryomycosis (disorder)'), ('plants - toxic effect (disorder)', 'family pinaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family scrophulariaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family meliaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'greasewood poisoning (disorder)'), ('disease (disorder)', 'botryomycosis (disorder)'), ('phototoxic dermatitis (disorder)', 'acute phytophotodermatitis (disorder)'), ('plants - toxic effect (disorder)', 'family lamiaceae poisoning (disorder)'), ('bacterial infectious disease (disorder)', 'pseudomonal botryomycosis (disorder)'), ('plants - toxic effect (disorder)', 'family iridaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family malvaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family crassulaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family primulaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family araliaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family combretaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family equisetaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family taxaceae poisoning (disorder)'), ('toxic effect from eating berries and/or other plants (disorder)', 'NULL'), ('poisoning (disorder)', 'toxic effect from eating berries and/or other plants (disorder)'), ('phototoxic dermatitis (disorder)', 'chronic phytophotodermatitis (disorder)'), ('plants - toxic effect (disorder)', 'family buxaceae poisoning (disorder)'), ('plants - toxic effect (disorder)', 'family loganiaceae poisoning (disorder)')]\n",
            "pred_idx_tuples[0]: [('[EX.](<http://snomed.info/id/609096000> [EX.](<http://snomed.info/id/47429007> <http://snomed.info/id/238521005>))', 'SCTID_NULL'), ('105715005', 'SCTID_NULL'), ('SCTID_THING', '[EX.](<http://snomed.info/id/609096000> [EX.](<http://snomed.info/id/47429007> <http://snomed.info/id/238521005>))'), ('238521005', '58306008'), ('238521005', 'SCTID_NULL'), ('36465006', '123932008'), ('238521005', '58419006'), ('58306008', 'SCTID_NULL'), ('64572001', '105715005'), ('275390005', '275389001'), ('238521005', '95349002'), ('64572001', '241897004'), ('64572001', '241893000'), ('275390005', '36465006'), ('242838009', 'SCTID_NULL'), ('64572001', '275390005'), ('64572001', '241783002'), ('95349002', 'SCTID_NULL'), ('87628006', '238415008'), ('371582002', '238414007'), ('87628006', '238413001'), ('38729007', 'SCTID_NULL'), ('275390005', '123932008'), ('60225005', 'SCTID_NULL'), ('58419006', 'SCTID_NULL'), ('275390005', 'SCTID_NULL'), ('86068002', '123932008'), ('275390005', '123933003'), ('87628006', '238414007'), ('275390005', '123934009'), ('275390005', '123905003'), ('275390005', '123927007'), ('275390005', '359680004'), ('64572001', '238413001'), ('53597009', '58306008'), ('275390005', '123926003'), ('87628006', '402936008'), ('275390005', '123917001'), ('275390005', '123915009'), ('275390005', '123899003'), ('275390005', '123886001'), ('275390005', '123916005'), ('275390005', '123938007'), ('275390005', '123907006'), ('275390005', '123937002'), ('36465006', 'SCTID_NULL'), ('75478009', '36465006'), ('53597009', '58419006'), ('275390005', '123895009'), ('275390005', '123914008')]\n",
            "scores[0]: [0.03384427, 0.0298103, 0.034475707, 0.041690204, 0.030237012, 0.061222207, 0.039181605, 0.030340446, 0.034083866, 0.037464947, 0.03805389, 0.03335334, 0.037879497, 0.041928645, 0.029814776, 0.031372603, 0.036620125, 0.03211992, 0.041467417, 0.07536459, 0.043730285, 0.03213898, 0.042205077, 0.03565743, 0.02992385, 0.028894447, 0.040680435, 0.033110797, 0.04300612, 0.03313205, 0.033487733, 0.03293949, 0.03203769, 0.038924333, 0.034898974, 0.032522906, 0.055148754, 0.033478122, 0.036899194, 0.034189858, 0.033741754, 0.032977745, 0.03330519, 0.03458554, 0.034786094, 0.03190654, 0.037848383, 0.03455329, 0.033835515, 0.032370508]\n",
            "gold_idx_tuples[0]: [('404684003', '264491001')]\n",
            "label_input/samples-aggregated:83/83\n",
            "INFO:Blink:save predictions to models/crossencoder/mm+Disease-2017AA/original-NIL-top50-pubmedbert\n",
            "INFO:Blink:mm-valid-NIL\n",
            "INFO:Blink:test dataset mode\n",
            "INFO:Blink:672/672 samples considered\n",
            "keep_all: False\n",
            "INFO:Blink:preparing data for biencoder\n",
            "args.use_NIL_tag: False\n",
            "args.use_NIL_desc: False\n",
            "args.use_NIL_desc_tag: False\n",
            "num of samples: 672\n",
            "label_text_vecs: tensor([[    2,  4770,  1927,  ...,     0,     0,     0],\n",
            "        [    2,  4604,  6159,  ...,     0,     0,     0],\n",
            "        [    2,  4604,  6159,  ...,     0,     0,     0],\n",
            "        ...,\n",
            "        [    2, 13027,  6379,  ...,     0,     0,     0],\n",
            "        [    2,  4770,  1927,  ...,     0,     0,     0],\n",
            "        [    2,  6558,  2798,  ...,     0,     0,     0]])\n",
            "100% 237826/237826 [00:02<00:00, 86133.06it/s] \n",
            "INFO:Blink:run biencoder\n",
            "100% 84/84 [00:17<00:00,  4.92it/s]\n",
            "labels: 672 1\n",
            "nns: 672 50\n",
            "all_scores_sorted: 672 50\n",
            "all_scores: 672 237826\n",
            "INFO:Blink:Enriching edge candidates\n",
            "INFO:Blink:args.top_k_cand_seed: 25\n",
            "100% 672/672 [00:00<00:00, 1126.55it/s]\n",
            "nns: 672 50\n",
            "labels: 672 [36106] <class 'numpy.ndarray'>\n",
            "nns: 672 [ 23866 167207  92773  66908 165232 165228 165229  92771 168213  84463\n",
            " 106706 165224  49848 165233 187881  30144 149003 159849 212961  23865\n",
            "  83763  84523 110845  36106  25846 234675 109450   7136 165225  49628\n",
            "  49681 234684  66198 165230  84456 165231   2241 157149  23598 109461\n",
            "  49310  27504  84464  49161    291  18643  25917    248  41321 167276]\n",
            "missed case (ind 1, within first 5): ('236423003', '431856006') not in [('106000008', '236455003'), ('90708001', '236519008'), ('90708001', '236430009'), ('90708001', '42927005'), ('90708001', '104931000119100'), ('90708001', '236514003'), ('90708001', '428720002'), ('236425005', '104931000119100'), ('90708001', '76224000'), ('90708001', '26367008'), ('64572001', '9918001'), ('236425005', '129151000119102'), ('90708001', '95572004'), ('236423003', '104931000119100'), ('90708001', '236495001'), ('90708001', '50581000'), ('90708001', '13886001'), ('106000008', '190829000'), ('236425005', '700379002'), ('90708001', '186235002'), ('90708001', '93425004'), ('90708001', '370494002'), ('236425005', '129171000119106'), ('90708001', '307532008'), ('90708001', '236515002'), ('64572001', '61852001'), ('236425005', '129181000119109'), ('236519008', '236520002'), ('90708001', '236425005'), ('90708001', '197752005'), ('236425005', '700378005'), ('90708001', '236526008'), ('236425005', 'SCTID_NULL'), ('90708001', '197753000'), ('90708001', '236523000'), ('90708001', '236428007'), ('90708001', '32093003'), ('236423003', 'SCTID_NULL'), ('106000008', 'SCTID_NULL'), ('90708001', 'SCTID_NULL'), ('90708001', '23697004'), ('236425005', '118781000119108'), ('90708001', '123753008'), ('42030000', '370494002'), ('42030000', '236514003'), ('42030000', '93425004'), ('75934005', 'SCTID_NULL'), ('64572001', 'SCTID_NULL'), ('42030000', 'SCTID_NULL'), ('128121009', 'SCTID_NULL')]\n",
            "missed case (ind 2, within first 5): ('236423003', '431855005') not in [('90708001', '236519008'), ('236519008', 'SCTID_NULL'), ('64572001', '9918001'), ('307532008', 'SCTID_NULL'), ('90708001', '307532008'), ('90708001', '236526008'), ('90708001', '50581000'), ('90708001', '104931000119100'), ('[EX.](<609096000> [EX.](<47429007> <90708001>))', '234348004'), ('236425005', '104931000119100'), ('[EX.](<609096000> [EX.](<42752001> <236423003>))', 'SCTID_NULL'), ('[EX.](<http://snomed.info/id/609096000> [EX.](<http://snomed.info/id/42752001> <http://snomed.info/id/236423003>))', 'SCTID_NULL'), ('236423003', '104931000119100'), ('90708001', '79385002'), ('236425005', '129151000119102'), ('64572001', '61852001'), ('236425005', '129171000119106'), ('236425005', '117681000119102'), ('[EX.](<609096000> [EX.](<47429007> <236423003>))', 'SCTID_NULL'), ('[EX.](<http://snomed.info/id/609096000> [EX.](<http://snomed.info/id/47429007> <http://snomed.info/id/236423003>))', 'SCTID_NULL'), ('236425005', 'SCTID_NULL'), ('[EX.](<609096000> [EX.](<47429007> <90708001>))', '14973001'), ('90708001', '106000008'), ('106000008', '236455003'), ('90708001', '197752005'), ('[EX.](<609096000> [EX.](<47429007> <90708001>))', '310647000'), ('[EX.](<609096000> [EX.](<47429007> <90708001>))', 'SCTID_NULL'), ('90708001', '42927005'), ('236423003', 'SCTID_NULL'), ('90708001', 'SCTID_NULL'), ('90708001', '236495001'), ('90708001', '290006'), ('106000008', 'SCTID_NULL'), ('236425005', '118781000119108'), ('90708001', '54879000'), ('90708001', '109477002'), ('90708001', '67132008'), ('90708001', '399190000'), ('90708001', '33763006'), ('90708001', '444645005'), ('236514003', 'SCTID_NULL'), ('90708001', '399094007'), ('90708001', '236535001'), ('90708001', '236713006'), ('90708001', '236534002'), ('90708001', '236532003'), ('90708001', '236531005'), ('90708001', '236528009'), ('90708001', '85487008'), ('42030000', '197752005')]\n",
            "biencoder accuracy: 0.0045\n",
            "biencoder recall@50: 0.3497\n",
            "  0% 0/672 [00:00<?, ?it/s]context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'inflammatory', 'skin', 'disease', '[unused1]'], 'mention_ids': [1, 3769, 4407, 2573, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'pus', '##t', '##ulosis', '(', 'ppp', ')', 'is', 'a', 'recalc', '##itr', '##ant', 'chronic', '[unused0]', 'inflammatory', 'skin', 'disease', '[unused1]', '.', 'data', 'relevant', 'for', 'the', 'medical', 'care', 'of', 'patients', 'with', 'ppp', 'are', 'scarce', '[SEP]'], 'ids': [2, 24093, 1031, 8071, 12, 22150, 13, 1977, 43, 25470, 4250, 2031, 3724, 1, 3769, 4407, 2573, 1, 18, 2230, 4494, 1958, 1920, 3423, 2859, 1927, 2132, 1956, 22150, 2032, 16937, 3], 'contextual_tokens': ['[CLS]', 'pus', '##t', '##ulosis', '(', 'ppp', ')', 'is', 'a', 'recalc', '##itr', '##ant', 'chronic', '[unused0]', 'inflammatory', 'skin', 'disease', '[unused1]', '.', 'data', 'relevant', 'for', 'the', 'medical', 'care', 'of', 'patients', 'with', 'ppp', 'are', 'scarce', '[SEP]'], 'contextual_ids': [2, 24093, 1031, 8071, 12, 22150, 13, 1977, 43, 25470, 4250, 2031, 3724, 1, 3769, 4407, 2573, 1, 18, 2230, 4494, 1958, 1920, 3423, 2859, 1927, 2132, 1956, 22150, 2032, 16937, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'ckd', '[unused1]'], 'mention_ids': [1, 9928, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'catheter', '-', 'associated', 'inferior', 'vena', 'cava', 'thrombosis', 'patients', 'with', 'chronic', 'kidney', 'disease', '(', '[unused0]', 'ckd', '[unused1]', ')', 'are', 'at', 'increased', 'risk', 'for', 'thrombotic', 'complications', '.', 'the', 'use', 'of', 'central', 'venous', '[SEP]'], 'ids': [2, 8421, 17, 2458, 8756, 22876, 23326, 11354, 2132, 1956, 3724, 5245, 2573, 12, 1, 9928, 1, 13, 2032, 2019, 2502, 2565, 1958, 21039, 5021, 18, 1920, 2485, 1927, 4101, 8562, 3], 'contextual_tokens': ['[CLS]', 'catheter', '-', 'associated', 'inferior', 'vena', 'cava', 'thrombosis', 'patients', 'with', 'chronic', 'kidney', 'disease', '(', '[unused0]', 'ckd', '[unused1]', ')', 'are', 'at', 'increased', 'risk', 'for', 'thrombotic', 'complications', '.', 'the', 'use', 'of', 'central', 'venous', '[SEP]'], 'contextual_ids': [2, 8421, 17, 2458, 8756, 22876, 23326, 11354, 2132, 1956, 3724, 5245, 2573, 12, 1, 9928, 1, 13, 2032, 2019, 2502, 2565, 1958, 21039, 5021, 18, 1920, 2485, 1927, 4101, 8562, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'chronic', 'kidney', 'disease', '[unused1]'], 'mention_ids': [1, 3724, 5245, 2573, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'syndrome', 'secondary', 'to', 'catheter', '-', 'associated', 'inferior', 'vena', 'cava', 'thrombosis', 'patients', 'with', '[unused0]', 'chronic', 'kidney', 'disease', '[unused1]', '(', 'ckd', ')', 'are', 'at', 'increased', 'risk', 'for', 'thrombotic', 'complications', '.', 'the', 'use', '[SEP]'], 'ids': [2, 4741, 4067, 1942, 8421, 17, 2458, 8756, 22876, 23326, 11354, 2132, 1956, 1, 3724, 5245, 2573, 1, 12, 9928, 13, 2032, 2019, 2502, 2565, 1958, 21039, 5021, 18, 1920, 2485, 3], 'contextual_tokens': ['[CLS]', 'syndrome', 'secondary', 'to', 'catheter', '-', 'associated', 'inferior', 'vena', 'cava', 'thrombosis', 'patients', 'with', '[unused0]', 'chronic', 'kidney', 'disease', '[unused1]', '(', 'ckd', ')', 'are', 'at', 'increased', 'risk', 'for', 'thrombotic', 'complications', '.', 'the', 'use', '[SEP]'], 'contextual_ids': [2, 4741, 4067, 1942, 8421, 17, 2458, 8756, 22876, 23326, 11354, 2132, 1956, 1, 3724, 5245, 2573, 1, 12, 9928, 13, 2032, 2019, 2502, 2565, 1958, 21039, 5021, 18, 1920, 2485, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'lynch', 'syndrome', '[unused1]'], 'mention_ids': [1, 27452, 4741, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'genetic', 'background', '.', 'here', ',', 'we', 'describe', 'the', 'case', 'of', 'a', 'patient', 'with', '[unused0]', 'lynch', 'syndrome', '[unused1]', 'that', 'presented', 'with', 'six', 'synchronous', 'crc', '##s', '.', 'microsatellite', 'instability', '(', 'msi', ')', '[SEP]'], 'ids': [2, 3299, 3872, 18, 3410, 16, 2038, 5993, 1920, 3087, 1927, 43, 2774, 1956, 1, 27452, 4741, 1, 1988, 3549, 1956, 3798, 17669, 8223, 1036, 18, 16069, 9557, 12, 17355, 13, 3], 'contextual_tokens': ['[CLS]', 'genetic', 'background', '.', 'here', ',', 'we', 'describe', 'the', 'case', 'of', 'a', 'patient', 'with', '[unused0]', 'lynch', 'syndrome', '[unused1]', 'that', 'presented', 'with', 'six', 'synchronous', 'crc', '##s', '.', 'microsatellite', 'instability', '(', 'msi', ')', '[SEP]'], 'contextual_ids': [2, 3299, 3872, 18, 3410, 16, 2038, 5993, 1920, 3087, 1927, 43, 2774, 1956, 1, 27452, 4741, 1, 1988, 3549, 1956, 3798, 17669, 8223, 1036, 18, 16069, 9557, 12, 17355, 13, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'vt', '[unused1]'], 'mention_ids': [1, 12195, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'year', 'after', 'crt', 'had', 'a', 'higher', 'rate', 'of', 'lv', 'reverse', 'remodeling', 'and', 'less', '[unused0]', 'vt', '[unused1]', '/', 'vf', 'episodes', '.', 'the', 'acute', 'changes', 'of', 'tp', '##te', 'after', 'crt', 'have', 'minimal', '[SEP]'], 'ids': [2, 2476, 2256, 13705, 2430, 43, 2580, 2769, 1927, 6701, 5267, 8378, 1930, 2941, 1, 12195, 1, 19, 15691, 9232, 18, 1920, 3939, 2840, 1927, 6213, 2057, 2256, 13705, 2162, 6343, 3], 'contextual_tokens': ['[CLS]', 'year', 'after', 'crt', 'had', 'a', 'higher', 'rate', 'of', 'lv', 'reverse', 'remodeling', 'and', 'less', '[unused0]', 'vt', '[unused1]', '/', 'vf', 'episodes', '.', 'the', 'acute', 'changes', 'of', 'tp', '##te', 'after', 'crt', 'have', 'minimal', '[SEP]'], 'contextual_ids': [2, 2476, 2256, 13705, 2430, 43, 2580, 2769, 1927, 6701, 5267, 8378, 1930, 2941, 1, 12195, 1, 19, 15691, 9232, 18, 1920, 3939, 2840, 1927, 6213, 2057, 2256, 13705, 2162, 6343, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'wol', '##man', '[unused1]'], 'mention_ids': [1, 12697, 3862, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', '##ho', '##ff', ',', 'gm1', ',', 'muc', '##opl', '##ys', '##acchar', '##ido', '##se', '##s', ',', '[unused0]', 'wol', '##man', '[unused1]', ',', 'kra', '##bb', '##e', ',', 'metac', '##hr', '##omatic', 'leuk', '##odys', '##tr', '##oph', '##y', '[SEP]'], 'ids': [2, 10647, 2040, 16, 28673, 16, 3422, 3513, 2012, 7364, 15072, 4127, 1036, 16, 1, 12697, 3862, 1, 16, 29995, 6936, 1021, 16, 17265, 6570, 21832, 5104, 26812, 1997, 2688, 1012, 3], 'contextual_tokens': ['[CLS]', '##ho', '##ff', ',', 'gm1', ',', 'muc', '##opl', '##ys', '##acchar', '##ido', '##se', '##s', ',', '[unused0]', 'wol', '##man', '[unused1]', ',', 'kra', '##bb', '##e', ',', 'metac', '##hr', '##omatic', 'leuk', '##odys', '##tr', '##oph', '##y', '[SEP]'], 'contextual_ids': [2, 10647, 2040, 16, 28673, 16, 3422, 3513, 2012, 7364, 15072, 4127, 1036, 16, 1, 12697, 3862, 1, 16, 29995, 6936, 1021, 16, 17265, 6570, 21832, 5104, 26812, 1997, 2688, 1012, 3]}\n",
            "100% 672/672 [00:00<00:00, 921.31it/s]\n",
            "topk in prepare_crossencoder_candidates: 50\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'pap', '##ul', '##ono', '##du', '##la', '##r', 'disease', 'of', 'skin', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 3860, 1944, 23557, 10015, 5148, 1005, 2573, 1927, 4407, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'arte', '##fact', '##ual', 'skin', 'disease', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 20897, 4210, 2224, 4407, 2573, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'disorder', 'of', 'skin', '(', 'disorder', ')', '[unused5]', 'subc', '##orn', '##eal', 'pus', '##t', '##ular', 'dermat', '##osis', 'with', 'parap', '##rotein', '##emia', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 4770, 1927, 4407, 12, 4770, 13, 1, 5736, 5578, 5243, 24093, 1031, 2168, 13467, 2475, 1956, 22224, 5684, 3847, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'metabolic', 'renal', 'disease', '(', 'disorder', ')', '[unused5]', 'satur', '##ni', '##ne', 'nephropathy', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 4487, 4604, 2573, 12, 4770, 13, 1, 20924, 8635, 2912, 15582, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'kidney', 'disease', '(', 'disorder', ')', '[unused5]', 'chronic', 'drug', '-', 'induced', 'renal', 'disease', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 5245, 2573, 12, 4770, 13, 1, 3724, 2838, 17, 2719, 4604, 2573, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'kidney', 'disease', '(', 'disorder', ')', '[unused5]', 'pigment', 'nephropathy', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 5245, 2573, 12, 4770, 13, 1, 12905, 15582, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'kidney', 'disease', '(', 'disorder', ')', '[unused5]', 'chronic', 'drug', '-', 'induced', 'renal', 'disease', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 5245, 2573, 12, 4770, 13, 1, 3724, 2838, 17, 2719, 4604, 2573, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'chronic', 'drug', '-', 'induced', 'renal', 'disease', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 3724, 2838, 17, 2719, 4604, 2573, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'disease', '(', 'disorder', ')', '[unused5]', 'mobile', 'kidney', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 2573, 12, 4770, 13, 1, 8785, 5245, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "label_input_list: [23 -1 -1 28  9 -1 -1 -1 -1 -1 -1 28 25 11 -1 -1 23 -1 -1 -1 -1 48 30 -1\n",
            " -1 18 -1 -1  0 -1 -1 11 -1 -1 48 32 -1 -1 49 -1 -1 -1 26 -1 -1 31 46 26\n",
            " -1 -1 -1 -1 49 -1 -1 43 -1 42 49 -1 -1 -1 -1 -1 49 -1  5 -1 46 -1 49 -1\n",
            " -1 43 -1 -1 -1 49 34 -1 -1 -1 -1 47 -1 48 47 -1 -1 45  9 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 35 10 -1 27 -1 -1 -1 44 47 33 -1 -1 24 -1 49 -1 -1 13  4 15\n",
            " -1 49 -1 -1 -1 47 -1 -1 40 33 49 36 -1 -1 30 43 -1 -1 49 46 48  3 -1 27\n",
            " 31 10 29  3 -1 -1 -1 -1 -1 -1 48 -1 -1 -1 -1 -1  1 13 26 -1  2 -1 47 -1\n",
            " 14 -1 42 -1 15 11 -1 -1 -1 49 29 -1 -1 48 49 -1 -1  4  2 -1 -1 -1 -1 -1\n",
            " 38 -1 45 -1 43 -1 49 -1 -1 -1 -1 49 -1 -1 23 -1  7 26 45 -1 49 -1 -1 -1\n",
            " -1 -1 49 -1  4 18 -1  4 -1 -1 -1 -1 -1 24 -1 -1 -1 -1 27 -1 49 44 -1 -1\n",
            " -1 28 -1 35 -1 -1 14 37 -1 -1 -1  0 37 34 -1 -1 -1 -1 -1 -1 -1 47 -1 -1\n",
            " -1 25 -1  4 -1 -1 -1 -1 -1  2 -1 -1 47 -1 -1 49 -1 -1 40 49 14 -1 -1 13\n",
            "  8 -1 24 -1 -1 -1 -1  7 14 -1 -1 47 -1 -1 -1 -1 -1 34 35 -1 -1 49 -1 -1\n",
            " -1 -1 30 33 -1 -1 49 -1 44 -1 47 42 -1 -1 -1 19 -1 -1 -1 -1 25 -1 28 -1\n",
            " -1  3 -1 -1 -1 -1  0 -1 34 -1 -1 -1 -1 29 -1 -1 -1 -1 -1 -1 -1 -1 -1 49\n",
            " 43 -1 -1 -1 -1 -1 48 44 -1 -1 -1 43 10 -1  8 -1 -1 -1 48 -1 -1  2 -1 49\n",
            " -1 21 -1 -1 -1 -1 -1 -1 39 45 26 27  8 -1 40 45  7 -1 -1 -1 -1 -1 -1 48\n",
            " -1 49 47 -1 38 -1 -1 -1 -1 -1 -1 30 -1 -1 -1 45 47 -1 -1 49 -1 -1 -1 -1\n",
            " -1 -1 -1 27 -1 48 48 -1 31 -1 48 20 -1 -1 -1 -1 -1 45  6 -1 -1 -1 39 -1\n",
            " -1  3 -1 -1 -1 -1 -1 -1 27 49 49 -1 -1 -1 -1 -1  7 -1 -1 -1 -1 -1 27 -1\n",
            " -1 30 46 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 13 -1 -1 -1 49 -1 22 -1\n",
            " 14 -1 -1 -1 -1 -1 46 46 49 -1 41 -1 -1  3 28 -1 -1 12 -1 -1 -1 48 -1 -1\n",
            " -1 46 -1 -1 -1 47 -1 -1 -1 29 -1 -1 -1 -1 -1 44 -1 -1 -1 -1 -1 -1 49 -1\n",
            " -1 -1 22 -1 -1 25 -1 -1 -1 -1 -1 32 -1 40  4 -1 -1 -1 -1 32 -1 -1 -1  6\n",
            " -1 36 -1 -1 -1 -1 -1 -1 -1 35 -1 26 -1 -1 -1 -1 36 -1 -1 26  7 -1 45 -1\n",
            " 44 -1 10 -1 -1 -1 -1 -1 -1 -1 24 -1 49 -1 -1 -1 49 -1 -1 -1 -1 44 45 -1\n",
            " 14 34 28 -1 -1 -1 -1 -1 -1 33 -1 34 -1 -1  3  6 -1 33 -1 -1 -1 44 -1 -1\n",
            "  2 48 -1 49 -1 -1 47 -1 -1 -1 29 -1 -1 30 -1 -1 -1 -1 -1 27 -1 46 -1 26]\n",
            "label_input: tensor([23, -1, -1, 28,  9, -1, -1, -1, -1, -1, -1, 28, 25, 11, -1, -1, 23, -1,\n",
            "        -1, -1, -1, 48, 30, -1, -1, 18, -1, -1,  0, -1, -1, 11, -1, -1, 48, 32,\n",
            "        -1, -1, 49, -1, -1, -1, 26, -1, -1, 31, 46, 26, -1, -1, -1, -1, 49, -1,\n",
            "        -1, 43, -1, 42, 49, -1, -1, -1, -1, -1, 49, -1,  5, -1, 46, -1, 49, -1,\n",
            "        -1, 43, -1, -1, -1, 49, 34, -1, -1, -1, -1, 47, -1, 48, 47, -1, -1, 45,\n",
            "         9, -1, -1, -1, -1, -1, -1, -1, -1, -1, 35, 10, -1, 27, -1, -1, -1, 44,\n",
            "        47, 33, -1, -1, 24, -1, 49, -1, -1, 13,  4, 15, -1, 49, -1, -1, -1, 47,\n",
            "        -1, -1, 40, 33, 49, 36, -1, -1, 30, 43, -1, -1, 49, 46, 48,  3, -1, 27,\n",
            "        31, 10, 29,  3, -1, -1, -1, -1, -1, -1, 48, -1, -1, -1, -1, -1,  1, 13,\n",
            "        26, -1,  2, -1, 47, -1, 14, -1, 42, -1, 15, 11, -1, -1, -1, 49, 29, -1,\n",
            "        -1, 48, 49, -1, -1,  4,  2, -1, -1, -1, -1, -1, 38, -1, 45, -1, 43, -1,\n",
            "        49, -1, -1, -1, -1, 49, -1, -1, 23, -1,  7, 26, 45, -1, 49, -1, -1, -1,\n",
            "        -1, -1, 49, -1,  4, 18, -1,  4, -1, -1, -1, -1, -1, 24, -1, -1, -1, -1,\n",
            "        27, -1, 49, 44, -1, -1, -1, 28, -1, 35, -1, -1, 14, 37, -1, -1, -1,  0,\n",
            "        37, 34, -1, -1, -1, -1, -1, -1, -1, 47, -1, -1, -1, 25, -1,  4, -1, -1,\n",
            "        -1, -1, -1,  2, -1, -1, 47, -1, -1, 49, -1, -1, 40, 49, 14, -1, -1, 13,\n",
            "         8, -1, 24, -1, -1, -1, -1,  7, 14, -1, -1, 47, -1, -1, -1, -1, -1, 34,\n",
            "        35, -1, -1, 49, -1, -1, -1, -1, 30, 33, -1, -1, 49, -1, 44, -1, 47, 42,\n",
            "        -1, -1, -1, 19, -1, -1, -1, -1, 25, -1, 28, -1, -1,  3, -1, -1, -1, -1,\n",
            "         0, -1, 34, -1, -1, -1, -1, 29, -1, -1, -1, -1, -1, -1, -1, -1, -1, 49,\n",
            "        43, -1, -1, -1, -1, -1, 48, 44, -1, -1, -1, 43, 10, -1,  8, -1, -1, -1,\n",
            "        48, -1, -1,  2, -1, 49, -1, 21, -1, -1, -1, -1, -1, -1, 39, 45, 26, 27,\n",
            "         8, -1, 40, 45,  7, -1, -1, -1, -1, -1, -1, 48, -1, 49, 47, -1, 38, -1,\n",
            "        -1, -1, -1, -1, -1, 30, -1, -1, -1, 45, 47, -1, -1, 49, -1, -1, -1, -1,\n",
            "        -1, -1, -1, 27, -1, 48, 48, -1, 31, -1, 48, 20, -1, -1, -1, -1, -1, 45,\n",
            "         6, -1, -1, -1, 39, -1, -1,  3, -1, -1, -1, -1, -1, -1, 27, 49, 49, -1,\n",
            "        -1, -1, -1, -1,  7, -1, -1, -1, -1, -1, 27, -1, -1, 30, 46, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 13, -1, -1, -1, 49, -1, 22, -1,\n",
            "        14, -1, -1, -1, -1, -1, 46, 46, 49, -1, 41, -1, -1,  3, 28, -1, -1, 12,\n",
            "        -1, -1, -1, 48, -1, -1, -1, 46, -1, -1, -1, 47, -1, -1, -1, 29, -1, -1,\n",
            "        -1, -1, -1, 44, -1, -1, -1, -1, -1, -1, 49, -1, -1, -1, 22, -1, -1, 25,\n",
            "        -1, -1, -1, -1, -1, 32, -1, 40,  4, -1, -1, -1, -1, 32, -1, -1, -1,  6,\n",
            "        -1, 36, -1, -1, -1, -1, -1, -1, -1, 35, -1, 26, -1, -1, -1, -1, 36, -1,\n",
            "        -1, 26,  7, -1, 45, -1, 44, -1, 10, -1, -1, -1, -1, -1, -1, -1, 24, -1,\n",
            "        49, -1, -1, -1, 49, -1, -1, -1, -1, 44, 45, -1, 14, 34, 28, -1, -1, -1,\n",
            "        -1, -1, -1, 33, -1, 34, -1, -1,  3,  6, -1, 33, -1, -1, -1, 44, -1, -1,\n",
            "         2, 48, -1, 49, -1, -1, 47, -1, -1, -1, 29, -1, -1, 30, -1, -1, -1, -1,\n",
            "        -1, 27, -1, 46, -1, 26])\n",
            "tensor_is_NIL_labels: tensor([False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False])\n",
            "label_input[0]: tensor(23)\n",
            "samples[0]: {'context_left': 'palmoplantar pustulosis - a cross-sectional analysis in germany palmoplantar pustulosis (ppp) is a recalcitrant chronic ', 'mention': 'inflammatory skin disease', 'context_right': '. data relevant for the medical care of patients with ppp are scarce. thus, the aim of this work was to investigate the disease burden, clinical characteristics, and comorbidity of ppp patients in germany. ppp patients were examined in a crosssectional study at seven specialized psoriasis centers in germany. of the 172 included patients with ppp, 79.1% were female and 69.8% were smokers .in addition, 25.0% suffered from psoriasis vulgaris, 28.2% had documented psoriatic arthritis, and 30.2% had a family history of psoriasis. in 77 patients the mean dermatology life quality index (dlqi) was 12.2 ± 7.7 (mean ± sd). the mean psoriasis palmoplantar pustulosis area and severity index (pppasi) was 12.6 ± 8.6. mean body mass index was above average at 27.1 ± 5.5. the ppp patients', 'label_concept_UMLS': 'C3875321', 'label_concept': 36106, 'label_concept_ori': '703938007', 'entity_label_id': 64900, 'entity_label': '', 'entity_label_title': 'NIL', 'parent_concept': '95320005', 'child_concept': '277408007', 'parent': 'disorder of skin (disorder)', 'child': 'histologic type of inflammatory skin disorder (disorder)', 'edge_label_id': 36106, 'label_concept_str': ('95320005', '277408007'), 'row_ind_men': 0}\n",
            "max_seq_length: 160\n",
            "doc_ind_aggregated_strs:\n",
            " 0|17|23|29|41|72|76|80|95|102|148|153|159|190|195|201|280|307|319|333|336|350|351|353|357|358|361|364|389|397|406|414|416|460|470|475|476|508|520|526|530|556|559|604|606|629|631|655|657\n",
            "1|18|117|213|230|317|324|567|583|632|656\n",
            "2|88|126|310|331|521|581|584|591|619|643\n",
            "3|622\n",
            "4\n",
            "5|127|151|300|365|610\n",
            "6|15|50|293|352|451|483|499|549|553|609\n",
            "7|413\n",
            "8|53|71|81|91|111|115|136|149|150|163|171|176|180|200|205|231|258|281|303|304|335|345|356|380|388|422|434|447|461|495|501|503|538|541|547|555|562|568|605|611|615|640|650|652|664|665|668|670\n",
            "9\n",
            "10\n",
            "11|106|134|169|173|382|464|524|587|658|663\n",
            "12\n",
            "13\n",
            "14\n",
            "16|47|84|96|184|277|284|315|327|485|626\n",
            "19|48|321|441|446|456|492|498|536|564|628\n",
            "20\n",
            "21\n",
            "22|183|627\n",
            "24\n",
            "25|32|105|156|222|445|484|523|561|594|653\n",
            "26\n",
            "27|566\n",
            "28|33|540\n",
            "30\n",
            "31|112|571\n",
            "34\n",
            "35|502|535\n",
            "36|61|97|104|232|260|291|312|489|586|647\n",
            "37\n",
            "38\n",
            "39\n",
            "40|145\n",
            "42|412\n",
            "43\n",
            "44|418\n",
            "45|399\n",
            "46\n",
            "49\n",
            "51\n",
            "52\n",
            "54|197\n",
            "55\n",
            "56|98|209|228|245|262|263|311|411|459|532\n",
            "57|607\n",
            "58\n",
            "59\n",
            "60\n",
            "62\n",
            "63|381|633\n",
            "64\n",
            "65|217|440\n",
            "66\n",
            "67|372|601\n",
            "68\n",
            "69|113|432\n",
            "70\n",
            "73\n",
            "74|132|157|167|302|436|452|458|477|494|599\n",
            "75\n",
            "77\n",
            "78\n",
            "79|198\n",
            "82|179|219|235|254|285|308|433|527|531|539\n",
            "83\n",
            "85\n",
            "86\n",
            "87\n",
            "89\n",
            "90\n",
            "92|116\n",
            "93|242|256|266|462|467|488|515|519|603|620\n",
            "94\n",
            "99|575\n",
            "100\n",
            "101|162|253\n",
            "103|669\n",
            "107|265\n",
            "108\n",
            "109\n",
            "110\n",
            "114\n",
            "118\n",
            "119\n",
            "120|588\n",
            "121\n",
            "122|123|255\n",
            "124\n",
            "125\n",
            "128\n",
            "129|185|296\n",
            "130\n",
            "131\n",
            "133\n",
            "135\n",
            "137\n",
            "138\n",
            "139\n",
            "140\n",
            "141|369\n",
            "142\n",
            "143|511\n",
            "144|247|576\n",
            "146\n",
            "147|618\n",
            "152|329|330\n",
            "154\n",
            "155\n",
            "158\n",
            "160\n",
            "161\n",
            "164|246|419\n",
            "165\n",
            "166\n",
            "168\n",
            "170|359\n",
            "172\n",
            "174\n",
            "175\n",
            "177\n",
            "178\n",
            "181\n",
            "182\n",
            "186\n",
            "187|286|602\n",
            "188\n",
            "189\n",
            "191\n",
            "192\n",
            "193\n",
            "194\n",
            "196\n",
            "199\n",
            "202\n",
            "203|660\n",
            "204|298|400\n",
            "206\n",
            "207|617\n",
            "208|471\n",
            "210\n",
            "211\n",
            "212\n",
            "214\n",
            "215|238|401|403|417|493|522|534|548|613|648\n",
            "216\n",
            "218\n",
            "220\n",
            "221|241|290|395|509\n",
            "223\n",
            "224|486|546\n",
            "225\n",
            "226\n",
            "227|472|624\n",
            "229|480|551\n",
            "233\n",
            "234|394|481|563|585\n",
            "236|322|360|510|621\n",
            "237\n",
            "239|542\n",
            "240\n",
            "243\n",
            "244\n",
            "248\n",
            "249\n",
            "250|270|557\n",
            "251\n",
            "252\n",
            "257\n",
            "259|579|671\n",
            "261\n",
            "264|314|593\n",
            "267|292\n",
            "268|368|639\n",
            "269\n",
            "271\n",
            "272\n",
            "273\n",
            "274|363\n",
            "275|455\n",
            "276\n",
            "278\n",
            "279\n",
            "282|349|444\n",
            "283\n",
            "287\n",
            "288|346|393\n",
            "289\n",
            "294|474\n",
            "295\n",
            "297\n",
            "299\n",
            "301|421\n",
            "305|386\n",
            "306|666\n",
            "309\n",
            "313\n",
            "316\n",
            "318\n",
            "320\n",
            "323|343\n",
            "325\n",
            "326\n",
            "328|404\n",
            "332|554\n",
            "334|469\n",
            "337|505\n",
            "338|370\n",
            "339|402|457\n",
            "340\n",
            "341\n",
            "342\n",
            "344\n",
            "347\n",
            "348|385|544\n",
            "354\n",
            "355|487\n",
            "362|396\n",
            "366\n",
            "367\n",
            "371\n",
            "373\n",
            "374\n",
            "375\n",
            "376\n",
            "377\n",
            "378\n",
            "379|580\n",
            "383\n",
            "384|435\n",
            "387|565\n",
            "390\n",
            "391\n",
            "392\n",
            "398\n",
            "405\n",
            "407\n",
            "408\n",
            "409\n",
            "410\n",
            "415\n",
            "420\n",
            "423|635\n",
            "424\n",
            "425\n",
            "426\n",
            "427\n",
            "428\n",
            "429\n",
            "430\n",
            "431\n",
            "437\n",
            "438\n",
            "439\n",
            "442\n",
            "443\n",
            "448\n",
            "449\n",
            "450\n",
            "453|490|577\n",
            "454\n",
            "463\n",
            "465\n",
            "466\n",
            "468\n",
            "473\n",
            "478\n",
            "479\n",
            "482\n",
            "491\n",
            "496\n",
            "497\n",
            "500|625\n",
            "504|537\n",
            "506\n",
            "507\n",
            "512\n",
            "513\n",
            "514\n",
            "516\n",
            "517\n",
            "518\n",
            "525\n",
            "528\n",
            "529\n",
            "533\n",
            "543\n",
            "545\n",
            "550\n",
            "552\n",
            "558\n",
            "560\n",
            "569\n",
            "570\n",
            "572\n",
            "573\n",
            "574\n",
            "578\n",
            "582\n",
            "589\n",
            "590\n",
            "592\n",
            "595\n",
            "596\n",
            "597\n",
            "598\n",
            "600\n",
            "608\n",
            "612\n",
            "614\n",
            "616\n",
            "623\n",
            "630\n",
            "634\n",
            "636\n",
            "637\n",
            "638\n",
            "641\n",
            "642\n",
            "644\n",
            "645\n",
            "646\n",
            "649\n",
            "651\n",
            "654\n",
            "659\n",
            "661\n",
            "662\n",
            "667\n",
            "context_input: 672\n",
            "dict_context_to_label_inputs: 329\n",
            "label_single-label[0]: [23, -1]\n",
            "label_multi-label[0]: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n",
            "label_multi_hot_is_complete: tensor([False, False, False,  True,  True, False, False, False, False, False,\n",
            "        False, False,  True,  True, False, False, False, False,  True, False,\n",
            "        False, False, False, False, False, False,  True,  True, False, False,\n",
            "        False,  True, False, False,  True, False, False,  True,  True, False,\n",
            "        False,  True, False,  True, False, False,  True, False, False, False,\n",
            "        False,  True, False,  True, False,  True, False,  True,  True, False,\n",
            "        False,  True,  True, False, False,  True,  True,  True, False,  True,\n",
            "         True, False, False, False, False,  True,  True,  True,  True,  True,\n",
            "         True, False,  True,  True,  True, False,  True, False, False,  True,\n",
            "         True,  True,  True,  True, False,  True, False,  True,  True,  True,\n",
            "        False, False,  True, False,  True, False, False,  True, False, False,\n",
            "         True,  True,  True, False,  True,  True,  True,  True, False, False,\n",
            "         True,  True,  True,  True,  True, False, False, False, False,  True,\n",
            "        False,  True,  True, False, False, False, False,  True, False, False,\n",
            "         True, False,  True, False, False, False,  True,  True, False,  True,\n",
            "        False, False, False, False, False, False,  True,  True,  True, False,\n",
            "        False,  True, False, False, False, False,  True,  True, False, False,\n",
            "         True, False, False, False, False, False, False,  True, False, False,\n",
            "         True, False,  True, False,  True,  True, False, False, False,  True,\n",
            "        False,  True, False, False, False,  True, False, False,  True,  True,\n",
            "        False, False, False, False,  True, False, False, False, False, False,\n",
            "        False,  True,  True, False, False, False, False, False,  True,  True,\n",
            "         True, False,  True, False, False, False,  True, False,  True, False,\n",
            "        False, False, False,  True,  True, False,  True, False,  True,  True,\n",
            "        False, False,  True,  True, False, False,  True, False, False, False,\n",
            "        False,  True,  True, False,  True,  True, False,  True,  True, False,\n",
            "         True, False,  True,  True, False, False,  True, False,  True, False,\n",
            "         True, False,  True,  True, False, False,  True, False,  True, False,\n",
            "         True,  True,  True, False,  True,  True,  True, False,  True, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False,  True,  True,  True, False,  True,  True, False,  True, False,\n",
            "         True, False, False, False, False, False,  True,  True, False, False,\n",
            "         True, False,  True,  True,  True, False,  True, False,  True])\n",
            "label_input[0]-multi-label: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n",
            "evaluate crossencoder on all candidate labels\n",
            "Evaluation:   0% 0/329 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
            "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
            "Evaluation: 100% 329/329 [02:43<00:00,  2.01it/s]\n",
            "num_labels: 50\n",
            "/content/LM-ontology-concept-placement/blink/candidate_ranking/multi_label_evaluation.py:233: RuntimeWarning: invalid value encountered in scalar divide\n",
            "  return intersect_size(yhatmic, ymic, 0) / yhatmic.sum(axis=0)\n",
            "/content/LM-ontology-concept-placement/blink/candidate_ranking/multi_label_evaluation.py:203: RuntimeWarning: invalid value encountered in divide\n",
            "  vals.append(num_true_in_top_k / float(denom))\n",
            "num in insertion_all: 0 num_ave: 0.0\n",
            "num in insertion_any: 0 num_ave: 0.0\n",
            "\n",
            "[MACRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.3837\n",
            "[MICRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, nan, 0.0000, nan, 0.3404\n",
            "rec_at_1: 0.0012\n",
            "prec_at_1: 0.0061\n",
            "rec_at_3: 0.0087\n",
            "prec_at_3: 0.0061\n",
            "rec_at_5: 0.0172\n",
            "prec_at_5: 0.0073\n",
            "rec_at_10: 0.0296\n",
            "prec_at_10: 0.0061\n",
            "rec_at_50: 0.5745\n",
            "prec_at_50: 0.0143\n",
            "ins_at_1_all: 0.0000\n",
            "ins_at_1_any: 0.0061\n",
            "ins_at_3_all: 0.0030\n",
            "ins_at_3_any: 0.0152\n",
            "ins_at_5_all: 0.0061\n",
            "ins_at_5_any: 0.0213\n",
            "ins_at_10_all: 0.0182\n",
            "ins_at_10_any: 0.0304\n",
            "ins_at_50_all: 0.4286\n",
            "ins_at_50_any: 0.5745\n",
            "ins_all: 0.0000\n",
            "ins_any: 0.0000\n",
            "\n",
            "labels_aggregated: [[36106, 36143, 36153, 36121, 36117, 36146, 25337, 36119, 36130, 36125, 36122, 36116, 36154, 23599, 36129, 36131, 36149, 36112, 36127, 36114, 36135, 36151, 36120, 36152, 36109, 36140, 36110, 36126, 36138, 36133, 36136, 36134, 36147, 36137, 36113, 36124, 36139, 36144, 36105, 36118, 36128, 36107, 36141, 36111, 36142, 36132, 36150, 36115, 36123], [78182, 78186, 78188, 78185, 78179, 78181, 78184, 78190, 78180, 78187, 78189], [78181, 78185, 78190, 78187, 78180, 78188, 78186, 78184, 78179, 78189, 78182], [39163, 4302], [92674], [81833, 148461, 221228, 221226, 221227, 221225], [78180, 78186, 78190, 78182, 78187, 78189, 78179, 78185, 78188, 78181, 78184], [29419, 34030], [36132, 36109, 36127, 36139, 36121, 36147, 36105, 36150, 36120, 36115, 36129, 36151, 36114, 36154, 36153, 36135, 36123, 36126, 36140, 36134, 36149, 36116, 36144, 36112, 25337, 36131, 36119, 36130, 36117, 36141, 36138, 36143, 36107, 36136, 36106, 36124, 36113, 36142, 36137, 36146, 36110, 36111, 36128, 23599, 36122, 36133, 36118, 36125, 36152], [248], [248], [78182, 78190, 78184, 78186, 78188, 78180, 78185, 78187, 78181, 78179, 78189], [92674], [37046], [248], [78182, 78181, 78189, 78179, 78180, 78187, 78190, 78188, 78184, 78186, 78185], [78184, 78181, 78186, 78188, 78179, 78189, 78180, 78187, 78185, 78182, 78190], [248], [248], [38385, 179750, 181305], [85722], [78188, 78179, 78190, 78181, 78182, 78187, 78184, 78185, 78189, 78186, 78180], [248], [83514, 162767], [189415, 53213, 11205], [248], [179750, 38385, 181305], [248], [38385, 179750, 181305], [78186, 78179, 78182, 78188, 78189, 78184, 78187, 78181, 78185, 78190, 78180], [248], [248], [248], [42688, 67605], [34030, 29419], [248], [34030, 29419], [39163, 4302], [248], [248], [248], [248], [29419, 34030], [248], [78190, 78182, 78185, 78186, 78188, 78181, 78189, 78179, 78187, 78180, 78184], [248, 3093], [248], [85722], [151525], [248], [181305, 179750, 38385], [248], [181305, 179750, 38385], [79357], [181305, 38385, 179750], [248], [181305, 179750, 38385], [248], [248], [78185, 78182, 78190, 78179, 78184, 78181, 78186, 78188, 78189, 78180, 78187], [248], [248], [151525], [3093, 248], [78190, 78182, 78187, 78188, 78189, 78186, 78184, 78179, 78181, 78185, 78180], [248], [248], [92674], [248], [248], [92674], [248, 3093], [78187, 78179, 78190, 78186, 78180, 78182, 78184, 78189, 78181, 78185, 78188], [248], [42688, 67605], [92674], [179750, 38385, 181305], [39163, 4302], [4302, 39163], [248], [248], [248], [248], [92674], [169117], [83514, 162767], [248], [38385, 179750, 181305], [248], [248], [248], [38385, 181305, 179750], [248], [248], [248], [248], [248], [248], [248], [248], [67605, 42688], [248], [39163, 4302], [38385, 181305, 179750], [19146], [162767, 83514], [181305, 179750, 38385], [248], [248], [92674], [151525], [169117], [179750, 38385, 181305], [37046], [248], [169117], [3093, 248], [92674], [248], [248], [248], [92674], [248], [248], [37046], [181305, 179750, 38385], [248], [248], [248], [248], [248], [248], [248], [37046], [248], [248, 3093], [179750, 181305, 38385], [85722], [42688, 67605], [67605, 42688], [248], [248], [248], [248], [78185, 78189, 78188, 78187, 78182, 78180, 78181, 78184, 78190, 78179, 78186], [248], [248], [85722], [104738, 104736, 104740, 104739, 104737], [151525], [181305, 179750, 38385], [92674], [248], [181305, 179750, 38385], [38385, 181305, 179750], [248], [104740, 104738, 104737, 104739, 104736], [104736, 104739, 104737, 104740, 104738], [248], [93235, 3413], [248], [7133], [248], [248], [248], [181305, 179750, 38385], [85722], [248], [248], [181305, 179750, 38385], [248], [179750, 38385, 181305], [67605, 42688], [38385, 179750, 181305], [248], [248], [248], [92674], [29419, 34030], [3093, 248], [248], [248], [248], [181305, 38385, 179750], [248], [92674], [38385, 179750, 181305], [177189], [83514, 162767], [92674], [248], [248], [3413, 93235], [3093, 248], [248, 3093], [248], [151525], [248], [248], [248], [3413, 93235], [248], [248], [248, 3093], [29419, 34030], [67605, 42688], [162767, 83514], [83514, 162767], [179750, 181305, 38385], [248], [85722], [37046], [248], [248], [181305, 38385, 179750], [169012], [151320, 9836], [42688, 67605], [248], [248], [248], [248], [92674], [248], [248], [248], [248], [67605, 42688], [248], [42688, 67605], [93235, 3413], [151525], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [4302, 39163], [248], [37046], [85722], [248], [37046], [248], [248], [248], [248], [248], [248], [248], [151525], [248], [248], [85722], [181305, 179750, 38385], [175844], [248], [248], [248], [248], [92674], [248], [248], [248], [248], [69827], [248], [248, 3093], [29419, 34030], [248], [248], [248], [85722], [248], [248], [177045], [92674], [248], [248], [248], [248], [248], [248], [248], [4539], [85722], [248], [248], [248], [92674], [248], [248], [248], [248], [248], [248], [92674], [151525], [92674], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [37046], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [92674]]\n",
            "predictions[0]: [('papulonodular disease of skin (disorder)', 'NULL'), ('artefactual skin disease (disorder)', 'NULL'), ('disorder of skin (disorder)', 'subcorneal pustular dermatosis with paraproteinemia (disorder)'), ('histologic type of inflammatory skin disorder (disorder)', 'papulosquamous dermatosis (disorder)'), ('factitious skin disease (disorder)', 'pseudostigmata (disorder)'), ('artefactual skin disease (disorder)', 'pseudostigmata (disorder)'), ('artefactual skin disease (disorder)', 'religious stigmata (disorder)'), ('disorder of skin (disorder)', 'subcorneal pustular dermatosis (disorder)'), ('inflammatory pseudotumor of the skin (disorder)', 'NULL'), ('dermatological non-disease (disorder)', 'NULL'), ('skin lesion (disorder)', 'pustule (disorder)'), ('factitious skin disease (disorder)', 'artefactual skin disease (disorder)'), ('disorder of skin (disorder)', 'exuberant cicatrix (disorder)'), ('factitious skin disease (disorder)', 'religious stigmata (disorder)'), ('skin lesion (disorder)', 'chronic ulcer of skin (disorder)'), ('histologic type of inflammatory skin disorder (disorder)', 'NULL'), ('chronic inflammatory disorder (disorder)', 'chronic generalized exfoliative dermatitis (disorder)'), ('disorder of skin appendage (disorder)', 'pustular folliculitis (disorder)'), ('skin lesion (disorder)', 'post-inflammatory scarring (disorder)'), ('disorder of skin (disorder)', 'papulonodular disease of skin (disorder)'), ('pustule (disorder)', 'NULL'), ('systemic diseases affecting skin (disorder)', 'NULL'), ('histologic type of inflammatory skin disorder (disorder)', 'intraepidermal pustular dermatitis (disorder)'), ('disorder of skin (disorder)', 'histologic type of inflammatory skin disorder (disorder)'), ('inflammatory disorder (disorder)', 'chronic dermatitis (disorder)'), ('disorder of integument (disorder)', 'subcorneal pustular dermatosis (disorder)'), ('histologic type of inflammatory skin disorder (disorder)', 'perifolliculitis (disorder)'), ('disorder of skin (disorder)', 'NULL'), ('self-inflicted skin lesions (disorder)', 'artefactual skin disease (disorder)'), ('skin lesion (disorder)', 'NULL'), ('disorder of skin appendage (disorder)', 'NULL'), ('disorder of integument (disorder)', 'papulonodular disease of skin (disorder)'), ('disorder of integument (disorder)', 'NULL'), ('factitious skin disease (disorder)', 'factitious blistering (disorder)'), ('self-inflicted skin lesions (disorder)', 'NULL'), ('factitious skin disease (disorder)', 'self-inflicted caustic burn (disorder)'), ('dermatitis (disorder)', 'NULL'), ('disorder of skin (disorder)', 'factitious blistering (disorder)'), ('disorder of skin (disorder)', 'inflammatory hyperkeratotic dermatosis, acute (disorder)'), ('dermatitis (disorder)', 'papulosquamous dermatosis (disorder)'), ('eruption of skin (disorder)', 'NULL'), ('cutaneous pseudolymphoma (disorder)', 'NULL'), ('disorders of cutaneous image and perception (disorder)', 'NULL'), ('disorder by body site (disorder)', 'NULL'), ('chronic inflammatory disorder (disorder)', 'NULL'), ('inflammatory disorder (disorder)', 'NULL'), ('chronic disease (disorder)', 'NULL'), ('disease (disorder)', 'NULL'), ('mental disorder (disorder)', 'NULL'), ('cutaneous b-cell pseudolymphoma (disorder)', 'NULL')]\n",
            "pred_idx_tuples[0]: [('128042009', 'SCTID_NULL'), ('402737007', 'SCTID_NULL'), ('95320005', '403217003'), ('277408007', '22585003'), ('402736003', '403581008'), ('402737007', '403581008'), ('402737007', '403580009'), ('95320005', '25147002'), ('404100005', 'SCTID_NULL'), ('238960002', 'SCTID_NULL'), ('95324001', '271760008'), ('402736003', '402737007'), ('95320005', '201308009'), ('402736003', '403580009'), ('95324001', '19429009'), ('277408007', 'SCTID_NULL'), ('128294001', '31922003'), ('238714008', '53651004'), ('95324001', '402680006'), ('95320005', '128042009'), ('271760008', 'SCTID_NULL'), ('238980001', 'SCTID_NULL'), ('277408007', '78182002'), ('95320005', '277408007'), ('128139000', '34936007'), ('128598002', '25147002'), ('277408007', '83341004'), ('95320005', 'SCTID_NULL'), ('274949005', '402737007'), ('95324001', 'SCTID_NULL'), ('238714008', 'SCTID_NULL'), ('128598002', '128042009'), ('128598002', 'SCTID_NULL'), ('402736003', '403578003'), ('274949005', 'SCTID_NULL'), ('402736003', '403582001'), ('182782007', 'SCTID_NULL'), ('95320005', '403578003'), ('95320005', '123689005'), ('182782007', '22585003'), ('271807003', 'SCTID_NULL'), ('128862000', 'SCTID_NULL'), ('238959007', 'SCTID_NULL'), ('123946008', 'SCTID_NULL'), ('128294001', 'SCTID_NULL'), ('128139000', 'SCTID_NULL'), ('27624003', 'SCTID_NULL'), ('64572001', 'SCTID_NULL'), ('74732009', 'SCTID_NULL'), ('128865003', 'SCTID_NULL')]\n",
            "scores[0]: [0.032883584, 0.032326624, 0.04588513, 0.04756957, 0.03808202, 0.035291035, 0.033692017, 0.044417407, 0.032551453, 0.031250726, 0.03896258, 0.03890996, 0.035760086, 0.03455624, 0.03486605, 0.032473907, 0.044378348, 0.07341105, 0.03611136, 0.038349528, 0.03199406, 0.030923324, 0.0540531, 0.037689142, 0.03342786, 0.05105779, 0.04039114, 0.0308478, 0.039219618, 0.031973317, 0.031892207, 0.03958072, 0.031415857, 0.040717263, 0.032257497, 0.04129006, 0.031079402, 0.038761765, 0.039228063, 0.03817737, 0.031285454, 0.03195308, 0.031508133, 0.031317152, 0.031188209, 0.030924603, 0.031473488, 0.03133813, 0.034141827, 0.03240272]\n",
            "gold_idx_tuples[0]: [('95320005', '277408007'), ('95320005', '447360007'), ('95320005', '402587003'), ('95320005', '10599009'), ('95320005', '22649008'), ('95320005', '50563003'), ('95320005', '34936007'), ('95320005', '88697005'), ('95320005', '400020001'), ('95320005', '111196000'), ('95320005', '275447008'), ('95320005', '397845003'), ('95320005', '93423006'), ('95320005', '26435006'), ('95320005', '200890005'), ('95320005', '399992009'), ('95320005', '95393002'), ('95320005', '49084001'), ('95320005', '109248001'), ('95320005', '109249009'), ('95320005', '9480005'), ('95320005', '402432002'), ('95320005', '13600006'), ('95320005', '24152003'), ('95320005', '446932005'), ('95320005', '39524009'), ('95320005', '238751002'), ('95320005', '3014005'), ('95320005', '240897005'), ('95320005', '238471005'), ('95320005', '111921000'), ('95320005', '90900004'), ('95320005', '88558006'), ('95320005', '12873001'), ('95320005', '425267003'), ('95320005', '27720003'), ('95320005', '236125004'), ('95320005', '70139002'), ('95320005', '400009001'), ('95320005', '53107009'), ('95320005', '35498005'), ('95320005', '397741002'), ('95320005', '13582007'), ('95320005', '279353003'), ('95320005', '7392002'), ('95320005', '8197001'), ('95320005', '58759008'), ('95320005', '70759006'), ('95320005', '58921009')]\n",
            "label_input/samples-aggregated:329/329\n",
            "INFO:Blink:save predictions to models/crossencoder/mm+Disease-2017AA/original-NIL-top50-pubmedbert\n",
            "INFO:Blink:mm-test-NIL\n",
            "INFO:Blink:test dataset mode\n",
            "INFO:Blink:965/965 samples considered\n",
            "keep_all: False\n",
            "INFO:Blink:preparing data for biencoder\n",
            "args.use_NIL_tag: False\n",
            "args.use_NIL_desc: False\n",
            "args.use_NIL_desc_tag: False\n",
            "num of samples: 965\n",
            "label_text_vecs: tensor([[   2, 4770, 1927,  ...,    0,    0,    0],\n",
            "        [   2, 4770, 1927,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0]])\n",
            "100% 237826/237826 [00:01<00:00, 168365.52it/s]\n",
            "INFO:Blink:run biencoder\n",
            "100% 121/121 [00:24<00:00,  4.91it/s]\n",
            "labels: 965 1\n",
            "nns: 965 50\n",
            "all_scores_sorted: 965 50\n",
            "all_scores: 965 237826\n",
            "INFO:Blink:Enriching edge candidates\n",
            "INFO:Blink:args.top_k_cand_seed: 25\n",
            "100% 965/965 [00:00<00:00, 4376.05it/s]\n",
            "nns: 965 50\n",
            "labels: 965 [36144] <class 'numpy.ndarray'>\n",
            "nns: 965 [163910 148993 110542 163934  49848 162620 182332 162624 163921 163931\n",
            "  10656 182331 234764 163018  36119 146000 148998 234669 225075 162627\n",
            "  83653 164725 164717 164983 173910 163935 234657 163911 234707 163932\n",
            "   3593 164724  84034  83674 149003 164726 162625  77976   2241   7136\n",
            "  66198 162623   7877 162626 163936     76  49291 163926  32179 175496]\n",
            "missed case (ind 0, within first 5): ('95320005', '70139002') not in [('402277003', 'SCTID_NULL'), ('128236002', '31922003'), ('95320005', '90446007'), ('402282005', 'SCTID_NULL'), ('95320005', '201308009'), ('95320005', '400037008'), ('444004001', 'SCTID_NULL'), ('95320005', '275448003'), ('236124000', '402281003'), ('402281003', '402282005'), ('95320005', '238811006'), ('95320005', '444004001'), ('95320005', '70582006'), ('95320005', '403669004'), ('95320005', '88697005'), ('34936007', '31922003'), ('182782007', '31922003'), ('128598002', '400037008'), ('32110003', '195469007'), ('95320005', '95546008'), ('238698004', '238702000'), ('[EX.](<609096000> [EX.](<42752001> <70582006>))', '403273006'), ('[EX.](<609096000> [EX.](<42752001> <70582006>))', '402622001'), ('95320005', '402682003'), ('95320005', '418242004'), ('402281003', 'SCTID_NULL'), ('128598002', '418242004'), ('402276007', 'SCTID_NULL'), ('128598002', '444004001'), ('402278008', '402282005'), ('34936007', 'SCTID_NULL'), ('[EX.](<609096000> [EX.](<42752001> <70582006>))', '403692004'), ('95320005', '403671004'), ('414662006', '238702000'), ('128294001', '31922003'), ('[EX.](<609096000> [EX.](<42752001> <70582006>))', '403698000'), ('95320005', '109287001'), ('236124000', 'SCTID_NULL'), ('182782007', 'SCTID_NULL'), ('95320005', 'SCTID_NULL'), ('128598002', 'SCTID_NULL'), ('95320005', '238919009'), ('32110003', 'SCTID_NULL'), ('95320005', '402709000'), ('402283000', 'SCTID_NULL'), ('128236002', 'SCTID_NULL'), ('110979008', 'SCTID_NULL'), ('402278008', 'SCTID_NULL'), ('238698004', 'SCTID_NULL'), ('414032001', 'SCTID_NULL')]\n",
            "missed case (ind 1, within first 5): ('95320005', '447360007') not in [('301810000', '279324004'), ('301810000', '430348006'), ('19824006', 'SCTID_NULL'), ('301810000', '19824006'), ('64572001', '19824006'), ('123946008', '19824006'), ('279324004', 'SCTID_NULL'), ('19824006', '238389005'), ('19824006', '430348006'), ('301810000', '108365000'), ('80659006', '430348006'), ('430348006', 'SCTID_NULL'), ('40733004', '430348006'), ('80659006', '19824006'), ('19824006', '85791004'), ('19824006', '135849005'), ('80659006', '95381006'), ('10443009', '430348006'), ('108365000', '397176006'), ('301810000', '397176006'), ('19824006', '278038007'), ('108365000', '279324004'), ('64572001', '108365000'), ('19824006', '48277006'), ('123946008', '363168001'), ('301810000', '206554002'), ('108365000', 'SCTID_NULL'), ('301810000', '128937004'), ('301810000', '2622007'), ('301810000', '432728006'), ('301810000', '128936008'), ('301810000', '95374004'), ('64572001', '206554002'), ('80659006', '278038007'), ('301810000', '128938009'), ('301810000', 'SCTID_NULL'), ('80659006', 'SCTID_NULL'), ('80659006', '70759006'), ('301810000', '367522007'), ('301810000', '400079002'), ('301810000', '402706007'), ('80659006', '111181004'), ('301810000', '111181004'), ('301810000', '278038007'), ('301810000', '70759006'), ('64572001', '128938009'), ('301810000', '43591003'), ('80659006', '238450000'), ('301810000', '95381006'), ('301810000', '3533007')]\n",
            "missed case (ind 2, within first 5): ('236423003', '431857002') not in [('236498004', '236500003'), ('236498004', '236511006'), ('236498004', '13886001'), ('236498004', '236499007'), ('236498004', '445170001'), ('236498004', '420279001'), ('106000008', '236455003'), ('236498004', '421893009'), ('90708001', '236519008'), ('106000008', '190829000'), ('106000008', '236495001'), ('90708001', '106000008'), ('236498004', '127013003'), ('90708001', '197752005'), ('90708001', '236498004'), ('106000008', '236496000'), ('236498004', '197605007'), ('90708001', '236495001'), ('236514003', '236519008'), ('236498004', '32278006'), ('236498004', '236506009'), ('236498004', '110996009'), ('236498004', '50581000'), ('236498004', '55006001'), ('236498004', '236508005'), ('236498004', '54181000'), ('236498004', 'SCTID_NULL'), ('90708001', '363288006'), ('90708001', '127013003'), ('236498004', '309426007'), ('236498004', '63510008'), ('90708001', '13886001'), ('236498004', '310387003'), ('90708001', '236507000'), ('75934005', '236495001'), ('90708001', '50581000'), ('90708001', '79385002'), ('106000008', 'SCTID_NULL'), ('90708001', '67132008'), ('90708001', '42927005'), ('90708001', '33763006'), ('90708001', '54879000'), ('90708001', '236502006'), ('236498004', '311366001'), ('90708001', 'SCTID_NULL'), ('90708001', '236504007'), ('42030000', '197752005'), ('236514003', 'SCTID_NULL'), ('236514003', '59400006'), ('236514003', '236520002')]\n",
            "missed case (ind 3, within first 5): ('236423003', '771000119108') not in [('106000008', '236455003'), ('90708001', '236479001'), ('106000008', '236495001'), ('90708001', '48638002'), ('90708001', '197664009'), ('90708001', '85487008'), ('106000008', '33763006'), ('106000008', '42927005'), ('106000008', '85487008'), ('106000008', '190829000'), ('106000008', '54879000'), ('90708001', '236495001'), ('90708001', '236430009'), ('90708001', '367540006'), ('90708001', '33763006'), ('106000008', '236496000'), ('75934005', '236495001'), ('90708001', '54879000'), ('90708001', '236710009'), ('90708001', '50581000'), ('75934005', '48713002'), ('90708001', '427649000'), ('236437007', '236479001'), ('90708001', '370494002'), ('90708001', '106000008'), ('90708001', '42927005'), ('90708001', '276584001'), ('75934005', '54879000'), ('236437007', '236481004'), ('75934005', '33763006'), ('106000008', 'SCTID_NULL'), ('236437007', '236480003'), ('75934005', '42927005'), ('90708001', '79385002'), ('75934005', '85487008'), ('90708001', 'SCTID_NULL'), ('90708001', '67132008'), ('90708001', '23697004'), ('106000008', '46785007'), ('90708001', '17901006'), ('90708001', '41305006'), ('42030000', '370494002'), ('75934005', 'SCTID_NULL'), ('236437007', 'SCTID_NULL'), ('90708001', '124147007'), ('64572001', '15123008'), ('42030000', '367540006'), ('42030000', '106000008'), ('64572001', 'SCTID_NULL'), ('128121009', 'SCTID_NULL')]\n",
            "biencoder accuracy: 0.0000\n",
            "biencoder recall@50: 0.1482\n",
            "  0% 0/965 [00:00<?, ?it/s]context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'dermatitis', '[unused1]'], 'mention_ids': [1, 18006, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'incontinence', '-', 'associated', 'dermatitis', ':', 'reducing', 'adverse', 'events', 'incontinence', '-', 'associated', '[unused0]', 'dermatitis', '[unused1]', '(', 'iad', ')', 'is', 'a', 'common', 'problem', 'in', 'patients', 'with', 'faecal', 'and', '/', 'or', 'urinary', 'incontinence', '[SEP]'], 'ids': [2, 16783, 17, 2458, 18006, 30, 5526, 5246, 3916, 16783, 17, 2458, 1, 18006, 1, 12, 26842, 13, 1977, 43, 3059, 4915, 1922, 2132, 1956, 15380, 1930, 19, 2014, 7075, 16783, 3], 'contextual_tokens': ['[CLS]', 'incontinence', '-', 'associated', 'dermatitis', ':', 'reducing', 'adverse', 'events', 'incontinence', '-', 'associated', '[unused0]', 'dermatitis', '[unused1]', '(', 'iad', ')', 'is', 'a', 'common', 'problem', 'in', 'patients', 'with', 'faecal', 'and', '/', 'or', 'urinary', 'incontinence', '[SEP]'], 'contextual_ids': [2, 16783, 17, 2458, 18006, 30, 5526, 5246, 3916, 16783, 17, 2458, 1, 18006, 1, 12, 26842, 13, 1977, 43, 3059, 4915, 1922, 2132, 1956, 15380, 1930, 19, 2014, 7075, 16783, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'iad', '[unused1]'], 'mention_ids': [1, 26842, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'penetrate', 'the', 'skin', ',', 'increasing', 'the', 'risk', 'of', 'secondary', 'infection', '.', 'however', ',', '[unused0]', 'iad', '[unused1]', 'can', 'be', 'prevented', 'and', 'healed', 'with', 'timely', 'and', 'appropriate', 'skin', 'clean', '##sin', '##g', 'and', '[SEP]'], 'ids': [2, 21224, 1920, 4407, 16, 3605, 1920, 2565, 1927, 4067, 2909, 18, 2406, 16, 1, 26842, 1, 2112, 1998, 8715, 1930, 25515, 1956, 14918, 1930, 4703, 4407, 8901, 6777, 1029, 1930, 3], 'contextual_tokens': ['[CLS]', 'penetrate', 'the', 'skin', ',', 'increasing', 'the', 'risk', 'of', 'secondary', 'infection', '.', 'however', ',', '[unused0]', 'iad', '[unused1]', 'can', 'be', 'prevented', 'and', 'healed', 'with', 'timely', 'and', 'appropriate', 'skin', 'clean', '##sin', '##g', 'and', '[SEP]'], 'contextual_ids': [2, 21224, 1920, 4407, 16, 3605, 1920, 2565, 1927, 4067, 2909, 18, 2406, 16, 1, 26842, 1, 2112, 1998, 8715, 1930, 25515, 1956, 14918, 1930, 4703, 4407, 8901, 6777, 1029, 1930, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'chronic', 'kidney', 'disease', '[unused1]'], 'mention_ids': [1, 3724, 5245, 2573, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'to', 'determine', 'the', 'relationship', 'between', 'obstructive', 'sleep', 'apnea', '(', 'osa', ')', 'and', '[unused0]', 'chronic', 'kidney', 'disease', '[unused1]', '(', 'ckd', ')', '.', 'previous', 'population', 'studies', 'of', 'the', 'association', 'are', 'sparse', ',', '[SEP]'], 'ids': [2, 1942, 3415, 1920, 3303, 2192, 12234, 5006, 17578, 12, 15052, 13, 1930, 1, 3724, 5245, 2573, 1, 12, 9928, 13, 18, 2555, 2973, 2351, 1927, 1920, 3279, 2032, 13766, 16, 3], 'contextual_tokens': ['[CLS]', 'to', 'determine', 'the', 'relationship', 'between', 'obstructive', 'sleep', 'apnea', '(', 'osa', ')', 'and', '[unused0]', 'chronic', 'kidney', 'disease', '[unused1]', '(', 'ckd', ')', '.', 'previous', 'population', 'studies', 'of', 'the', 'association', 'are', 'sparse', ',', '[SEP]'], 'contextual_ids': [2, 1942, 3415, 1920, 3303, 2192, 12234, 5006, 17578, 12, 15052, 13, 1930, 1, 3724, 5245, 2573, 1, 12, 9928, 13, 18, 2555, 2973, 2351, 1927, 1920, 3279, 2032, 13766, 16, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'ckd', '[unused1]'], 'mention_ids': [1, 9928, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', '##phosphate', '##mia', 'is', 'a', 'major', 'determinant', 'of', 'mortality', 'in', 'chronic', 'kidney', 'disease', '(', '[unused0]', 'ckd', '[unused1]', ')', '.', 'vascular', 'calcification', 'is', 'driven', 'by', 'aldosterone', '-', 'sensitive', 'osteogenic', 'transformation', 'of', 'vascular', '[SEP]'], 'ids': [2, 25324, 6026, 1977, 43, 3044, 12483, 1927, 3885, 1922, 3724, 5245, 2573, 12, 1, 9928, 1, 13, 18, 4541, 14640, 1977, 6446, 2007, 16944, 17, 4532, 15035, 7002, 1927, 4541, 3], 'contextual_tokens': ['[CLS]', '##phosphate', '##mia', 'is', 'a', 'major', 'determinant', 'of', 'mortality', 'in', 'chronic', 'kidney', 'disease', '(', '[unused0]', 'ckd', '[unused1]', ')', '.', 'vascular', 'calcification', 'is', 'driven', 'by', 'aldosterone', '-', 'sensitive', 'osteogenic', 'transformation', 'of', 'vascular', '[SEP]'], 'contextual_ids': [2, 25324, 6026, 1977, 43, 3044, 12483, 1927, 3885, 1922, 3724, 5245, 2573, 12, 1, 9928, 1, 13, 18, 4541, 14640, 1977, 6446, 2007, 16944, 17, 4532, 15035, 7002, 1927, 4541, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'head', 'and', 'neck', 'squamous', 'cell', 'carcinoma', '[unused1]'], 'mention_ids': [1, 4469, 1930, 7467, 10364, 2024, 5373, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', 'reaction', '(', 'rt', '-', 'pcr', ')', 'in', '22', 'patients', 'with', 'primary', '[unused0]', 'head', 'and', 'neck', 'squamous', 'cell', 'carcinoma', '[unused1]', '(', 'hnscc', ')', '.', 'statistically', 'significant', 'associations', 'were', 'found', 'between', 'herg', '[SEP]'], 'ids': [2, 3227, 12, 4341, 17, 3343, 13, 1922, 2950, 2132, 1956, 3065, 1, 4469, 1930, 7467, 10364, 2024, 5373, 1, 12, 16499, 13, 18, 4388, 2228, 5253, 1985, 2435, 2192, 25411, 3], 'contextual_tokens': ['[CLS]', 'reaction', '(', 'rt', '-', 'pcr', ')', 'in', '22', 'patients', 'with', 'primary', '[unused0]', 'head', 'and', 'neck', 'squamous', 'cell', 'carcinoma', '[unused1]', '(', 'hnscc', ')', '.', 'statistically', 'significant', 'associations', 'were', 'found', 'between', 'herg', '[SEP]'], 'contextual_ids': [2, 3227, 12, 4341, 17, 3343, 13, 1922, 2950, 2132, 1956, 3065, 1, 4469, 1930, 7467, 10364, 2024, 5373, 1, 12, 16499, 13, 18, 4388, 2228, 5253, 1985, 2435, 2192, 25411, 3]}\n",
            "context_tokens in prepare_crossencoder_mentions: {'mention_tokens': ['[unused0]', 'iad', '[unused1]'], 'mention_ids': [1, 26842, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['[CLS]', '##mann', 'incontinence', 'pads', 'that', 'have', 'been', 'developed', 'to', 'absorb', 'the', 'fluids', 'that', 'cause', '[unused0]', 'iad', '[unused1]', 'and', 'maintain', 'the', 'skin', \"'\", 's', 'acidic', 'ph', '.', 'the', 'acidic', 'ph', 'of', 'the', '[SEP]'], 'ids': [2, 8890, 16783, 24258, 1988, 2162, 2252, 3530, 1942, 7322, 1920, 13315, 1988, 3936, 1, 26842, 1, 1930, 6698, 1920, 4407, 11, 61, 9931, 2087, 18, 1920, 9931, 2087, 1927, 1920, 3], 'contextual_tokens': ['[CLS]', '##mann', 'incontinence', 'pads', 'that', 'have', 'been', 'developed', 'to', 'absorb', 'the', 'fluids', 'that', 'cause', '[unused0]', 'iad', '[unused1]', 'and', 'maintain', 'the', 'skin', \"'\", 's', 'acidic', 'ph', '.', 'the', 'acidic', 'ph', 'of', 'the', '[SEP]'], 'contextual_ids': [2, 8890, 16783, 24258, 1988, 2162, 2252, 3530, 1942, 7322, 1920, 13315, 1988, 3936, 1, 26842, 1, 1930, 6698, 1920, 4407, 11, 61, 9931, 2087, 18, 1920, 9931, 2087, 1927, 1920, 3]}\n",
            "100% 965/965 [00:00<00:00, 992.63it/s] \n",
            "topk in prepare_crossencoder_candidates: 50\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'chronic', 'papill', '##omatous', 'dermatitis', 'due', 'to', 'incontinence', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 3724, 14668, 13844, 18006, 2810, 1942, 16783, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'chronic', 'disease', 'of', 'skin', '(', 'disorder', ')', '[unused5]', 'chronic', 'generalized', 'exfoli', '##ative', 'dermatitis', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 3724, 2573, 1927, 4407, 12, 4770, 13, 1, 3724, 9612, 28972, 2178, 18006, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'disorder', 'of', 'skin', '(', 'disorder', ')', '[unused5]', 'pruritus', 'an', '##i', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 4770, 1927, 4407, 12, 4770, 13, 1, 27496, 1925, 1033, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'infection', 'by', 'site', '(', 'disorder', ')', '[unused5]', 'site', '-', 'specific', 'infective', 'disorders', 'of', 'skin', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 2909, 2007, 3200, 12, 4770, 13, 1, 3200, 17, 2487, 20905, 4414, 1927, 4407, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'infection', 'by', 'site', '(', 'disorder', ')', '[unused5]', 'localized', 'infection', 'of', 'skin', 'and', '/', 'or', 'subcutaneous', 'tissue', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 2909, 2007, 3200, 12, 4770, 13, 1, 6579, 2909, 1927, 4407, 1930, 19, 2014, 8634, 2960, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'infection', 'of', 'skin', 'and', '/', 'or', 'subcutaneous', 'tissue', '(', 'disorder', ')', '[unused5]', 'null', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 2909, 1927, 4407, 1930, 19, 2014, 8634, 2960, 12, 4770, 13, 1, 6764, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'renal', 'disorders', 'in', 'systemic', 'disease', '(', 'disorder', ')', '[unused5]', 'protein', '##uric', 'diabetic', 'nephropathy', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 4604, 4414, 1922, 5510, 2573, 12, 4770, 13, 1, 2213, 15861, 6152, 15582, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'renal', 'disorders', 'in', 'systemic', 'disease', '(', 'disorder', ')', '[unused5]', 'macro', '##globulin', '##emic', 'nephropathy', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 4604, 4414, 1922, 5510, 2573, 12, 4770, 13, 1, 10133, 27583, 4197, 15582, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "rep in prepare_crossencoder_candidates: {'tokens': ['[CLS]', 'renal', 'disorders', 'in', 'systemic', 'disease', '(', 'disorder', ')', '[unused5]', 'sickle', 'cell', 'nephropathy', '(', 'disorder', ')', '[unused6]', '[unused2]', '[SEP]'], 'ids': [2, 4604, 4414, 1922, 5510, 2573, 12, 4770, 13, 1, 21414, 2024, 15582, 12, 4770, 13, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}\n",
            "label_input_list: [-1 -1 -1 -1 49 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 44 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 48 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 15 -1 -1 -1 -1 -1 -1 -1 45 46 -1 -1 -1  1 -1 -1 -1 46 -1 47 -1 19 -1\n",
            " -1 -1 -1 -1 49 -1 -1 -1 -1 -1 -1 -1 -1 30 -1  1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 47 -1 -1 -1 14 -1 45 -1 33 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 43 -1 -1 -1 -1 -1 -1 -1 -1 44 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " 47 -1 -1 -1 -1 -1 49 -1 47 -1 -1 -1 -1 -1 -1 -1 -1 -1 16 -1  7 -1 -1 46\n",
            " -1 44 11 -1 -1 -1 -1  5 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 46 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 36 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 44 43 -1 -1 -1 -1 34 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 47 19 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1  4 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 35 -1 37 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 49 -1 -1 28 44 -1 41 -1 -1 29 -1 -1 48 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1  6 -1 -1 42 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            "  1 -1 -1 43 -1 44 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1  4 -1 -1 -1 -1 -1 -1 -1 -1 -1 12 48 -1 -1 -1 -1\n",
            " -1 -1 -1 17 -1 -1 -1 -1 34 -1 -1 -1 -1 -1 -1 -1 -1 33 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 41 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 45 -1 -1 47 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 36 -1 32 -1 -1 -1 -1 -1 -1 -1 -1 41 -1 -1 -1 -1 -1 -1 -1 44\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 45 45 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 48 -1 -1 -1 -1 -1 -1 -1 -1 -1 15 -1 -1 -1 -1 47 -1 -1 -1 -1 -1 -1 -1\n",
            " 14 -1 -1 -1 46 46 -1 48 -1 -1 -1 -1 -1 -1 30  9 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " 41 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 49 -1 -1 -1 16 -1 -1 46 -1 19 43\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 40 -1 -1 48 -1 22 43 -1\n",
            " 49 -1 -1 -1 -1 -1 47 13 -1 -1 -1 45 -1 47 49 -1 -1 -1 -1 -1 -1 17 -1 -1\n",
            " -1 -1 49 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 49 48 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 40 -1 -1 -1 -1 -1\n",
            " -1 -1 19 -1 -1 -1 -1 48 18 -1 49 -1 -1 20 -1 -1 42 -1 -1 -1 -1  7 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 46 -1 -1 -1 -1 47 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 48 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 45 -1 -1 -1 41\n",
            " -1 46 49 -1 -1 -1  2 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1  7\n",
            " -1 -1 -1 -1 -1 -1 45 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1  6 -1 -1\n",
            " 47 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 40 -1 -1 -1 16 -1 24 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 33 -1 -1 -1 -1 49 -1 -1 13 -1 -1 -1 -1 -1 -1 45 -1\n",
            " -1 -1 13 45 -1 -1 49 -1 -1 -1 49 47 -1 -1 -1 49 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 47 -1 -1 43 -1 -1  5 -1 -1 -1  1 47 -1 -1 -1 -1 -1 -1 49 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 47 -1 -1 -1 -1 -1 -1 39 24 -1 -1 -1 -1 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 31 -1 -1 -1 -1 -1 -1 -1 -1 -1 30 -1 -1 -1 -1 27 -1 -1 -1 -1\n",
            " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 46 15 -1 -1 -1 -1 -1 -1\n",
            " -1 32 27 -1 -1]\n",
            "label_input: tensor([-1, -1, -1, -1, 49, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 44,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, 48, -1, -1, -1, -1, -1, -1, -1, -1, 15, -1, -1, -1, -1,\n",
            "        -1, -1, -1, 45, 46, -1, -1, -1,  1, -1, -1, -1, 46, -1, 47, -1, 19, -1,\n",
            "        -1, -1, -1, -1, 49, -1, -1, -1, -1, -1, -1, -1, -1, 30, -1,  1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 47, -1, -1,\n",
            "        -1, 14, -1, 45, -1, 33, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 43, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, 44, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        47, -1, -1, -1, -1, -1, 49, -1, 47, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        16, -1,  7, -1, -1, 46, -1, 44, 11, -1, -1, -1, -1,  5, 24, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, 46, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, 36, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, 44, 43, -1, -1, -1, -1, 34, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, 47, 19, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1,  4, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 35, -1, 37, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, 49, -1, -1, 28, 44, -1, 41, -1, -1,\n",
            "        29, -1, -1, 48, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  6, -1, -1, 42,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  1, -1, -1, 43, -1, 44,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1,  4, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        12, 48, -1, -1, -1, -1, -1, -1, -1, 17, -1, -1, -1, -1, 34, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, 33, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 41,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 45, -1, -1, 47, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, 36, -1, 32, -1, -1, -1, -1, -1, -1, -1, -1, 41, -1, -1,\n",
            "        -1, -1, -1, -1, -1, 44, -1, -1, -1, -1, -1, -1, -1, -1, 45, 45, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 48, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, 15, -1, -1, -1, -1, 47, -1, -1, -1, -1, -1, -1, -1,\n",
            "        14, -1, -1, -1, 46, 46, -1, 48, -1, -1, -1, -1, -1, -1, 30,  9, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 41, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, 49, -1, -1, -1, 16, -1, -1, 46, -1, 19, 43,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 40, -1,\n",
            "        -1, 48, -1, 22, 43, -1, 49, -1, -1, -1, -1, -1, 47, 13, -1, -1, -1, 45,\n",
            "        -1, 47, 49, -1, -1, -1, -1, -1, -1, 17, -1, -1, -1, -1, 49, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 49, 48, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        40, -1, -1, -1, -1, -1, -1, -1, 19, -1, -1, -1, -1, 48, 18, -1, 49, -1,\n",
            "        -1, 20, -1, -1, 42, -1, -1, -1, -1,  7, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, 46, -1, -1, -1, -1, 47, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, 48, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, 45, -1, -1, -1, 41, -1, 46, 49, -1, -1, -1,  2, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  7, -1, -1, -1, -1, -1, -1,\n",
            "        45, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  6, -1, -1,\n",
            "        47, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 40, -1, -1,\n",
            "        -1, 16, -1, 24, -1, -1, -1, -1, -1, -1, -1, -1, -1, 33, -1, -1, -1, -1,\n",
            "        49, -1, -1, 13, -1, -1, -1, -1, -1, -1, 45, -1, -1, -1, 13, 45, -1, -1,\n",
            "        49, -1, -1, -1, 49, 47, -1, -1, -1, 49, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, 47, -1, -1, 43, -1, -1,  5, -1, -1, -1,  1, 47, -1, -1, -1,\n",
            "        -1, -1, -1, 49, -1, -1, -1, -1, -1, -1, -1, -1, -1, 47, -1, -1, -1, -1,\n",
            "        -1, -1, 39, 24, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 31, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, 30, -1, -1, -1, -1, 27, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 46, 15,\n",
            "        -1, -1, -1, -1, -1, -1, -1, 32, 27, -1, -1])\n",
            "tensor_is_NIL_labels: tensor([False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False])\n",
            "label_input[0]: tensor(-1)\n",
            "samples[0]: {'context_left': 'incontinence -associated dermatitis: reducing adverse events incontinence -associated ', 'mention': 'dermatitis', 'context_right': \" (iad) is a common problem in patients with faecal and/or urinary incontinence. urine alters the normal skin flora and increases permeability of the stratum corneum and faecal enzymes on the skin contribute to skin damage. faecal bacteria can then penetrate the skin, increasing the risk of secondary infection. however, iad can be prevented and healed with timely and appropriate skin cleansing and skin protection. this includes appropriate use of containment devices. this article also looks at hartmann incontinence pads that have been developed to absorb the fluids that cause iad and maintain the skin's acidic ph. the acidic ph of the skin contributes to its barrier function and defence against infection. therefore, maintaining an acidic ph will help protect the skin from damage.\", 'label_concept_UMLS': 'C0011603', 'label_concept': 36144, 'label_concept_ori': '703938007', 'entity_label_id': 64900, 'entity_label': '', 'entity_label_title': 'NIL', 'parent_concept': '95320005', 'child_concept': '70139002', 'parent': 'disorder of skin (disorder)', 'child': 'harara (disorder)', 'edge_label_id': 36144, 'label_concept_str': ('95320005', '70139002'), 'row_ind_men': 0}\n",
            "max_seq_length: 160\n",
            "doc_ind_aggregated_strs:\n",
            " 0|21|32|34|43|47|56|75|78|84|88|109|146|153|157|189|190|201|223|239|252|273|278|286|307|328|340|348|350|356|370|402|412|421|452|456|467|501|514|523|537|538|554|556|559|563|590|593|596\n",
            "1|8|9|11|13|31|37|46|59|61|63|98|104|107|108|112|121|172|179|184|194|196|210|216|229|241|253|256|266|268|270|296|321|331|334|342|343|354|385|417|434|441|469|483|510|517|529|550|578\n",
            "2|140|147|257|289|326|485|532|536|609|627\n",
            "3|28|114|129|154|155|228|367|444|451|591\n",
            "4\n",
            "5|12|39|41|50|69|81|99|100|126|127|130|135|136|149|181|185|192|200|203|232|233|249|251|275|280|288|318|329|352|357|430|431|433|453|477|480|484|488|489|499|520|533|539|541|560|562|587|628\n",
            "6|72|123|141|143|180|204|238|333|416|500\n",
            "7\n",
            "10|89|118|128|263|295|314|522|534|553|603\n",
            "14\n",
            "15|80|163|236|255|338|374|399|468|542|557\n",
            "16|22|29|65|73|79|90|92|103|106|158|160|171|174|188|193|208|222|250|258|262|279|299|330|346|349|351|363|366|376|382|403|422|426|443|445|446|450|476|490|497|507|521|526|547|549|581|615|620\n",
            "17\n",
            "18|23|36|53|54|82|101|115|119|125|138|159|173|176|177|182|197|206|211|225|227|235|264|267|269|281|287|304|308|313|315|344|359|364|369|371|381|390|396|398|410|461|478|486|503|512|528|605|619\n",
            "19|94|137|175|261|324|337|368|378|462|540\n",
            "20|27|44|74|139|319|400|437|457|470|568\n",
            "24|110|142|335|355|423|449|460|506|573|577\n",
            "25|35|71|102|178|191|244|276|383|414|475\n",
            "26|38|156|231|265|302|361|585|608|616|618\n",
            "30|60|67|148|298|360|365|391|404|567|589\n",
            "33|70|170|218|290|380|387|388|474|584|607\n",
            "40\n",
            "42|51|83|93|95|219|247|327|362|409|576\n",
            "45|48|134|237|294|322|373|386|463|479|586\n",
            "49\n",
            "52\n",
            "55|207|212|215|353|406|411|415|428|564|580\n",
            "57\n",
            "58\n",
            "62\n",
            "64\n",
            "66\n",
            "68\n",
            "76\n",
            "77|187|195|245|332|442|493|515|544|551|597\n",
            "85\n",
            "86\n",
            "87\n",
            "91\n",
            "96|97|310|325|358|395|440|473|487|548|574\n",
            "105\n",
            "111\n",
            "113\n",
            "116|166|274|292|498|502|513|558|561|617|623\n",
            "117|132|151|161|226|277|377|407|482|535|610\n",
            "120|230|254|347|393|425|471|505|524|555|612\n",
            "122|221|246|285|389|394|429|435|492|527|588\n",
            "124|282|569\n",
            "131\n",
            "133\n",
            "144\n",
            "145\n",
            "150\n",
            "152\n",
            "162|320|519\n",
            "164|495\n",
            "165\n",
            "167\n",
            "168\n",
            "169\n",
            "183\n",
            "186\n",
            "198\n",
            "199|214|224|234|439|454|458|459|583|601|604\n",
            "202\n",
            "205\n",
            "209\n",
            "213\n",
            "217\n",
            "220\n",
            "240\n",
            "242\n",
            "243\n",
            "248\n",
            "259\n",
            "260\n",
            "271\n",
            "272\n",
            "283\n",
            "284\n",
            "291\n",
            "293|300|336\n",
            "297\n",
            "301\n",
            "303\n",
            "305\n",
            "306\n",
            "309\n",
            "311\n",
            "312\n",
            "316\n",
            "317\n",
            "323\n",
            "339\n",
            "341\n",
            "345\n",
            "372\n",
            "375\n",
            "379\n",
            "384\n",
            "392\n",
            "397\n",
            "401\n",
            "405\n",
            "408\n",
            "413\n",
            "418\n",
            "419\n",
            "420\n",
            "424\n",
            "427\n",
            "432\n",
            "436\n",
            "438\n",
            "447\n",
            "448\n",
            "455\n",
            "464\n",
            "465\n",
            "466\n",
            "472\n",
            "481\n",
            "491\n",
            "494\n",
            "496\n",
            "504|579\n",
            "508\n",
            "509\n",
            "511\n",
            "516\n",
            "518\n",
            "525\n",
            "530\n",
            "531\n",
            "543\n",
            "545\n",
            "546\n",
            "552\n",
            "565\n",
            "566\n",
            "570\n",
            "571\n",
            "572\n",
            "575\n",
            "582\n",
            "592\n",
            "594\n",
            "595\n",
            "598\n",
            "599\n",
            "600\n",
            "602\n",
            "606\n",
            "611\n",
            "613\n",
            "614\n",
            "621\n",
            "622\n",
            "624\n",
            "625\n",
            "626\n",
            "629\n",
            "630|827|959\n",
            "631|657|665|779|804|826|859|888|893|918|933\n",
            "632|639|651|668|671|695|730|787|835|881|894\n",
            "633|684|780|836|869|906|909|917|935|938|958\n",
            "634|654|667|715|727|738|742|757|805|921|964\n",
            "635|647|680|691|776|786|847|863|913|936|947\n",
            "636|652|710|735|756|765|768|825|848|857|922\n",
            "637|662|690|717|785|794|821|833|840|904|929\n",
            "638|640|648|733|759|760|783|784|791|875|923\n",
            "641\n",
            "642\n",
            "643|645|736|806|818|852|854|860|886|910|931\n",
            "644|672|686|751|758|764|834|844|861|899|914\n",
            "646\n",
            "649|689|697|714|761|788|790|897|900|932|960\n",
            "650|677|694|712|731|752|772|871|889|898|927\n",
            "653\n",
            "655|718|724|732|755|812|872|908|940|941|949\n",
            "656|658|700|722|798|819|853|865|891|924|930\n",
            "659\n",
            "660|687|719|744|749|800|841|882|892|911|944\n",
            "661\n",
            "663|696|703|721|753|769|777|820|822|920|937\n",
            "664|674|685|740|770|775|916|919|942|954|962\n",
            "666\n",
            "669\n",
            "670|683|723|747|802|849|874|879|883|939|955\n",
            "673\n",
            "675|728|734|766|771|799|801|824|901|943|963\n",
            "676\n",
            "678\n",
            "679\n",
            "681|702|720|793|815|816|831|862|890|907|946\n",
            "682\n",
            "688\n",
            "692|716|726|762|797|808|814|829|856|868|905\n",
            "693\n",
            "698\n",
            "699\n",
            "701\n",
            "704\n",
            "705\n",
            "706\n",
            "707\n",
            "708\n",
            "709\n",
            "711\n",
            "713\n",
            "725\n",
            "729\n",
            "737\n",
            "739\n",
            "741\n",
            "743\n",
            "745\n",
            "746\n",
            "748|842|957\n",
            "750|813|953\n",
            "754\n",
            "763\n",
            "767\n",
            "773\n",
            "774\n",
            "778\n",
            "781\n",
            "782\n",
            "789\n",
            "792\n",
            "795\n",
            "796\n",
            "803\n",
            "807\n",
            "809\n",
            "810\n",
            "811|873\n",
            "817\n",
            "823|877|903\n",
            "828\n",
            "830\n",
            "832\n",
            "837\n",
            "838\n",
            "839\n",
            "843\n",
            "845\n",
            "846\n",
            "850\n",
            "851\n",
            "855\n",
            "858|934|961\n",
            "864\n",
            "866\n",
            "867\n",
            "870\n",
            "876\n",
            "878\n",
            "880\n",
            "884\n",
            "885\n",
            "887\n",
            "895\n",
            "896\n",
            "902\n",
            "912\n",
            "915\n",
            "925\n",
            "926\n",
            "928\n",
            "945\n",
            "948\n",
            "950\n",
            "951\n",
            "952\n",
            "956\n",
            "context_input: 965\n",
            "dict_context_to_label_inputs: 276\n",
            "label_single-label[0]: [-1, 14]\n",
            "label_multi-label[0]: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n",
            "label_multi_hot_is_complete: tensor([False, False, False, False,  True, False, False, False, False, False,\n",
            "        False, False,  True, False, False, False, False, False, False, False,\n",
            "        False,  True, False, False,  True, False, False,  True,  True,  True,\n",
            "        False,  True,  True,  True, False,  True, False,  True, False, False,\n",
            "         True,  True,  True, False, False, False, False,  True, False,  True,\n",
            "         True, False,  True,  True,  True, False, False,  True, False,  True,\n",
            "        False,  True, False, False, False, False, False, False,  True, False,\n",
            "        False,  True,  True,  True, False,  True,  True, False, False,  True,\n",
            "        False, False,  True,  True,  True, False,  True,  True, False, False,\n",
            "        False, False,  True,  True,  True, False, False, False,  True, False,\n",
            "         True, False,  True, False, False,  True, False, False, False,  True,\n",
            "         True, False,  True,  True,  True, False,  True,  True,  True, False,\n",
            "        False,  True,  True, False,  True, False,  True,  True,  True, False,\n",
            "         True, False, False, False, False, False, False,  True,  True, False,\n",
            "        False, False,  True,  True, False,  True, False,  True,  True, False,\n",
            "         True, False,  True,  True,  True,  True,  True, False, False, False,\n",
            "         True, False, False, False, False, False, False, False, False, False,\n",
            "        False,  True,  True, False, False, False, False, False, False, False,\n",
            "        False, False, False, False, False, False,  True, False, False, False,\n",
            "        False, False, False,  True, False,  True,  True, False,  True, False,\n",
            "        False, False, False, False, False, False,  True, False, False,  True,\n",
            "         True, False, False,  True, False,  True,  True,  True, False,  True,\n",
            "        False, False,  True, False,  True, False, False, False,  True,  True,\n",
            "        False, False, False,  True, False, False,  True, False,  True,  True,\n",
            "        False, False, False,  True, False,  True, False,  True,  True,  True,\n",
            "         True, False, False, False,  True,  True, False,  True, False, False,\n",
            "         True, False,  True, False,  True, False, False, False,  True, False,\n",
            "        False, False, False, False,  True, False])\n",
            "label_input[0]-multi-label: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
            "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n",
            "evaluate crossencoder on all candidate labels\n",
            "Evaluation:   0% 0/276 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
            "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
            "Evaluation: 100% 276/276 [02:18<00:00,  1.99it/s]\n",
            "num_labels: 50\n",
            "/content/LM-ontology-concept-placement/blink/candidate_ranking/multi_label_evaluation.py:233: RuntimeWarning: invalid value encountered in scalar divide\n",
            "  return intersect_size(yhatmic, ymic, 0) / yhatmic.sum(axis=0)\n",
            "/content/LM-ontology-concept-placement/blink/candidate_ranking/multi_label_evaluation.py:203: RuntimeWarning: invalid value encountered in divide\n",
            "  vals.append(num_true_in_top_k / float(denom))\n",
            "/content/LM-ontology-concept-placement/blink/candidate_ranking/multi_label_evaluation.py:54: RuntimeWarning: invalid value encountered in scalar divide\n",
            "  metrics['f1_at_%d' % k_i] = 2*(prec_at_k*rec_at_k)/(prec_at_k+rec_at_k)\n",
            "num in insertion_all: 0 num_ave: 0.0\n",
            "num in insertion_any: 0 num_ave: 0.0\n",
            "\n",
            "[MACRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, 0.0000, 0.0000, 0.0000, 0.3703\n",
            "[MICRO] accuracy, precision, recall, f-measure, AUC\n",
            "0.0000, nan, 0.0000, nan, 0.2684\n",
            "rec_at_1: 0.0000\n",
            "prec_at_1: 0.0000\n",
            "rec_at_3: 0.0163\n",
            "prec_at_3: 0.0060\n",
            "rec_at_5: 0.0236\n",
            "prec_at_5: 0.0051\n",
            "rec_at_10: 0.0254\n",
            "prec_at_10: 0.0029\n",
            "rec_at_50: 0.4493\n",
            "prec_at_50: 0.0104\n",
            "ins_at_1_all: 0.0000\n",
            "ins_at_1_any: 0.0000\n",
            "ins_at_3_all: 0.0145\n",
            "ins_at_3_any: 0.0181\n",
            "ins_at_5_all: 0.0181\n",
            "ins_at_5_any: 0.0254\n",
            "ins_at_10_all: 0.0181\n",
            "ins_at_10_any: 0.0290\n",
            "ins_at_50_all: 0.3949\n",
            "ins_at_50_any: 0.4493\n",
            "ins_all: 0.0000\n",
            "ins_any: 0.0000\n",
            "\n",
            "labels_aggregated: [[36144, 36120, 36131, 36117, 36137, 36149, 36129, 36121, 36152, 36140, 36126, 36119, 23599, 36124, 36112, 36150, 36139, 36122, 36105, 36109, 36153, 36151, 36146, 36138, 36132, 36115, 36142, 36123, 36133, 36136, 36125, 36135, 36128, 36127, 36114, 36154, 36107, 36134, 36130, 36110, 36116, 25337, 36141, 36113, 36118, 36147, 36106, 36143, 36111], [36143, 36137, 36128, 36119, 36112, 36146, 36151, 36138, 36127, 36120, 36116, 36150, 36110, 36129, 23599, 36139, 36109, 36135, 36149, 36133, 36106, 36113, 36115, 36117, 36134, 36142, 36121, 36107, 25337, 36132, 36111, 36118, 36122, 36147, 36105, 36154, 36152, 36136, 36126, 36130, 36114, 36125, 36153, 36144, 36141, 36140, 36124, 36131, 36123], [78190, 78180, 78184, 78187, 78186, 78188, 78181, 78185, 78189, 78179, 78182], [78179, 78182, 78188, 78190, 78185, 78187, 78189, 78181, 78180, 78186, 78184], [248], [36115, 36132, 36130, 36123, 36116, 36150, 36118, 36131, 36154, 36107, 36112, 36146, 36113, 36117, 36149, 36124, 36128, 36135, 23599, 36105, 36139, 36121, 36111, 36127, 36143, 36125, 36151, 36120, 36110, 36129, 36147, 36133, 36109, 25337, 36119, 36142, 36126, 36106, 36141, 36140, 36122, 36144, 36134, 36136, 36137, 36114, 36153, 36152, 36138], [78182, 78184, 78180, 78190, 78187, 78186, 78189, 78179, 78188, 78185, 78181], [248], [78181, 78187, 78190, 78188, 78185, 78189, 78184, 78179, 78186, 78182, 78180], [248], [78184, 78181, 78186, 78179, 78190, 78180, 78185, 78187, 78188, 78182, 78189], [36107, 36147, 36124, 36127, 36116, 36114, 36115, 36150, 23599, 36154, 36120, 36129, 36153, 36111, 36113, 36118, 36119, 36125, 36105, 36122, 36133, 36110, 36131, 36141, 25337, 36140, 36134, 36128, 36123, 36121, 36106, 36152, 36136, 36109, 36112, 36132, 36130, 36151, 36144, 36146, 36137, 36143, 36117, 36138, 36135, 36142, 36126, 36139, 36149], [248], [36133, 36152, 36151, 36128, 36120, 36106, 36124, 36116, 36132, 36115, 36134, 36121, 36110, 36140, 36127, 36146, 36154, 36137, 36111, 25337, 36114, 36117, 36113, 36136, 23599, 36112, 36142, 36141, 36131, 36105, 36150, 36143, 36126, 36123, 36118, 36149, 36130, 36129, 36135, 36109, 36153, 36125, 36147, 36139, 36119, 36107, 36144, 36138, 36122], [78187, 78188, 78180, 78184, 78182, 78186, 78179, 78185, 78190, 78189, 78181], [78181, 78185, 78188, 78189, 78180, 78182, 78187, 78179, 78184, 78190, 78186], [78186, 78179, 78184, 78181, 78188, 78182, 78185, 78189, 78180, 78187, 78190], [78189, 78182, 78181, 78180, 78187, 78186, 78188, 78184, 78179, 78185, 78190], [78180, 78182, 78189, 78181, 78187, 78179, 78184, 78190, 78185, 78186, 78188], [78188, 78189, 78187, 78182, 78184, 78185, 78181, 78180, 78186, 78190, 78179], [78182, 78188, 78190, 78180, 78181, 78186, 78185, 78187, 78179, 78189, 78184], [248], [78189, 78186, 78190, 78181, 78184, 78187, 78180, 78188, 78179, 78185, 78182], [78184, 78185, 78181, 78188, 78189, 78180, 78187, 78186, 78190, 78179, 78182], [13979], [248], [78187, 78186, 78179, 78190, 78189, 78180, 78182, 78181, 78184, 78185, 78188], [248], [248], [85722], [248], [151525], [248], [248], [78190, 78186, 78181, 78188, 78180, 78182, 78189, 78187, 78184, 78179, 78185], [103123], [248], [85722], [248], [78184, 78187, 78185, 78179, 78182, 78189, 78190, 78180, 78181, 78186, 78188], [248], [248], [103123], [78188, 78180, 78185, 78181, 78187, 78182, 78186, 78190, 78189, 78179, 78184], [78190, 78182, 78179, 78188, 78181, 78185, 78187, 78189, 78180, 78186, 78184], [78189, 78181, 78180, 78179, 78186, 78185, 78184, 78187, 78190, 78188, 78182], [78181, 78185, 78184, 78186, 78188, 78179, 78180, 78187, 78189, 78182, 78190], [181305, 38385, 179750], [151525], [248], [248], [248], [248], [248], [38385, 181305, 179750], [67605, 42688], [248], [248], [151525], [248], [248], [248], [248], [78184, 78181, 78188, 78179, 78190, 78189, 78182, 78187, 78186, 78185, 78180], [248], [248], [248], [248], [151525], [248], [248], [248], [248], [151525], [248], [248], [151525], [85722], [248], [248], [248], [181305, 38385, 179750], [248], [248], [248], [248], [103123], [248], [248], [248], [248], [248], [103123], [248], [248], [248], [248], [248], [248], [248], [151525], [248], [103123], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [103123], [151525], [248], [248], [248], [248], [177045], [248], [248], [13979], [248], [248], [67605, 42688], [248], [248], [248], [248], [151525], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [151525], [151525], [248], [248], [248], [248], [248], [248], [248], [248], [248], [248], [13979], [248], [248], [248], [248], [248], [181305, 38385, 179750], [78180, 78186, 78190, 78184, 78188, 78187, 78179, 78181, 78189, 78182, 78185], [78182, 78184, 78190, 78185, 78186, 78181, 78187, 78179, 78188, 78189, 78180], [78188, 78180, 78184, 78186, 78190, 78185, 78187, 78181, 78182, 78179, 78189], [78181, 78186, 78187, 78185, 78190, 78184, 78189, 78179, 78182, 78188, 78180], [78187, 78179, 78180, 78189, 78185, 78190, 78188, 78184, 78186, 78181, 78182], [78186, 78182, 78188, 78181, 78189, 78180, 78184, 78185, 78179, 78190, 78187], [78181, 78186, 78189, 78188, 78182, 78187, 78190, 78179, 78180, 78185, 78184], [78181, 78189, 78180, 78190, 78179, 78187, 78186, 78184, 78182, 78185, 78188], [248], [248], [78186, 78181, 78188, 78184, 78180, 78185, 78189, 78190, 78179, 78187, 78182], [78180, 78179, 78187, 78184, 78181, 78188, 78185, 78182, 78186, 78189, 78190], [248], [78182, 78188, 78189, 78190, 78187, 78181, 78184, 78180, 78179, 78185, 78186], [78188, 78187, 78190, 78186, 78189, 78184, 78179, 78180, 78182, 78185, 78181], [248], [78180, 78189, 78187, 78181, 78188, 78185, 78184, 78186, 78182, 78190, 78179], [78188, 78184, 78185, 78180, 78182, 78179, 78187, 78181, 78190, 78186, 78189], [248], [78190, 78180, 78189, 78187, 78184, 78186, 78181, 78182, 78185, 78179, 78188], [248], [78184, 78179, 78185, 78190, 78180, 78186, 78189, 78181, 78187, 78182, 78188], [78180, 78185, 78184, 78187, 78190, 78179, 78182, 78189, 78186, 78188, 78181], [248], [248], [78179, 78184, 78186, 78185, 78190, 78187, 78182, 78180, 78188, 78189, 78181], [248], [78185, 78189, 78190, 78180, 78181, 78179, 78188, 78182, 78184, 78187, 78186], [248], [248], [248], [78179, 78190, 78187, 78181, 78184, 78186, 78188, 78180, 78185, 78189, 78182], [248], [248], [78188, 78187, 78189, 78180, 78179, 78182, 78190, 78186, 78184, 78185, 78181], [85722], [248], [248], [248], [248], [248], [248], [248], [248], [222232], [151525], [248], [248], [222232], [222232], [248], [248], [248], [248], [248], [179750, 38385, 181305], [179750, 181305, 38385], [222232], [248], [85722], [222232], [248], [248], [248], [222232], [85722], [248], [248], [181409], [248], [248], [248], [248], [161473, 177587], [248], [38385, 179750, 181305], [248], [248], [248], [85722], [163192], [248], [248], [248], [248], [248], [248], [248], [181305, 179750, 38385], [85722], [248], [248], [248], [248], [248], [85722], [248], [248], [248], [248], [248], [248], [222232], [248], [248], [92674], [248], [248], [248], [248], [248], [248], [248]]\n",
            "predictions[0]: [('chronic papillomatous dermatitis due to incontinence (disorder)', 'NULL'), ('chronic disease of skin (disorder)', 'chronic generalized exfoliative dermatitis (disorder)'), ('disorder of skin (disorder)', 'pruritus ani (disorder)'), ('chronic papillomatous dermatitis due to urostomy (disorder)', 'NULL'), ('disorder of skin (disorder)', 'exuberant cicatrix (disorder)'), ('disorder of skin (disorder)', 'disorder of perianal skin (disorder)'), ('disorder of skin due to tattoo ink (disorder)', 'NULL'), ('disorder of skin (disorder)', 'perianal dermatitis (disorder)'), ('peristomal skin complication (disorder)', 'irritant contact dermatitis due to urostomy (disorder)'), ('irritant contact dermatitis due to urostomy (disorder)', 'chronic papillomatous dermatitis due to urostomy (disorder)'), ('disorder of skin (disorder)', 'biers spots (disorder)'), ('disorder of skin (disorder)', 'disorder of skin due to tattoo ink (disorder)'), ('disorder of skin (disorder)', 'scar of skin (disorder)'), ('disorder of skin (disorder)', 'atrophy of skin due to drug (disorder)'), ('disorder of skin (disorder)', 'papular dermatitis of pregnancy (disorder)'), ('chronic dermatitis (disorder)', 'chronic generalized exfoliative dermatitis (disorder)'), ('dermatitis (disorder)', 'chronic generalized exfoliative dermatitis (disorder)'), ('disorder of integument (disorder)', 'disorder of perianal skin (disorder)'), ('disorder of anus (disorder)', 'anal skin tag (disorder)'), ('disorder of skin (disorder)', 'perianal skin tags (disorder)'), ('acquired hypermelanotic disorder (disorder)', 'pigmentary incontinence (disorder)'), ('something that is Role group (attribute) some something that Due to (attribute) some Scar of skin (disorder)', 'hypopigmented scar (disorder)'), ('something that is Role group (attribute) some something that Due to (attribute) some Scar of skin (disorder)', 'hypomelanosis due to scarring (disorder)'), ('disorder of skin (disorder)', 'dermal elastolysis (disorder)'), ('disorder of skin (disorder)', 'denuded skin (disorder)'), ('irritant contact dermatitis due to urostomy (disorder)', 'NULL'), ('disorder of integument (disorder)', 'denuded skin (disorder)'), ('irritant contact dermatitis due to incontinence (disorder)', 'NULL'), ('disorder of integument (disorder)', 'disorder of skin due to tattoo ink (disorder)'), ('irritant contact dermatitis due to stoma and/or fistula (disorder)', 'chronic papillomatous dermatitis due to urostomy (disorder)'), ('chronic dermatitis (disorder)', 'NULL'), ('something that is Role group (attribute) some something that Due to (attribute) some Scar of skin (disorder)', 'hypomelanosis due to cryotherapy (disorder)'), ('disorder of skin (disorder)', 'drug-induced anetoderma (disorder)'), ('melanosis (disorder)', 'pigmentary incontinence (disorder)'), ('chronic inflammatory disorder (disorder)', 'chronic generalized exfoliative dermatitis (disorder)'), ('something that is Role group (attribute) some something that Due to (attribute) some Scar of skin (disorder)', 'laser-induced hypopigmentation (disorder)'), ('disorder of skin (disorder)', 'melanocytic nevus of perianal skin (disorder)'), ('peristomal skin complication (disorder)', 'NULL'), ('dermatitis (disorder)', 'NULL'), ('disorder of skin (disorder)', 'NULL'), ('disorder of integument (disorder)', 'NULL'), ('disorder of skin (disorder)', 'hirsuties papillaris vulvae (disorder)'), ('disorder of anus (disorder)', 'NULL'), ('disorder of skin (disorder)', 'infective dermatosis of perianal skin (disorder)'), ('stoma-related chronic papillomatous dermatitis (disorder)', 'NULL'), ('chronic disease of skin (disorder)', 'NULL'), ('irritant contact dermatitis (disorder)', 'NULL'), ('irritant contact dermatitis due to stoma and/or fistula (disorder)', 'NULL'), ('acquired hypermelanotic disorder (disorder)', 'NULL'), ('disorder of pigmentation (disorder)', 'NULL')]\n",
            "pred_idx_tuples[0]: [('402277003', 'SCTID_NULL'), ('128236002', '31922003'), ('95320005', '90446007'), ('402282005', 'SCTID_NULL'), ('95320005', '201308009'), ('95320005', '400037008'), ('444004001', 'SCTID_NULL'), ('95320005', '275448003'), ('236124000', '402281003'), ('402281003', '402282005'), ('95320005', '238811006'), ('95320005', '444004001'), ('95320005', '70582006'), ('95320005', '403669004'), ('95320005', '88697005'), ('34936007', '31922003'), ('182782007', '31922003'), ('128598002', '400037008'), ('32110003', '195469007'), ('95320005', '95546008'), ('238698004', '238702000'), ('[EX.](<609096000> [EX.](<42752001> <70582006>))', '403273006'), ('[EX.](<609096000> [EX.](<42752001> <70582006>))', '402622001'), ('95320005', '402682003'), ('95320005', '418242004'), ('402281003', 'SCTID_NULL'), ('128598002', '418242004'), ('402276007', 'SCTID_NULL'), ('128598002', '444004001'), ('402278008', '402282005'), ('34936007', 'SCTID_NULL'), ('[EX.](<609096000> [EX.](<42752001> <70582006>))', '403692004'), ('95320005', '403671004'), ('414662006', '238702000'), ('128294001', '31922003'), ('[EX.](<609096000> [EX.](<42752001> <70582006>))', '403698000'), ('95320005', '109287001'), ('236124000', 'SCTID_NULL'), ('182782007', 'SCTID_NULL'), ('95320005', 'SCTID_NULL'), ('128598002', 'SCTID_NULL'), ('95320005', '238919009'), ('32110003', 'SCTID_NULL'), ('95320005', '402709000'), ('402283000', 'SCTID_NULL'), ('128236002', 'SCTID_NULL'), ('110979008', 'SCTID_NULL'), ('402278008', 'SCTID_NULL'), ('238698004', 'SCTID_NULL'), ('414032001', 'SCTID_NULL')]\n",
            "scores[0]: [0.03819798, 0.0386993, 0.03334175, 0.037001655, 0.035023954, 0.034950107, 0.033276618, 0.035736773, 0.11250171, 0.063133545, 0.033333957, 0.038447425, 0.033316363, 0.03587184, 0.035627425, 0.040036984, 0.03915773, 0.035321757, 0.034119915, 0.034722693, 0.03454063, 0.034938112, 0.035078015, 0.034089293, 0.033885267, 0.051956274, 0.034581885, 0.056235764, 0.03837175, 0.08956559, 0.031646814, 0.03555616, 0.036688082, 0.03353131, 0.03808971, 0.03601121, 0.035988335, 0.031155264, 0.031003065, 0.03050053, 0.030860243, 0.03500988, 0.03171405, 0.03579004, 0.03540995, 0.03074302, 0.033889968, 0.053677604, 0.032060612, 0.030810567]\n",
            "gold_idx_tuples[0]: [('95320005', '70139002'), ('95320005', '13600006'), ('95320005', '399992009'), ('95320005', '22649008'), ('95320005', '12873001'), ('95320005', '95393002'), ('95320005', '200890005'), ('95320005', '10599009'), ('95320005', '24152003'), ('95320005', '39524009'), ('95320005', '3014005'), ('95320005', '88697005'), ('95320005', '26435006'), ('95320005', '27720003'), ('95320005', '49084001'), ('95320005', '58759008'), ('95320005', '236125004'), ('95320005', '275447008'), ('95320005', '400009001'), ('95320005', '446932005'), ('95320005', '402587003'), ('95320005', '402432002'), ('95320005', '50563003'), ('95320005', '240897005'), ('95320005', '8197001'), ('95320005', '70759006'), ('95320005', '7392002'), ('95320005', '58921009'), ('95320005', '238471005'), ('95320005', '111921000'), ('95320005', '111196000'), ('95320005', '9480005'), ('95320005', '35498005'), ('95320005', '109248001'), ('95320005', '109249009'), ('95320005', '93423006'), ('95320005', '397741002'), ('95320005', '90900004'), ('95320005', '400020001'), ('95320005', '238751002'), ('95320005', '397845003'), ('95320005', '34936007'), ('95320005', '13582007'), ('95320005', '425267003'), ('95320005', '53107009'), ('95320005', '88558006'), ('95320005', '277408007'), ('95320005', '447360007'), ('95320005', '279353003')]\n",
            "label_input/samples-aggregated:276/276\n",
            "INFO:Blink:save predictions to models/crossencoder/mm+Disease-2017AA/original-NIL-top50-pubmedbert\n",
            "INFO:Blink:out_of_KB_thresholds_bi+cross: not applicable\n",
            "\n",
            "+-----------------------+-------------------+------------------+------------+------------+------------+------------+-------------+-------------+----------+----------+------+\n",
            "|        DATASET        | bi-enc acc, micro | rec at 50, micro | ins@1, all | ins@1, any | ins@5, all | ins@5, any | ins@10, all | ins@10, any | ins, all | ins, any | supp |\n",
            "+-----------------------+-------------------+------------------+------------+------------+------------+------------+-------------+-------------+----------+----------+------+\n",
            "| mm-valid-in-KB-sample |        0.0        |       0.05       |   0.012    |   0.012    |   0.0241   |   0.0241   |    0.0361   |    0.0361   |   0.0    |   0.0    | 100  |\n",
            "|      mm-valid-NIL     |       0.0045      |      0.3497      |    0.0     |   0.0061   |   0.0061   |   0.0213   |    0.0182   |    0.0304   |   0.0    |   0.0    | 672  |\n",
            "|      mm-test-NIL      |        0.0        |      0.1482      |    0.0     |    0.0     |   0.0181   |   0.0254   |    0.0181   |    0.029    |   0.0    |   0.0    | 965  |\n",
            "+-----------------------+-------------------+------------------+------------+------------+------------+------------+-------------+-------------+----------+----------+------+\n"
          ]
        }
      ],
      "source": [
        "# using debugging mode and using first 1000 mention-edge pairs for creating data for cross-encoder - this can take around 1 hour with the T4 GPU.\n",
        "# setting debugging variables\n",
        "debugging_mode_training = True\n",
        "# construct shell command with the debugging variables\n",
        "debug_tr_str = str(debugging_mode_training).lower()\n",
        "!bash Edge-Bi-enc+Cross-enc.sh Disease true 50 25 16 {debug_tr_str} true 1000 #> results_log_Disease_edge_biencoder_top50_1k.txt\n",
        "#!bash Edge-Bi-enc+prompt-generation.sh Disease true 10 5 16 > results_log_Disease_edge_biencoder_top10_bs16_final.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Generating prompts for GPT and Llama\n",
        "\n",
        "The script is Edge-Bi-enc+prompt-generation.sh \\$1 \\$2 \\$3 \\$4 \\$5 \\$6\n",
        "\n",
        "Parameters (marked with \\$1 to \\$6):\n",
        "1. subset string: \"Disease\" or \"CPP\"\n",
        "2. whether to use context\n",
        "3. top-k value\n",
        "4. number of edge seeds before edge enrichment into top-k\n",
        "5. biencoder training batch size\n",
        "6. whether to use debugging-trained binencoder model (which trained biencoder with first debug_max_lines, which have default value as 1000)"
      ],
      "metadata": {
        "id": "z5ulGbHMKraN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!bash Edge-Bi-enc+prompt-generation.sh Disease true 10 5 16 {debug_tr_str} #> results_log_Disease_edge_biencoder_top10_bs16_final.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EAhX7n7BLkmd",
        "outputId": "d8ec80d6-6d50-4ecf-e54d-6413cb193295"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Edge-Bi-enc+prompt-generation.sh: line 23: activate: No such file or directory\n",
            "INFO:deeponto:32g maximum memory allocated to JVM.\n",
            "INFO:deeponto:JVM started successfully.\n",
            "2024-08-07 14:56:16.972369: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2024-08-07 14:56:17.005457: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2024-08-07 14:56:17.015490: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2024-08-07 14:56:19.297329: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "INFO:numexpr.utils:NumExpr defaulting to 2 threads.\n",
            "Namespace(silent=False, debug=False, debug_max_lines=10000, data_parallel=True, no_cuda=False, top_k=10, seed=52313, zeshel=False, max_seq_length=256, max_context_length=32, max_cand_length=128, path_to_model='models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/pytorch_model.bin', bert_model='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', pull_from_layer=-1, lowercase=True, context_key='context', use_context=True, out_dim=1, add_linear=False, data_path='data/MedMentions-preprocessed+/Disease/st21pv_syn_attr-all-complexEdge-edges-final', output_path='models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug', use_triplet_loss_bi_enc=False, use_miner_bi_enc=False, use_ori_classification=False, use_NIL_classification=False, lambda_NIL=0.25, use_score_features=False, use_score_pooling=False, use_men_only_score_ft=False, use_extra_features=False, use_BM25=False, aggregating_factor=1, eval_batch_size=8, mode='valid-NIL,test-NIL', save_topk_result=True, encode_batch_size=8, use_synonyms=False, use_NIL_tag=False, use_NIL_desc=False, use_NIL_desc_tag=False, cand_pool_path='preprocessing/saved_cand_ids_snomedct-us-20140901-disease_edges_all-sapbert_re_tr.pt', cand_encode_path='models/SNOMEDCT-US-20140901-Disease_ent_enc_re_tr/SNOMEDCT-US-20140901-Disease_edges_all-sapbert_ent_enc_re_tr.t7', use_NIL_classification_infer=False, add_NIL_to_bi_enc_pred=True, NIL_ent_ind=64076, entity_dict_path='ontologies/SNOMEDCT-US-20140901-Disease-edges-all.jsonl', cand_pool_path_for_BM25=None, save_all_predictions=True, edge_cand_enrich=True, edge_catalogue_fn='ontologies/SNOMEDCT-US-20140901-Disease-edges-all.jsonl', top_k_cand_seed=5, use_leaf_edge_score=False, LEAF_EDGE_SCORE=1000, edge_ranking_by_score=True, use_fix_embs=False, debug_random_sample=False)\n",
            "self.n_gpu: 0\n",
            "self.device in BiEncoderRanker: cpu\n",
            "params[\"lowercase\"]: True\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model built\n",
            "model loaded\n",
            "moving model to cpu\n",
            "model moved to cpu\n",
            "INFO:Blink:Loading pre-generated candidate pool from: \n",
            "INFO:Blink:preprocessing/saved_cand_ids_snomedct-us-20140901-disease_edges_all-sapbert_re_tr.pt\n",
            "INFO:Blink:Loading pre-generated candidate encode path.\n",
            "local_id2wikipedia_id: 237826\n",
            "wikipedia_id2local_id: 237826\n",
            "id2synonyms: 237826\n",
            "100% 672/672 [00:00<00:00, 113200.22it/s]\n",
            "INFO:Blink:Read 672 test samples.\n",
            "num of samples: 672\n",
            "100% 672/672 [00:00<00:00, 941.88it/s]\n",
            "INFO:Blink:====Processed samples (first 10): ====\n",
            "INFO:Blink:Mention tokens : [unused0] inflammatory skin disease [unused1]\n",
            "INFO:Blink:Mention ids : 1 3769 4407 2573 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##393 ##800 ##7\n",
            "INFO:Blink:Concept ID ori ids : 3895 30408 26749 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] pus ##t ##ulosis ( ppp ) is a recalc ##itr ##ant chronic [unused0] inflammatory skin disease [unused1] . data relevant for the medical care of patients with ppp are scarce [SEP]\n",
            "INFO:Blink:Context ids : 2 24093 1031 8071 12 22150 13 1977 43 25470 4250 2031 3724 1 3769 4407 2573 1 18 2230 4494 1958 1920 3423 2859 1927 2132 1956 22150 2032 16937 3\n",
            "INFO:Blink:Contextual tokens : [CLS] pus ##t ##ulosis ( ppp ) is a recalc ##itr ##ant chronic [unused0] inflammatory skin disease [unused1] . data relevant for the medical care of patients with ppp are scarce [SEP]\n",
            "INFO:Blink:Contextual ids : 2 24093 1031 8071 12 22150 13 1977 43 25470 4250 2031 3724 1 3769 4407 2573 1 18 2230 4494 1958 1920 3423 2859 1927 2132 1956 22150 2032 16937 3\n",
            "INFO:Blink:Label tokens : [CLS] disorder of skin ( disorder ) [unused5] histologic type of inflammatory skin disorder ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4770 1927 4407 12 4770 13 1 12434 2601 1927 3769 4407 4770 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 36106 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] ckd [unused1]\n",
            "INFO:Blink:Mention ids : 1 9928 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##90 ##44 ##004\n",
            "INFO:Blink:Concept ID ori ids : 3895 5124 6609 22077 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] catheter - associated inferior vena cava thrombosis patients with chronic kidney disease ( [unused0] ckd [unused1] ) are at increased risk for thrombotic complications . the use of central venous [SEP]\n",
            "INFO:Blink:Context ids : 2 8421 17 2458 8756 22876 23326 11354 2132 1956 3724 5245 2573 12 1 9928 1 13 2032 2019 2502 2565 1958 21039 5021 18 1920 2485 1927 4101 8562 3\n",
            "INFO:Blink:Contextual tokens : [CLS] catheter - associated inferior vena cava thrombosis patients with chronic kidney disease ( [unused0] ckd [unused1] ) are at increased risk for thrombotic complications . the use of central venous [SEP]\n",
            "INFO:Blink:Contextual ids : 2 8421 17 2458 8756 22876 23326 11354 2132 1956 3724 5245 2573 12 1 9928 1 13 2032 2019 2502 2565 1958 21039 5021 18 1920 2485 1927 4101 8562 3\n",
            "INFO:Blink:Label tokens : [CLS] renal impairment ( disorder ) [unused5] chronic kidney disease stage 2 ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4604 6159 12 4770 13 1 3724 5245 2573 3587 22 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 78182 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] chronic kidney disease [unused1]\n",
            "INFO:Blink:Mention ids : 1 3724 5245 2573 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##90 ##44 ##004\n",
            "INFO:Blink:Concept ID ori ids : 3895 5124 6609 22077 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] syndrome secondary to catheter - associated inferior vena cava thrombosis patients with [unused0] chronic kidney disease [unused1] ( ckd ) are at increased risk for thrombotic complications . the use [SEP]\n",
            "INFO:Blink:Context ids : 2 4741 4067 1942 8421 17 2458 8756 22876 23326 11354 2132 1956 1 3724 5245 2573 1 12 9928 13 2032 2019 2502 2565 1958 21039 5021 18 1920 2485 3\n",
            "INFO:Blink:Contextual tokens : [CLS] syndrome secondary to catheter - associated inferior vena cava thrombosis patients with [unused0] chronic kidney disease [unused1] ( ckd ) are at increased risk for thrombotic complications . the use [SEP]\n",
            "INFO:Blink:Contextual ids : 2 4741 4067 1942 8421 17 2458 8756 22876 23326 11354 2132 1956 1 3724 5245 2573 1 12 9928 13 2032 2019 2502 2565 1958 21039 5021 18 1920 2485 3\n",
            "INFO:Blink:Label tokens : [CLS] renal impairment ( disorder ) [unused5] chronic kidney disease stage 1 ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4604 6159 12 4770 13 1 3724 5245 2573 3587 21 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 78181 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] lynch syndrome [unused1]\n",
            "INFO:Blink:Mention ids : 1 27452 4741 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 71 ##63 ##180 ##02\n",
            "INFO:Blink:Concept ID ori ids : 5858 7576 15884 11636 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] genetic background . here , we describe the case of a patient with [unused0] lynch syndrome [unused1] that presented with six synchronous crc ##s . microsatellite instability ( msi ) [SEP]\n",
            "INFO:Blink:Context ids : 2 3299 3872 18 3410 16 2038 5993 1920 3087 1927 43 2774 1956 1 27452 4741 1 1988 3549 1956 3798 17669 8223 1036 18 16069 9557 12 17355 13 3\n",
            "INFO:Blink:Contextual tokens : [CLS] genetic background . here , we describe the case of a patient with [unused0] lynch syndrome [unused1] that presented with six synchronous crc ##s . microsatellite instability ( msi ) [SEP]\n",
            "INFO:Blink:Contextual ids : 2 3299 3872 18 3410 16 2038 5993 1920 3087 1927 43 2774 1956 1 27452 4741 1 1988 3549 1956 3798 17669 8223 1036 18 16069 9557 12 17355 13 3\n",
            "INFO:Blink:Label tokens : [CLS] hereditary cancer - predisposing syndrome ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 16321 2539 17 20908 4741 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 39163 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] vt [unused1]\n",
            "INFO:Blink:Mention ids : 1 12195 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 66 ##24 ##005\n",
            "INFO:Blink:Concept ID ori ids : 4970 4561 24316 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] year after crt had a higher rate of lv reverse remodeling and less [unused0] vt [unused1] / vf episodes . the acute changes of tp ##te after crt have minimal [SEP]\n",
            "INFO:Blink:Context ids : 2 2476 2256 13705 2430 43 2580 2769 1927 6701 5267 8378 1930 2941 1 12195 1 19 15691 9232 18 1920 3939 2840 1927 6213 2057 2256 13705 2162 6343 3\n",
            "INFO:Blink:Contextual tokens : [CLS] year after crt had a higher rate of lv reverse remodeling and less [unused0] vt [unused1] / vf episodes . the acute changes of tp ##te after crt have minimal [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2476 2256 13705 2430 43 2580 2769 1927 6701 5267 8378 1930 2941 1 12195 1 19 15691 9232 18 1920 3939 2840 1927 6213 2057 2256 13705 2162 6343 3\n",
            "INFO:Blink:Label tokens : [CLS] ventricular arrhythmia ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 6493 18346 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 92674 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] wol ##man [unused1]\n",
            "INFO:Blink:Mention ids : 1 12697 3862 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 71 ##59 ##23 ##003\n",
            "INFO:Blink:Concept ID ori ids : 5858 7308 4553 23820 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] ##ho ##ff , gm1 , muc ##opl ##ys ##acchar ##ido ##se ##s , [unused0] wol ##man [unused1] , kra ##bb ##e , metac ##hr ##omatic leuk ##odys ##tr ##oph ##y [SEP]\n",
            "INFO:Blink:Context ids : 2 10647 2040 16 28673 16 3422 3513 2012 7364 15072 4127 1036 16 1 12697 3862 1 16 29995 6936 1021 16 17265 6570 21832 5104 26812 1997 2688 1012 3\n",
            "INFO:Blink:Contextual tokens : [CLS] ##ho ##ff , gm1 , muc ##opl ##ys ##acchar ##ido ##se ##s , [unused0] wol ##man [unused1] , kra ##bb ##e , metac ##hr ##omatic leuk ##odys ##tr ##oph ##y [SEP]\n",
            "INFO:Blink:Contextual ids : 2 10647 2040 16 28673 16 3422 3513 2012 7364 15072 4127 1036 16 1 12697 3862 1 16 29995 6936 1021 16 17265 6570 21832 5104 26812 1997 2688 1012 3\n",
            "INFO:Blink:Label tokens : [CLS] disorder of lipid storage and metabolism ( disorder ) [unused5] wol ##man ' s disease ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4770 1927 4486 7451 1930 4714 12 4770 13 1 12697 3862 11 61 2573 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 81833 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] ckd [unused1]\n",
            "INFO:Blink:Mention ids : 1 9928 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##90 ##44 ##004\n",
            "INFO:Blink:Concept ID ori ids : 3895 5124 6609 22077 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] not to use them this supplement reviews the evidence from clinical trials for the appropriate use of statins in patients with chronic kidney disease ( [unused0] ckd [unused1] ) . [SEP]\n",
            "INFO:Blink:Context ids : 2 2084 1942 2485 3581 2052 3029 8617 1920 3095 2037 2535 3848 1958 1920 4703 2485 1927 15511 1922 2132 1956 3724 5245 2573 12 1 9928 1 13 18 3\n",
            "INFO:Blink:Contextual tokens : [CLS] not to use them this supplement reviews the evidence from clinical trials for the appropriate use of statins in patients with chronic kidney disease ( [unused0] ckd [unused1] ) . [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2084 1942 2485 3581 2052 3029 8617 1920 3095 2037 2535 3848 1958 1920 4703 2485 1927 15511 1922 2132 1956 3724 5245 2573 12 1 9928 1 13 18 3\n",
            "INFO:Blink:Label tokens : [CLS] renal impairment ( disorder ) [unused5] hypertensive heart and chronic kidney disease ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4604 6159 12 4770 13 1 11706 4003 1930 3724 5245 2573 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 78180 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cp ##fe ##s [unused1]\n",
            "INFO:Blink:Mention ids : 1 4244 10140 1036 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##80 ##300 ##04\n",
            "INFO:Blink:Concept ID ori ids : 3895 4476 17450 14743 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] is relevant . however , the efficacy of pir ##fen ##idone in [unused0] cp ##fe ##s [unused1] patients is still not well - known . in this study we propose [SEP]\n",
            "INFO:Blink:Context ids : 2 1977 4494 18 2406 16 1920 4348 1927 13774 14946 15726 1922 1 4244 10140 1036 1 2132 1977 3974 2084 2486 17 3171 18 1922 2052 2161 2038 8040 3\n",
            "INFO:Blink:Contextual tokens : [CLS] is relevant . however , the efficacy of pir ##fen ##idone in [unused0] cp ##fe ##s [unused1] patients is still not well - known . in this study we propose [SEP]\n",
            "INFO:Blink:Contextual ids : 2 1977 4494 18 2406 16 1920 4348 1927 13774 14946 15726 1922 1 4244 10140 1036 1 2132 1977 3974 2084 2486 17 3171 18 1922 2052 2161 2038 8040 3\n",
            "INFO:Blink:Label tokens : [CLS] fibrosis of lung ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 6769 1927 3614 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 29419 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] inflammatory skin disease [unused1]\n",
            "INFO:Blink:Mention ids : 1 3769 4407 2573 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##393 ##800 ##7\n",
            "INFO:Blink:Concept ID ori ids : 3895 30408 26749 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] chronic hand eczema chronic hand eczema ( che ) is a common [unused0] inflammatory skin disease [unused1] that affects approximately 10 % of the population . systemic ali ##tre ##tin [SEP]\n",
            "INFO:Blink:Context ids : 2 3724 3897 26798 3724 3897 26798 12 10003 13 1977 43 3059 1 3769 4407 2573 1 1988 6886 3993 2119 9 1927 1920 2973 18 5510 15356 4265 2268 3\n",
            "INFO:Blink:Contextual tokens : [CLS] chronic hand eczema chronic hand eczema ( che ) is a common [unused0] inflammatory skin disease [unused1] that affects approximately 10 % of the population . systemic ali ##tre ##tin [SEP]\n",
            "INFO:Blink:Contextual ids : 2 3724 3897 26798 3724 3897 26798 12 10003 13 1977 43 3059 1 3769 4407 2573 1 1988 6886 3993 2119 9 1927 1920 2973 18 5510 15356 4265 2268 3\n",
            "INFO:Blink:Label tokens : [CLS] disorder of skin ( disorder ) [unused5] acr ##oderma ##titis ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4770 1927 4407 12 4770 13 1 10428 20532 11648 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 36132 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] hd ##v infection [unused1]\n",
            "INFO:Blink:Mention ids : 1 4515 1020 2909 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##73 ##41 ##005\n",
            "INFO:Blink:Concept ID ori ids : 3895 8966 8099 24316 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] in this study we present a semi - automated method to evaluate [unused0] hd ##v infection [unused1] and replication under the influence of different drugs . we utilized a huh [SEP]\n",
            "INFO:Blink:Context ids : 2 1922 2052 2161 2038 2481 43 8762 17 9210 2417 1942 4323 1 4515 1020 2909 1 1930 5158 2441 1920 4039 1927 2413 4195 18 2038 7079 43 17329 3\n",
            "INFO:Blink:Contextual tokens : [CLS] in this study we present a semi - automated method to evaluate [unused0] hd ##v infection [unused1] and replication under the influence of different drugs . we utilized a huh [SEP]\n",
            "INFO:Blink:Contextual ids : 2 1922 2052 2161 2038 2481 43 8762 17 9210 2417 1942 4323 1 4515 1020 2909 1 1930 5158 2441 1920 4039 1927 2413 4195 18 2038 7079 43 17329 3\n",
            "INFO:Blink:Label tokens : [CLS] disease ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 2573 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 248 is in-KB\n",
            "label_text_vecs: tensor([[    2,  4770,  1927,  ...,     0,     0,     0],\n",
            "        [    2,  4604,  6159,  ...,     0,     0,     0],\n",
            "        [    2,  4604,  6159,  ...,     0,     0,     0],\n",
            "        ...,\n",
            "        [    2, 13027,  6379,  ...,     0,     0,     0],\n",
            "        [    2,  4770,  1927,  ...,     0,     0,     0],\n",
            "        [    2,  6558,  2798,  ...,     0,     0,     0]])\n",
            "INFO:Blink:Getting top 10 predictions.\n",
            "INFO:Blink:Enriched from top 5 predictions.\n",
            "  0% 0/84 [00:00<?, ?it/s]\n",
            "  0% 0/237826 [00:00<?, ?it/s]\u001b[A\n",
            "  7% 17023/237826 [00:00<00:01, 170209.87it/s]\u001b[A\n",
            " 16% 37249/237826 [00:00<00:01, 188848.62it/s]\u001b[A\n",
            " 24% 57214/237826 [00:00<00:00, 193769.04it/s]\u001b[A\n",
            " 32% 76591/237826 [00:00<00:00, 188891.07it/s]\u001b[A\n",
            " 40% 96123/237826 [00:00<00:00, 191166.34it/s]\u001b[A\n",
            " 48% 115255/237826 [00:00<00:00, 185650.36it/s]\u001b[A\n",
            " 56% 133855/237826 [00:00<00:00, 180606.22it/s]\u001b[A\n",
            " 64% 151954/237826 [00:00<00:00, 153127.19it/s]\u001b[A\n",
            " 71% 167919/237826 [00:01<00:00, 139573.71it/s]\u001b[A\n",
            " 77% 182444/237826 [00:01<00:00, 131214.19it/s]\u001b[A\n",
            " 82% 195961/237826 [00:01<00:00, 127615.91it/s]\u001b[A\n",
            " 88% 208966/237826 [00:01<00:00, 122840.25it/s]\u001b[A\n",
            " 93% 221399/237826 [00:01<00:00, 122161.90it/s]\u001b[A\n",
            "100% 237826/237826 [00:01<00:00, 145810.26it/s]\n",
            "list_inds_enriched_w_scores, sorted: [(23866, 206.76457), (167207, 205.19095), (92773, 204.93996), (66908, 204.74101), (165232, 203.7695), (30144, 201.63081), (7136, 199.49182), (66198, 196.47934), (84456, 193.75784), (2241, 191.26074), (109461, 186.37703), (49161, 170.04095)]\n",
            "list_inds_enriched_w_scores, sorted: [(78438, 208.6022), (78508, 208.33272), (78514, 207.50537), (2412, 207.08981), (158301, 206.92142), (15763, 196.92891), (8071, 196.3623), (163236, 195.69127), (15556, 158.9631), (248, 154.94916), (37629, 132.22888), (47716, 129.5606)]\n",
            "list_inds_enriched_w_scores, sorted: [(78508, 214.47847), (30164, 213.24866), (33531, 212.63702), (78214, 212.30768), (78166, 212.18665), (151525, 206.70972), (8071, 205.73003), (14585, 198.0553), (248, 166.05707), (37629, 142.03278), (47716, 139.0034)]\n",
            "list_inds_enriched_w_scores, sorted: [(211243, 220.60956), (211234, 218.68376), (145600, 218.64781), (211269, 218.14905), (9262, 217.90608), (211259, 217.45078), (211284, 217.08365), (39163, 202.26941), (214800, 200.34267), (214801, 195.80377), (4866, 170.5304), (111604, 170.43619), (11205, 166.36736), (162433, 160.86188), (248, 151.08679)]\n",
            "list_inds_enriched_w_scores, sorted: [(210578, 228.96628), (177310, 228.81216), (25350, 228.80786), (2884, 228.5041), (188706, 228.28738), (25370, 226.68211), (210592, 219.94604), (25360, 218.8875), (44028, 217.10764), (14322, 209.55614), (73326, 209.12054), (73327, 207.49663), (248, 197.48116), (6866, 183.32205)]\n",
            "list_inds_enriched_w_scores, sorted: [(197457, 208.28299), (159219, 207.2304), (2828, 205.61026), (77080, 205.2974), (159217, 204.72296), (77095, 200.65538), (170928, 199.0188), (7968, 197.83846), (110460, 196.11337), (197459, 193.73326), (94855, 192.09932), (77110, 191.8438), (221227, 188.79422), (221228, 185.85895), (148461, 184.79489), (7672, 177.28723), (151319, 169.28316), (3413, 168.17653), (4866, 167.65912), (3414, 167.32848), (16784, 161.05058), (15556, 156.38492), (11205, 156.31271)]\n",
            "list_inds_enriched_w_scores, sorted: [(67256, 204.69429), (9787, 202.51086), (197829, 201.22067), (197830, 200.78767), (197831, 200.4477), (62730, 161.21158), (248, 160.58682), (1080, 154.7016), (9788, 154.28079), (9836, 147.5463), (170497, 133.10149), (214295, 129.54185)]\n",
            "list_inds_enriched_w_scores, sorted: [(65481, 173.05264), (127275, 171.55336), (117631, 167.91496), (117638, 167.3186), (132404, 165.4198), (117634, 156.85583), (248, 142.30287), (9836, 137.94113), (65471, 137.79837), (109276, 134.99107), (127257, 125.903), (65458, 113.07674), (127110, 107.531006)]\n",
            "  1% 1/84 [00:03<04:41,  3.39s/it]list_inds_enriched_w_scores, sorted: [(108466, 204.71461), (229206, 203.8631), (163756, 203.3562), (228205, 203.15529), (179827, 202.3775), (228207, 200.07336), (163757, 199.13416), (163758, 198.66634), (83093, 196.45764), (83110, 195.23828), (163778, 193.25613), (49628, 192.54118), (228219, 191.09174), (229211, 185.15768), (7023, 181.7407), (39246, 181.16281), (7136, 179.53181), (39431, 177.9302), (66198, 176.76935), (44366, 169.08118), (42257, 148.46692)]\n",
            "list_inds_enriched_w_scores, sorted: [(44602, 199.79741), (139447, 198.98111), (143817, 198.81363), (108871, 197.37558), (54982, 197.29295), (6836, 190.52637), (108880, 190.26703), (44631, 187.43683), (87684, 160.94113), (11263, 156.12576), (248, 150.88191)]\n",
            "100% 84/84 [01:09<00:00,  1.21it/s]\n",
            "nn_data['labels']: tensor([-1, -1, -1,  7, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1,  6, -1, -1,  5, -1, -1,  0, -1, -1, -1, -1, -1, -1,  8,\n",
            "        -1, -1, -1, -1, -1, -1,  5, -1, -1,  7, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  5, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  7, -1, -1, -1,\n",
            "         5, -1, -1, -1, -1, -1, -1, -1, -1, -1,  7, -1, -1,  9, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1,  6, -1, -1, -1, -1, -1,  4, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  3, -1,  7,\n",
            "        -1,  6, -1,  3, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  1, -1,\n",
            "         5, -1,  2, -1, -1, -1, -1, -1, -1, -1, -1,  6,  9, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1,  4,  2, -1, -1, -1, -1,  9, -1, -1, -1, -1,  9, -1,\n",
            "         9, -1, -1, -1, -1, -1, -1, -1, -1, -1,  5, -1, -1, -1,  9, -1, -1, -1,\n",
            "        -1, -1,  9, -1,  4, -1, -1,  4, -1, -1, -1, -1, -1,  7, -1, -1, -1, -1,\n",
            "         9, -1, -1, -1, -1, -1, -1, -1, -1,  8, -1, -1,  7, -1, -1, -1, -1,  0,\n",
            "        -1,  6, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  6, -1,  4, -1, -1,\n",
            "        -1, -1, -1,  2, -1, -1, -1, -1, -1, -1, -1, -1,  7, -1,  7, -1, -1,  6,\n",
            "         5, -1, -1, -1, -1, -1, -1,  5, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1,  6, -1, -1, -1, -1, -1, -1, -1, -1,  9,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  5, -1, -1,  3, -1, -1, -1, -1,\n",
            "         0, -1, -1, -1, -1, -1, -1,  5, -1, -1, -1, -1, -1, -1, -1, -1, -1,  9,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  5, -1,  7, -1, -1, -1,\n",
            "        -1, -1, -1,  2, -1, -1, -1,  8, -1, -1, -1, -1, -1, -1, -1, -1,  8, -1,\n",
            "         6, -1,  9, -1,  5, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1,  7, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1,  3, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  6, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "         6, -1, -1,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1,  3, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  5, -1, -1,  7,\n",
            "        -1, -1, -1, -1, -1, -1, -1,  9,  4, -1, -1, -1,  5, -1, -1,  8, -1,  5,\n",
            "        -1,  8, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  6, -1,\n",
            "        -1,  6,  7, -1, -1, -1, -1, -1,  6, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1,  9, -1, -1, -1, -1,  7, -1, -1,  5,  7, -1, -1, -1, -1,\n",
            "        -1, -1, -1,  6, -1,  9,  7, -1,  3,  6, -1,  8, -1, -1, -1, -1, -1, -1,\n",
            "         2, -1, -1, -1, -1, -1,  9, -1, -1, -1, -1, -1, -1,  7, -1, -1, -1, -1,\n",
            "        -1,  5, -1, -1, -1,  7])\n",
            "INFO:Blink:num of nn_data: 672\n",
            "INFO:Blink:biencoder recall@k: 0.14 (95/672)\n",
            "data saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/valid-NIL.t7\n",
            "self.n_gpu: 0\n",
            "self.device in BiEncoderRanker: cpu\n",
            "params[\"lowercase\"]: True\n",
            "model tokenizer: BertTokenizerFast(name_or_path='cambridgeltl/SapBERT-from-PubMedBERT-fulltext', vocab_size=30522, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, clean_up_tokenization_spaces=True)\n",
            "model built\n",
            "model loaded\n",
            "moving model to cpu\n",
            "model moved to cpu\n",
            "INFO:Blink:Loading pre-generated candidate pool from: \n",
            "INFO:Blink:preprocessing/saved_cand_ids_snomedct-us-20140901-disease_edges_all-sapbert_re_tr.pt\n",
            "INFO:Blink:Loading pre-generated candidate encode path.\n",
            "local_id2wikipedia_id: 237826\n",
            "wikipedia_id2local_id: 237826\n",
            "id2synonyms: 237826\n",
            "100% 965/965 [00:00<00:00, 114829.31it/s]\n",
            "INFO:Blink:Read 965 test samples.\n",
            "num of samples: 965\n",
            "100% 965/965 [00:01<00:00, 883.52it/s]\n",
            "INFO:Blink:====Processed samples (first 10): ====\n",
            "INFO:Blink:Mention tokens : [unused0] dermatitis [unused1]\n",
            "INFO:Blink:Mention ids : 1 18006 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##393 ##800 ##7\n",
            "INFO:Blink:Concept ID ori ids : 3895 30408 26749 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] incontinence - associated dermatitis : reducing adverse events incontinence - associated [unused0] dermatitis [unused1] ( iad ) is a common problem in patients with faecal and / or urinary incontinence [SEP]\n",
            "INFO:Blink:Context ids : 2 16783 17 2458 18006 30 5526 5246 3916 16783 17 2458 1 18006 1 12 26842 13 1977 43 3059 4915 1922 2132 1956 15380 1930 19 2014 7075 16783 3\n",
            "INFO:Blink:Contextual tokens : [CLS] incontinence - associated dermatitis : reducing adverse events incontinence - associated [unused0] dermatitis [unused1] ( iad ) is a common problem in patients with faecal and / or urinary incontinence [SEP]\n",
            "INFO:Blink:Contextual ids : 2 16783 17 2458 18006 30 5526 5246 3916 16783 17 2458 1 18006 1 12 26842 13 1977 43 3059 4915 1922 2132 1956 15380 1930 19 2014 7075 16783 3\n",
            "INFO:Blink:Label tokens : [CLS] disorder of skin ( disorder ) [unused5] har ##ara ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4770 1927 4407 12 4770 13 1 4303 9041 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 36144 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] iad [unused1]\n",
            "INFO:Blink:Mention ids : 1 26842 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##393 ##800 ##7\n",
            "INFO:Blink:Concept ID ori ids : 3895 30408 26749 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] penetrate the skin , increasing the risk of secondary infection . however , [unused0] iad [unused1] can be prevented and healed with timely and appropriate skin clean ##sin ##g and [SEP]\n",
            "INFO:Blink:Context ids : 2 21224 1920 4407 16 3605 1920 2565 1927 4067 2909 18 2406 16 1 26842 1 2112 1998 8715 1930 25515 1956 14918 1930 4703 4407 8901 6777 1029 1930 3\n",
            "INFO:Blink:Contextual tokens : [CLS] penetrate the skin , increasing the risk of secondary infection . however , [unused0] iad [unused1] can be prevented and healed with timely and appropriate skin clean ##sin ##g and [SEP]\n",
            "INFO:Blink:Contextual ids : 2 21224 1920 4407 16 3605 1920 2565 1927 4067 2909 18 2406 16 1 26842 1 2112 1998 8715 1930 25515 1956 14918 1930 4703 4407 8901 6777 1029 1930 3\n",
            "INFO:Blink:Label tokens : [CLS] disorder of skin ( disorder ) [unused5] perivascular dermatitis ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4770 1927 4407 12 4770 13 1 19946 18006 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 36143 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] chronic kidney disease [unused1]\n",
            "INFO:Blink:Mention ids : 1 3724 5245 2573 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##90 ##44 ##004\n",
            "INFO:Blink:Concept ID ori ids : 3895 5124 6609 22077 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] to determine the relationship between obstructive sleep apnea ( osa ) and [unused0] chronic kidney disease [unused1] ( ckd ) . previous population studies of the association are sparse , [SEP]\n",
            "INFO:Blink:Context ids : 2 1942 3415 1920 3303 2192 12234 5006 17578 12 15052 13 1930 1 3724 5245 2573 1 12 9928 13 18 2555 2973 2351 1927 1920 3279 2032 13766 16 3\n",
            "INFO:Blink:Contextual tokens : [CLS] to determine the relationship between obstructive sleep apnea ( osa ) and [unused0] chronic kidney disease [unused1] ( ckd ) . previous population studies of the association are sparse , [SEP]\n",
            "INFO:Blink:Contextual ids : 2 1942 3415 1920 3303 2192 12234 5006 17578 12 15052 13 1930 1 3724 5245 2573 1 12 9928 13 18 2555 2973 2351 1927 1920 3279 2032 13766 16 3\n",
            "INFO:Blink:Label tokens : [CLS] renal impairment ( disorder ) [unused5] chronic kidney disease stage 4 ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4604 6159 12 4770 13 1 3724 5245 2573 3587 24 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 78190 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] ckd [unused1]\n",
            "INFO:Blink:Mention ids : 1 9928 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##90 ##44 ##004\n",
            "INFO:Blink:Concept ID ori ids : 3895 5124 6609 22077 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] ##phosphate ##mia is a major determinant of mortality in chronic kidney disease ( [unused0] ckd [unused1] ) . vascular calcification is driven by aldosterone - sensitive osteogenic transformation of vascular [SEP]\n",
            "INFO:Blink:Context ids : 2 25324 6026 1977 43 3044 12483 1927 3885 1922 3724 5245 2573 12 1 9928 1 13 18 4541 14640 1977 6446 2007 16944 17 4532 15035 7002 1927 4541 3\n",
            "INFO:Blink:Contextual tokens : [CLS] ##phosphate ##mia is a major determinant of mortality in chronic kidney disease ( [unused0] ckd [unused1] ) . vascular calcification is driven by aldosterone - sensitive osteogenic transformation of vascular [SEP]\n",
            "INFO:Blink:Contextual ids : 2 25324 6026 1977 43 3044 12483 1927 3885 1922 3724 5245 2573 12 1 9928 1 13 18 4541 14640 1977 6446 2007 16944 17 4532 15035 7002 1927 4541 3\n",
            "INFO:Blink:Label tokens : [CLS] renal impairment ( disorder ) [unused5] chronic renal impairment associated with type ii diabetes mellitus ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4604 6159 12 4770 13 1 3724 4604 6159 2458 1956 2601 2890 4032 9695 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 78179 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] head and neck squamous cell carcinoma [unused1]\n",
            "INFO:Blink:Mention ids : 1 4469 1930 7467 10364 2024 5373 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 71 ##66 ##59 ##002\n",
            "INFO:Blink:Concept ID ori ids : 5858 7283 7308 22536 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] reaction ( rt - pcr ) in 22 patients with primary [unused0] head and neck squamous cell carcinoma [unused1] ( hnscc ) . statistically significant associations were found between herg [SEP]\n",
            "INFO:Blink:Context ids : 2 3227 12 4341 17 3343 13 1922 2950 2132 1956 3065 1 4469 1930 7467 10364 2024 5373 1 12 16499 13 18 4388 2228 5253 1985 2435 2192 25411 3\n",
            "INFO:Blink:Contextual tokens : [CLS] reaction ( rt - pcr ) in 22 patients with primary [unused0] head and neck squamous cell carcinoma [unused1] ( hnscc ) . statistically significant associations were found between herg [SEP]\n",
            "INFO:Blink:Contextual ids : 2 3227 12 4341 17 3343 13 1922 2950 2132 1956 3065 1 4469 1930 7467 10364 2024 5373 1 12 16499 13 18 4388 2228 5253 1985 2435 2192 25411 3\n",
            "INFO:Blink:Label tokens : [CLS] disease ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 2573 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 248 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] iad [unused1]\n",
            "INFO:Blink:Mention ids : 1 26842 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##393 ##800 ##7\n",
            "INFO:Blink:Concept ID ori ids : 3895 30408 26749 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] ##mann incontinence pads that have been developed to absorb the fluids that cause [unused0] iad [unused1] and maintain the skin ' s acidic ph . the acidic ph of the [SEP]\n",
            "INFO:Blink:Context ids : 2 8890 16783 24258 1988 2162 2252 3530 1942 7322 1920 13315 1988 3936 1 26842 1 1930 6698 1920 4407 11 61 9931 2087 18 1920 9931 2087 1927 1920 3\n",
            "INFO:Blink:Contextual tokens : [CLS] ##mann incontinence pads that have been developed to absorb the fluids that cause [unused0] iad [unused1] and maintain the skin ' s acidic ph . the acidic ph of the [SEP]\n",
            "INFO:Blink:Contextual ids : 2 8890 16783 24258 1988 2162 2252 3530 1942 7322 1920 13315 1988 3936 1 26842 1 1930 6698 1920 4407 11 61 9931 2087 18 1920 9931 2087 1927 1920 3\n",
            "INFO:Blink:Label tokens : [CLS] disorder of skin ( disorder ) [unused5] py ##oderma ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4770 1927 4407 12 4770 13 1 6085 20532 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 36115 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] ckd [unused1]\n",
            "INFO:Blink:Mention ids : 1 9928 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##90 ##44 ##004\n",
            "INFO:Blink:Concept ID ori ids : 3895 5124 6609 22077 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] relationship between obstructive sleep apnea ( osa ) and chronic kidney disease ( [unused0] ckd [unused1] ) . previous population studies of the association are sparse , conflicting and confined [SEP]\n",
            "INFO:Blink:Context ids : 2 3303 2192 12234 5006 17578 12 15052 13 1930 3724 5245 2573 12 1 9928 1 13 18 2555 2973 2351 1927 1920 3279 2032 13766 16 13784 1930 13184 3\n",
            "INFO:Blink:Contextual tokens : [CLS] relationship between obstructive sleep apnea ( osa ) and chronic kidney disease ( [unused0] ckd [unused1] ) . previous population studies of the association are sparse , conflicting and confined [SEP]\n",
            "INFO:Blink:Contextual ids : 2 3303 2192 12234 5006 17578 12 15052 13 1930 3724 5245 2573 12 1 9928 1 13 18 2555 2973 2351 1927 1920 3279 2032 13766 16 13784 1930 13184 3\n",
            "INFO:Blink:Label tokens : [CLS] renal impairment ( disorder ) [unused5] chronic kidney disease stage 2 ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4604 6159 12 4770 13 1 3724 5245 2573 3587 22 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 78182 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] cervical spondyl ##osis [unused1]\n",
            "INFO:Blink:Mention ids : 1 6775 20213 2475 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 387 ##800 ##004\n",
            "INFO:Blink:Concept ID ori ids : 28242 26749 22077 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] between multiple - level and single - level cervical disk replacement for [unused0] cervical spondyl ##osis [unused1] : a systematic review and meta - analysis this is a systematic review [SEP]\n",
            "INFO:Blink:Context ids : 2 2192 3137 17 2237 1930 2957 17 2237 6775 15598 7619 1958 1 6775 20213 2475 1 30 43 5614 3043 1930 5880 17 2333 2052 1977 43 5614 3043 3\n",
            "INFO:Blink:Contextual tokens : [CLS] between multiple - level and single - level cervical disk replacement for [unused0] cervical spondyl ##osis [unused1] : a systematic review and meta - analysis this is a systematic review [SEP]\n",
            "INFO:Blink:Contextual ids : 2 2192 3137 17 2237 1930 2957 17 2237 6775 15598 7619 1958 1 6775 20213 2475 1 30 43 5614 3043 1930 5880 17 2333 2052 1977 43 5614 3043 3\n",
            "INFO:Blink:Label tokens : [CLS] disease ( disorder ) [unused5] null [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 2573 12 4770 13 1 6764 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 248 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] iad [unused1]\n",
            "INFO:Blink:Mention ids : 1 26842 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##393 ##800 ##7\n",
            "INFO:Blink:Concept ID ori ids : 3895 30408 26749 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] penetrate the skin , increasing the risk of secondary infection . however , [unused0] iad [unused1] can be prevented and healed with timely and appropriate skin clean ##sin ##g and [SEP]\n",
            "INFO:Blink:Context ids : 2 21224 1920 4407 16 3605 1920 2565 1927 4067 2909 18 2406 16 1 26842 1 2112 1998 8715 1930 25515 1956 14918 1930 4703 4407 8901 6777 1029 1930 3\n",
            "INFO:Blink:Contextual tokens : [CLS] penetrate the skin , increasing the risk of secondary infection . however , [unused0] iad [unused1] can be prevented and healed with timely and appropriate skin clean ##sin ##g and [SEP]\n",
            "INFO:Blink:Contextual ids : 2 21224 1920 4407 16 3605 1920 2565 1927 4067 2909 18 2406 16 1 26842 1 2112 1998 8715 1930 25515 1956 14918 1930 4703 4407 8901 6777 1029 1930 3\n",
            "INFO:Blink:Label tokens : [CLS] disorder of skin ( disorder ) [unused5] lichen ##oid dermatitis ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4770 1927 4407 12 4770 13 1 28088 2642 18006 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 36137 is in-KB\n",
            "INFO:Blink:Mention tokens : [unused0] iad [unused1]\n",
            "INFO:Blink:Mention ids : 1 26842 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Concept ID ori tokens : 70 ##393 ##800 ##7\n",
            "INFO:Blink:Concept ID ori ids : 3895 30408 26749 1027 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:(Context) tokens : [CLS] penetrate the skin , increasing the risk of secondary infection . however , [unused0] iad [unused1] can be prevented and healed with timely and appropriate skin clean ##sin ##g and [SEP]\n",
            "INFO:Blink:Context ids : 2 21224 1920 4407 16 3605 1920 2565 1927 4067 2909 18 2406 16 1 26842 1 2112 1998 8715 1930 25515 1956 14918 1930 4703 4407 8901 6777 1029 1930 3\n",
            "INFO:Blink:Contextual tokens : [CLS] penetrate the skin , increasing the risk of secondary infection . however , [unused0] iad [unused1] can be prevented and healed with timely and appropriate skin clean ##sin ##g and [SEP]\n",
            "INFO:Blink:Contextual ids : 2 21224 1920 4407 16 3605 1920 2565 1927 4067 2909 18 2406 16 1 26842 1 2112 1998 8715 1930 25515 1956 14918 1930 4703 4407 8901 6777 1029 1930 3\n",
            "INFO:Blink:Label tokens : [CLS] disorder of skin ( disorder ) [unused5] sta ##sis dermatitis ( disorder ) [unused6] [unused2] [SEP]\n",
            "INFO:Blink:Label ids : 2 4770 1927 4407 12 4770 13 1 23891 8362 18006 12 4770 13 1 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:Blink:Label_id : 36128 is in-KB\n",
            "label_text_vecs: tensor([[   2, 4770, 1927,  ...,    0,    0,    0],\n",
            "        [   2, 4770, 1927,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        ...,\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0],\n",
            "        [   2, 4604, 6159,  ...,    0,    0,    0]])\n",
            "INFO:Blink:Getting top 10 predictions.\n",
            "INFO:Blink:Enriched from top 5 predictions.\n",
            "  0% 0/121 [00:00<?, ?it/s]\n",
            "  0% 0/237826 [00:00<?, ?it/s]\u001b[A\n",
            "  7% 17453/237826 [00:00<00:01, 174521.01it/s]\u001b[A\n",
            " 15% 35667/237826 [00:00<00:01, 178992.73it/s]\u001b[A\n",
            " 23% 53842/237826 [00:00<00:01, 180245.51it/s]\u001b[A\n",
            " 30% 71867/237826 [00:00<00:00, 177371.58it/s]\u001b[A\n",
            " 38% 89642/237826 [00:00<00:00, 177500.66it/s]\u001b[A\n",
            " 46% 109390/237826 [00:00<00:00, 184242.37it/s]\u001b[A\n",
            " 54% 129523/237826 [00:00<00:00, 189795.70it/s]\u001b[A\n",
            " 63% 148754/237826 [00:00<00:00, 190587.58it/s]\u001b[A\n",
            " 71% 167819/237826 [00:00<00:00, 190462.63it/s]\u001b[A\n",
            " 79% 186870/237826 [00:01<00:00, 187508.21it/s]\u001b[A\n",
            " 86% 205633/237826 [00:01<00:00, 185685.95it/s]\u001b[A\n",
            "100% 237826/237826 [00:01<00:00, 184437.19it/s]\n",
            "list_inds_enriched_w_scores, sorted: [(163910, 211.3115), (148993, 200.78502), (110542, 200.51764), (163934, 200.38452), (49848, 200.04936), (163935, 196.99533), (163911, 196.34363), (7136, 184.27718), (66198, 182.3548), (163936, 179.09445), (76, 178.14688), (163926, 171.97125), (49161, 156.92188), (25917, 136.28648), (248, 133.61766)]\n",
            "list_inds_enriched_w_scores, sorted: [(3301, 218.67715), (47321, 218.21251), (82544, 218.04367), (47309, 218.02255), (140291, 217.79596), (11263, 204.70457), (7119, 204.228), (47328, 199.6994), (47324, 199.30563), (47329, 198.09352), (47325, 194.40196), (49161, 193.37038), (47320, 188.79886), (47326, 187.2525), (47327, 182.0409), (47323, 174.70824), (47322, 172.37381), (248, 167.63428), (34779, 149.2564)]\n",
            "list_inds_enriched_w_scores, sorted: [(22012, 216.7738), (78486, 216.12253), (30153, 215.05133), (22014, 214.85704), (22017, 214.15579), (22007, 214.12213), (30157, 210.34064), (34158, 209.21158), (8071, 198.41986), (37629, 156.95111)]\n",
            "list_inds_enriched_w_scores, sorted: [(78438, 215.00961), (78233, 214.16385), (2403, 213.4259), (78237, 212.38547), (46642, 211.89621), (78439, 210.97916), (2413, 210.59543), (78440, 209.48117), (2423, 208.75218), (151447, 206.97536), (15763, 204.81015), (8071, 195.77646), (78441, 194.2979), (15556, 178.00874), (248, 160.7463), (47716, 139.64728), (37629, 133.15256)]\n",
            "list_inds_enriched_w_scores, sorted: [(98843, 195.65555), (98837, 193.75401), (116333, 192.53204), (98849, 192.38632), (106310, 191.93756), (5424, 185.58649), (164896, 168.61517), (3645, 166.71454), (21186, 159.83295), (19146, 157.81522)]\n",
            "list_inds_enriched_w_scores, sorted: [(83688, 192.03006), (75935, 184.51917), (143550, 179.2757), (213038, 178.7085), (213036, 176.74908), (8112, 172.86996), (177393, 164.6914), (32179, 156.9043), (53455, 154.6343), (66505, 147.16043), (75934, 135.15009), (10622, 135.13208), (75940, 127.220566)]\n",
            "list_inds_enriched_w_scores, sorted: [(22012, 212.23628), (78486, 212.03204), (78438, 211.43063), (2403, 211.21977), (2413, 210.87813), (78439, 208.915), (78440, 208.55797), (2423, 204.2803), (30157, 204.199), (15763, 200.23383), (8071, 193.46304), (78441, 183.94415), (15556, 166.79205), (248, 148.62411), (37629, 148.04204), (47716, 144.3245)]\n",
            "list_inds_enriched_w_scores, sorted: [(157925, 224.06534), (221057, 222.87988), (208950, 222.52197), (157924, 222.46393), (157926, 222.25423), (222026, 220.65265), (42204, 211.66571), (180022, 207.89801), (51161, 207.24747), (25064, 187.16908)]\n",
            "  1% 1/121 [00:02<04:14,  2.12s/it]list_inds_enriched_w_scores, sorted: [(3301, 218.67715), (47321, 218.21251), (82544, 218.04367), (47309, 218.02255), (140291, 217.79596), (11263, 204.70457), (7119, 204.228), (47328, 199.6994), (47324, 199.30563), (47329, 198.09352), (47325, 194.40196), (49161, 193.37038), (47320, 188.79886), (47326, 187.2525), (47327, 182.0409), (47323, 174.70824), (47322, 172.37381), (248, 167.63428), (34779, 149.2564)]\n",
            "list_inds_enriched_w_scores, sorted: [(3301, 218.67715), (47321, 218.21251), (82544, 218.04367), (47309, 218.02255), (140291, 217.79596), (11263, 204.70457), (7119, 204.228), (47328, 199.6994), (47324, 199.30563), (47329, 198.09352), (47325, 194.40196), (49161, 193.37038), (47320, 188.79886), (47326, 187.2525), (47327, 182.0409), (47323, 174.70824), (47322, 172.37381), (248, 167.63428), (34779, 149.2564)]\n",
            "100% 121/121 [01:38<00:00,  1.23it/s]\n",
            "nn_data['labels']: tensor([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1,  8, -1, -1, -1, -1, -1, -1, -1, -1,  5, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1,  9, -1, -1, -1,  1, -1,  9, -1,  8, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1,  9, -1, -1, -1, -1, -1, -1, -1, -1,  7, -1,  1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1,  8, -1,  7, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "         7, -1,  6, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1,  8, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1,  4, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  7, -1,  8, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  7, -1, -1, -1, -1, -1,\n",
            "         9, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  7, -1, -1,  5, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1,  4, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  8, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1,  7, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  9,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1,  7, -1, -1, -1, -1, -1, -1, -1, -1,  8, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  8, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1,  5, -1, -1, -1, -1,  9, -1, -1, -1, -1, -1, -1, -1,\n",
            "         7, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1,  6,  8, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "         7, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1,  9, -1, -1, -1, -1,  9, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1,  7, -1, -1, -1, -1,  2, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "         8, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  9, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1,  9, -1, -1, -1, -1, -1,  7, -1, -1, -1, -1,\n",
            "         9, -1, -1, -1, -1, -1, -1, -1, -1, -1,  9, -1, -1, -1,  6, -1, -1, -1,\n",
            "         9, -1, -1, -1,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  1, -1, -1, -1, -1,\n",
            "        -1, -1, -1,  9, -1,  8, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1,  9, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  9, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,  7,\n",
            "        -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1])\n",
            "INFO:Blink:num of nn_data: 965\n",
            "INFO:Blink:biencoder recall@k: 0.06 (59/965)\n",
            "data saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/test-NIL.t7\n",
            "INFO:deeponto:32g maximum memory allocated to JVM.\n",
            "INFO:deeponto:JVM started successfully.\n",
            "2024-08-07 14:59:42.180147: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2024-08-07 14:59:42.200838: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2024-08-07 14:59:42.207180: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2024-08-07 14:59:43.458837: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "INFO:pytorch_transformers.modeling_bert:Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex .\n",
            "INFO:pytorch_transformers.modeling_xlnet:Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex .\n",
            "Namespace(data_path='models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates', original_data_path='data/MedMentions-preprocessed+/Disease/st21pv_syn_attr-all-complexEdge-edges-final', data_split='test', data_splits='valid-NIL,test-NIL', ontology_fn='ontologies/SNOMEDCT-US-20140901-Disease-final.owl', iri_prefix='http://snomed.info/id/', edge_catalogue_fn='ontologies/SNOMEDCT-US-20140901-Disease-edges-all.jsonl', top_k_filtering=10, filter_by_degree=False, gen_prompts=True, eval_leaf_and_non_leaf_results=True, measure_wp=False)\n",
            "fname: models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/valid-NIL.t7\n",
            "100% 237826/237826 [00:03<00:00, 69017.72it/s] \n",
            "edge catalogue loaded\n",
            "dict_ind_edge_json_info: 237826\n",
            "len(doc): 672\n",
            "len(edge_inds): 672\n",
            "100% 672/672 [00:00<00:00, 5456.96it/s]\n",
            "all edges results:\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_1: 0.00911854103343465 r_at_1: 0.004464285714285714\n",
            "r_at_1_any: 0.00911854103343465\n",
            "r_at_1_all: 0.0060790273556231\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 9 tp-any-mentions: 9 tp-all-mentions: 5 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_3: 0.00911854103343465 r_at_3: 0.013392857142857142\n",
            "r_at_3_any: 0.02735562310030395\n",
            "r_at_3_all: 0.015197568389057751\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 21 tp-any-mentions: 21 tp-all-mentions: 10 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_5: 0.01276595744680851 r_at_5: 0.03125\n",
            "r_at_5_any: 0.06382978723404255\n",
            "r_at_5_all: 0.030395136778115502\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 95 tp-any-mentions: 90 tp-all-mentions: 37 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_10: 0.028875379939209727 r_at_10: 0.14136904761904762\n",
            "r_at_10_any: 0.2735562310030395\n",
            "r_at_10_all: 0.11246200607902736\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 95 tp-any-mentions: 90 tp-all-mentions: 37 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_20: 0.014437689969604863 r_at_20: 0.14136904761904762\n",
            "r_at_20_any: 0.2735562310030395\n",
            "r_at_20_all: 0.11246200607902736\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 95 tp-any-mentions: 90 tp-all-mentions: 37 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_50: 0.005775075987841946 r_at_50: 0.14136904761904762\n",
            "r_at_50_any: 0.2735562310030395\n",
            "r_at_50_all: 0.11246200607902736\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 95 tp-any-mentions: 90 tp-all-mentions: 37 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_100: 0.002887537993920973 r_at_100: 0.14136904761904762\n",
            "r_at_100_any: 0.2735562310030395\n",
            "r_at_100_all: 0.11246200607902736\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 95 tp-any-mentions: 90 tp-all-mentions: 37 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_150: 0.0019250253292806485 r_at_150: 0.14136904761904762\n",
            "r_at_150_any: 0.2735562310030395\n",
            "r_at_150_all: 0.11246200607902736\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 95 tp-any-mentions: 90 tp-all-mentions: 37 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_200: 0.0014437689969604864 r_at_200: 0.14136904761904762\n",
            "r_at_200_any: 0.2735562310030395\n",
            "r_at_200_all: 0.11246200607902736\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 95 tp-any-mentions: 90 tp-all-mentions: 37 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_250: 0.001155015197568389 r_at_250: 0.14136904761904762\n",
            "r_at_250_any: 0.2735562310030395\n",
            "r_at_250_all: 0.11246200607902736\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 95 tp-any-mentions: 90 tp-all-mentions: 37 num_mentions: 329 num_gold_edges: 672\n",
            "p_at_300: 0.0009625126646403242 r_at_300: 0.14136904761904762\n",
            "r_at_300_any: 0.2735562310030395\n",
            "r_at_300_all: 0.11246200607902736\n",
            "leaf edges results:\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 2 tp-any-mentions: 2 tp-all-mentions: 1 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_1: 0.006802721088435374 r_at_1: 0.005154639175257732\n",
            "r_at_1_any: 0.006802721088435374\n",
            "r_at_1_all: 0.003401360544217687\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 7 tp-any-mentions: 7 tp-all-mentions: 4 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_3: 0.007936507936507936 r_at_3: 0.01804123711340206\n",
            "r_at_3_any: 0.023809523809523808\n",
            "r_at_3_all: 0.013605442176870748\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 15 tp-any-mentions: 15 tp-all-mentions: 8 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_5: 0.010204081632653062 r_at_5: 0.03865979381443299\n",
            "r_at_5_any: 0.05102040816326531\n",
            "r_at_5_all: 0.027210884353741496\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 83 tp-any-mentions: 79 tp-all-mentions: 35 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_10: 0.0282312925170068 r_at_10: 0.21391752577319587\n",
            "r_at_10_any: 0.2687074829931973\n",
            "r_at_10_all: 0.11904761904761904\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 83 tp-any-mentions: 79 tp-all-mentions: 35 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_20: 0.0141156462585034 r_at_20: 0.21391752577319587\n",
            "r_at_20_any: 0.2687074829931973\n",
            "r_at_20_all: 0.11904761904761904\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 83 tp-any-mentions: 79 tp-all-mentions: 35 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_50: 0.00564625850340136 r_at_50: 0.21391752577319587\n",
            "r_at_50_any: 0.2687074829931973\n",
            "r_at_50_all: 0.11904761904761904\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 83 tp-any-mentions: 79 tp-all-mentions: 35 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_100: 0.00282312925170068 r_at_100: 0.21391752577319587\n",
            "r_at_100_any: 0.2687074829931973\n",
            "r_at_100_all: 0.11904761904761904\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 83 tp-any-mentions: 79 tp-all-mentions: 35 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_150: 0.0018820861678004535 r_at_150: 0.21391752577319587\n",
            "r_at_150_any: 0.2687074829931973\n",
            "r_at_150_all: 0.11904761904761904\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 83 tp-any-mentions: 79 tp-all-mentions: 35 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_200: 0.00141156462585034 r_at_200: 0.21391752577319587\n",
            "r_at_200_any: 0.2687074829931973\n",
            "r_at_200_all: 0.11904761904761904\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 83 tp-any-mentions: 79 tp-all-mentions: 35 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_250: 0.001129251700680272 r_at_250: 0.21391752577319587\n",
            "r_at_250_any: 0.2687074829931973\n",
            "r_at_250_all: 0.11904761904761904\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 83 tp-any-mentions: 79 tp-all-mentions: 35 num_mentions: 294 num_gold_edges: 388\n",
            "p_at_300: 0.0009410430839002267 r_at_300: 0.21391752577319587\n",
            "r_at_300_any: 0.2687074829931973\n",
            "r_at_300_all: 0.11904761904761904\n",
            "non-leaf edges results:\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 1 tp-any-mentions: 1 tp-all-mentions: 1 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_1: 0.02857142857142857 r_at_1: 0.0035211267605633804\n",
            "r_at_1_any: 0.02857142857142857\n",
            "r_at_1_all: 0.02857142857142857\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 2 tp-any-mentions: 2 tp-all-mentions: 1 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_3: 0.019047619047619046 r_at_3: 0.007042253521126761\n",
            "r_at_3_any: 0.05714285714285714\n",
            "r_at_3_all: 0.02857142857142857\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 6 tp-any-mentions: 6 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_5: 0.03428571428571429 r_at_5: 0.02112676056338028\n",
            "r_at_5_any: 0.17142857142857143\n",
            "r_at_5_all: 0.05714285714285714\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 12 tp-any-mentions: 11 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_10: 0.03428571428571429 r_at_10: 0.04225352112676056\n",
            "r_at_10_any: 0.3142857142857143\n",
            "r_at_10_all: 0.05714285714285714\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 12 tp-any-mentions: 11 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_20: 0.017142857142857144 r_at_20: 0.04225352112676056\n",
            "r_at_20_any: 0.3142857142857143\n",
            "r_at_20_all: 0.05714285714285714\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 12 tp-any-mentions: 11 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_50: 0.006857142857142857 r_at_50: 0.04225352112676056\n",
            "r_at_50_any: 0.3142857142857143\n",
            "r_at_50_all: 0.05714285714285714\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 12 tp-any-mentions: 11 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_100: 0.0034285714285714284 r_at_100: 0.04225352112676056\n",
            "r_at_100_any: 0.3142857142857143\n",
            "r_at_100_all: 0.05714285714285714\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 12 tp-any-mentions: 11 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_150: 0.002285714285714286 r_at_150: 0.04225352112676056\n",
            "r_at_150_any: 0.3142857142857143\n",
            "r_at_150_all: 0.05714285714285714\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 12 tp-any-mentions: 11 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_200: 0.0017142857142857142 r_at_200: 0.04225352112676056\n",
            "r_at_200_any: 0.3142857142857143\n",
            "r_at_200_all: 0.05714285714285714\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 12 tp-any-mentions: 11 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_250: 0.0013714285714285714 r_at_250: 0.04225352112676056\n",
            "r_at_250_any: 0.3142857142857143\n",
            "r_at_250_all: 0.05714285714285714\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 12 tp-any-mentions: 11 tp-all-mentions: 2 num_mentions: 35 num_gold_edges: 284\n",
            "p_at_300: 0.001142857142857143 r_at_300: 0.04225352112676056\n",
            "r_at_300_any: 0.3142857142857143\n",
            "r_at_300_all: 0.05714285714285714\n",
            "prompts in .txt saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/valid-NIL-top10-preds-prompts.txt\n",
            "prompts in .txt saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/valid-NIL-top10-preds-prompts-by-edges.txt\n",
            "len(dict_prompt_strs): 3096\n",
            "prompts in .csv saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/valid-NIL-top10-preds-prompts.csv\n",
            "len(dict_prompt_strs_by_edge): 329\n",
            "prompts in .csv saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/valid-NIL-top10-preds-prompts-by-edges.csv\n",
            "fname: models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/test-NIL.t7\n",
            "100% 237826/237826 [00:02<00:00, 86170.23it/s]\n",
            "edge catalogue loaded\n",
            "dict_ind_edge_json_info: 237826\n",
            "len(doc): 965\n",
            "len(edge_inds): 965\n",
            "100% 965/965 [00:00<00:00, 982.63it/s] \n",
            "all edges results:\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 0 tp-any-mentions: 0 tp-all-mentions: 0 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_1: 0.0 r_at_1: 0.0\n",
            "r_at_1_any: 0.0\n",
            "r_at_1_all: 0.0\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 5 tp-any-mentions: 5 tp-all-mentions: 2 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_3: 0.006038647342995169 r_at_3: 0.0051813471502590676\n",
            "r_at_3_any: 0.018115942028985508\n",
            "r_at_3_all: 0.007246376811594203\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 7 tp-any-mentions: 7 tp-all-mentions: 3 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_5: 0.0050724637681159425 r_at_5: 0.007253886010362694\n",
            "r_at_5_any: 0.025362318840579712\n",
            "r_at_5_all: 0.010869565217391304\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 59 tp-any-mentions: 55 tp-all-mentions: 46 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_10: 0.021376811594202898 r_at_10: 0.061139896373056994\n",
            "r_at_10_any: 0.19927536231884058\n",
            "r_at_10_all: 0.16666666666666666\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 59 tp-any-mentions: 55 tp-all-mentions: 46 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_20: 0.010688405797101449 r_at_20: 0.061139896373056994\n",
            "r_at_20_any: 0.19927536231884058\n",
            "r_at_20_all: 0.16666666666666666\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 59 tp-any-mentions: 55 tp-all-mentions: 46 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_50: 0.00427536231884058 r_at_50: 0.061139896373056994\n",
            "r_at_50_any: 0.19927536231884058\n",
            "r_at_50_all: 0.16666666666666666\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 59 tp-any-mentions: 55 tp-all-mentions: 46 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_100: 0.00213768115942029 r_at_100: 0.061139896373056994\n",
            "r_at_100_any: 0.19927536231884058\n",
            "r_at_100_all: 0.16666666666666666\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 59 tp-any-mentions: 55 tp-all-mentions: 46 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_150: 0.00142512077294686 r_at_150: 0.061139896373056994\n",
            "r_at_150_any: 0.19927536231884058\n",
            "r_at_150_all: 0.16666666666666666\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 59 tp-any-mentions: 55 tp-all-mentions: 46 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_200: 0.001068840579710145 r_at_200: 0.061139896373056994\n",
            "r_at_200_any: 0.19927536231884058\n",
            "r_at_200_all: 0.16666666666666666\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 59 tp-any-mentions: 55 tp-all-mentions: 46 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_250: 0.0008550724637681159 r_at_250: 0.061139896373056994\n",
            "r_at_250_any: 0.19927536231884058\n",
            "r_at_250_all: 0.16666666666666666\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 59 tp-any-mentions: 55 tp-all-mentions: 46 num_mentions: 276 num_gold_edges: 965\n",
            "p_at_300: 0.00071256038647343 r_at_300: 0.061139896373056994\n",
            "r_at_300_any: 0.19927536231884058\n",
            "r_at_300_all: 0.16666666666666666\n",
            "leaf edges results:\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 0 tp-any-mentions: 0 tp-all-mentions: 0 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_1: 0.0 r_at_1: 0.0\n",
            "r_at_1_any: 0.0\n",
            "r_at_1_all: 0.0\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 0 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_3: 0.0045871559633027525 r_at_3: 0.012711864406779662\n",
            "r_at_3_any: 0.013761467889908258\n",
            "r_at_3_all: 0.0\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 4 tp-any-mentions: 4 tp-all-mentions: 1 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_5: 0.003669724770642202 r_at_5: 0.01694915254237288\n",
            "r_at_5_any: 0.01834862385321101\n",
            "r_at_5_all: 0.0045871559633027525\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 56 tp-any-mentions: 52 tp-all-mentions: 44 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_10: 0.025688073394495414 r_at_10: 0.23728813559322035\n",
            "r_at_10_any: 0.23853211009174313\n",
            "r_at_10_all: 0.2018348623853211\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 56 tp-any-mentions: 52 tp-all-mentions: 44 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_20: 0.012844036697247707 r_at_20: 0.23728813559322035\n",
            "r_at_20_any: 0.23853211009174313\n",
            "r_at_20_all: 0.2018348623853211\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 56 tp-any-mentions: 52 tp-all-mentions: 44 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_50: 0.005137614678899083 r_at_50: 0.23728813559322035\n",
            "r_at_50_any: 0.23853211009174313\n",
            "r_at_50_all: 0.2018348623853211\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 56 tp-any-mentions: 52 tp-all-mentions: 44 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_100: 0.0025688073394495416 r_at_100: 0.23728813559322035\n",
            "r_at_100_any: 0.23853211009174313\n",
            "r_at_100_all: 0.2018348623853211\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 56 tp-any-mentions: 52 tp-all-mentions: 44 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_150: 0.0017125382262996943 r_at_150: 0.23728813559322035\n",
            "r_at_150_any: 0.23853211009174313\n",
            "r_at_150_all: 0.2018348623853211\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 56 tp-any-mentions: 52 tp-all-mentions: 44 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_200: 0.0012844036697247708 r_at_200: 0.23728813559322035\n",
            "r_at_200_any: 0.23853211009174313\n",
            "r_at_200_all: 0.2018348623853211\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 56 tp-any-mentions: 52 tp-all-mentions: 44 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_250: 0.0010275229357798166 r_at_250: 0.23728813559322035\n",
            "r_at_250_any: 0.23853211009174313\n",
            "r_at_250_all: 0.2018348623853211\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 56 tp-any-mentions: 52 tp-all-mentions: 44 num_mentions: 218 num_gold_edges: 236\n",
            "p_at_300: 0.0008562691131498471 r_at_300: 0.23728813559322035\n",
            "r_at_300_any: 0.23853211009174313\n",
            "r_at_300_all: 0.2018348623853211\n",
            "non-leaf edges results:\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 0 tp-any-mentions: 0 tp-all-mentions: 0 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_1: 0.0 r_at_1: 0.0\n",
            "r_at_1_any: 0.0\n",
            "r_at_1_all: 0.0\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 2 tp-any-mentions: 2 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_3: 0.011494252873563218 r_at_3: 0.0027434842249657062\n",
            "r_at_3_any: 0.034482758620689655\n",
            "r_at_3_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_5: 0.010344827586206896 r_at_5: 0.00411522633744856\n",
            "r_at_5_any: 0.05172413793103448\n",
            "r_at_5_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_10: 0.005172413793103448 r_at_10: 0.00411522633744856\n",
            "r_at_10_any: 0.05172413793103448\n",
            "r_at_10_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_20: 0.002586206896551724 r_at_20: 0.00411522633744856\n",
            "r_at_20_any: 0.05172413793103448\n",
            "r_at_20_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_50: 0.0010344827586206897 r_at_50: 0.00411522633744856\n",
            "r_at_50_any: 0.05172413793103448\n",
            "r_at_50_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_100: 0.0005172413793103448 r_at_100: 0.00411522633744856\n",
            "r_at_100_any: 0.05172413793103448\n",
            "r_at_100_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_150: 0.0003448275862068965 r_at_150: 0.00411522633744856\n",
            "r_at_150_any: 0.05172413793103448\n",
            "r_at_150_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_200: 0.0002586206896551724 r_at_200: 0.00411522633744856\n",
            "r_at_200_any: 0.05172413793103448\n",
            "r_at_200_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_250: 0.00020689655172413793 r_at_250: 0.00411522633744856\n",
            "r_at_250_any: 0.05172413793103448\n",
            "r_at_250_all: 0.034482758620689655\n",
            "ave_edge_cand_len: 10.0\n",
            "tp-edges: 3 tp-any-mentions: 3 tp-all-mentions: 2 num_mentions: 58 num_gold_edges: 729\n",
            "p_at_300: 0.00017241379310344826 r_at_300: 0.00411522633744856\n",
            "r_at_300_any: 0.05172413793103448\n",
            "r_at_300_all: 0.034482758620689655\n",
            "prompts in .txt saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/test-NIL-top10-preds-prompts.txt\n",
            "prompts in .txt saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/test-NIL-top10-preds-prompts-by-edges.txt\n",
            "len(dict_prompt_strs): 2676\n",
            "prompts in .csv saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/test-NIL-top10-preds-prompts.csv\n",
            "len(dict_prompt_strs_by_edge): 275\n",
            "prompts in .csv saved to models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/test-NIL-top10-preds-prompts-by-edges.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Display an intermediary prompt below (the first prompt in the test-NIL dataset).\n",
        "\n",
        "Note that the intermediary prompt is not the final promot - which is later preprocessed with\n",
        "\n",
        "```\n",
        "from blink.prompting.Llama_2_finetuning import prompt_concat_ctx_and_men, prompt_id_removal, prompt_correction\n",
        "from blink.prompting.Llama_2_prompting import format_prompt\n",
        "prompt = prompt_correction(prompt) # correct prompt\n",
        "prompt = prompt_concat_ctx_and_men(prompt) # concatenate context and mention in the prompt (i.e. use the non-separated paragraph with mention marked as *mention*.)\n",
        "prompt = prompt_id_removal(prompt) # remove ids (and parentheses) in prompt\n",
        "```\n",
        "\n",
        "For Llama 2 models, this further processing is applied\n",
        "```\n",
        "prompt_formatted = format_prompt(prompt,cot=True) # format prompt for Llama 2 (same as in fine-tuning), here cot means using explaination before getting the answer\n",
        "```\n",
        "\n",
        "For the final version of a prompt, see Section 4.3 on page 8-9 in the [paper](https://arxiv.org/pdf/2402.17897).\n"
      ],
      "metadata": {
        "id": "dXsEOyTRzy0Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# showing the intermediary prompt for test-NIL's first piece of data (each prompt takes 12+top_k_value=22 lines)\n",
        "!head -22 models/biencoder/mm+Disease2017AA-tl-sapbert-NIL-bs16-debug/top10_candidates/test-NIL-top10-preds-prompts-by-edges.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bbNEumzVbVce",
        "outputId": "f7f9fc2f-f73a-4b90-d34f-767b30b97834"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "﻿Can you identify the correct ontological edges for the given mention based on the provided context (context_left and context_right)? The ontological edge consists of a pair where the left concept represents the parent of the mention, and the right concept represents the child of the mention. If the mention is a leaf node, the right side of the edges will be NULL (SCTID_NULL). If the context is not relevant to the options, make your decision solely based on the mention itself. There may be multiple correct options. Please answer briefly using option numbers, separated by a colon. If none of the options is correct, please answer None.\n",
            "\n",
            "context_left:\n",
            "Incontinence -associated dermatitis: reducing adverse events Incontinence -associated \n",
            "\n",
            "mention:\n",
            "dermatitis\n",
            "\n",
            "context_right:\n",
            " (IAD) is a common problem in patients with faecal and/or urinary incontinence. Urine alters the normal skin flora and increases permeability of the stratum corneum and faecal enzymes on the skin contribute to skin damage. Faecal bacteria can then penetrate the skin, increasing the risk of secondary infection. However, IAD can be prevented and healed with timely and appropriate skin cleansing and skin protection. This includes appropriate use of containment devices. This article also looks at HARTMANN incontinence pads that have been developed to absorb the fluids that cause IAD and maintain the skin's acidic pH. The acidic pH of the skin contributes to its barrier function and defence against infection. Therefore, maintaining an acidic pH will help protect the skin from damage.\n",
            "\n",
            "options:\n",
            "0.chronic papillomatous dermatitis due to incontinence (disorder) (402277003) -> NULL (SCTID_NULL)\n",
            "1.chronic disease of skin (disorder) (128236002) -> chronic generalized exfoliative dermatitis (disorder) (31922003)\n",
            "2.disorder of skin (disorder) (95320005) -> pruritus ani (disorder) (90446007)\n",
            "3.chronic papillomatous dermatitis due to urostomy (disorder) (402282005) -> NULL (SCTID_NULL)\n",
            "4.disorder of skin (disorder) (95320005) -> exuberant cicatrix (disorder) (201308009)\n",
            "5.irritant contact dermatitis due to urostomy (disorder) (402281003) -> NULL (SCTID_NULL)\n",
            "6.irritant contact dermatitis due to incontinence (disorder) (402276007) -> NULL (SCTID_NULL)\n",
            "7.disorder of skin (disorder) (95320005) -> NULL (SCTID_NULL)\n",
            "8.disorder of integument (disorder) (128598002) -> NULL (SCTID_NULL)\n",
            "9.stoma-related chronic papillomatous dermatitis (disorder) (402283000) -> NULL (SCTID_NULL)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# installing packages for prompting and fine-tuning LLMs.\n",
        "!pip install -r requirements-LLM.txt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8HcWbAxSdedF",
        "outputId": "84cc79ce-c04e-4f32-ef44-0cf82343e1ce"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting torch==2.0.1 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl.metadata (24 kB)\n",
            "Collecting tqdm==4.64.1 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 4))\n",
            "  Downloading tqdm-4.64.1-py2.py3-none-any.whl.metadata (57 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.3/57.3 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting deeponto==0.8.7 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5))\n",
            "  Downloading deeponto-0.8.7-py3-none-any.whl.metadata (12 kB)\n",
            "Collecting peft==0.4.0 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 6))\n",
            "  Downloading peft-0.4.0-py3-none-any.whl.metadata (21 kB)\n",
            "Collecting bitsandbytes==0.40.2 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 7))\n",
            "  Downloading bitsandbytes-0.40.2-py3-none-any.whl.metadata (9.8 kB)\n",
            "Collecting trl==0.4.7 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 8))\n",
            "  Downloading trl-0.4.7-py3-none-any.whl.metadata (10 kB)\n",
            "Collecting transformers==4.31.0 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9))\n",
            "  Downloading transformers-4.31.0-py3-none-any.whl.metadata (116 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.9/116.9 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting openai==0.27.4 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10))\n",
            "  Downloading openai-0.27.4-py3-none-any.whl.metadata (13 kB)\n",
            "Collecting openpyxl==3.1.2 (from -r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 11))\n",
            "  Downloading openpyxl-3.1.2-py2.py3-none-any.whl.metadata (2.5 kB)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (3.15.4)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (4.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (1.13.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (3.1.4)\n",
            "Collecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-runtime-cu11==11.7.99 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cuda-cupti-cu11==11.7.101 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cudnn-cu11==8.5.0.96 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cublas-cu11==11.10.3.66 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cufft-cu11==10.9.0.58 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-curand-cu11==10.2.10.91 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cusolver-cu11==11.4.0.1 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cusparse-cu11==11.7.4.91 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-nccl-cu11==2.14.3 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)\n",
            "Collecting nvidia-nvtx-cu11==11.7.91 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)\n",
            "Collecting triton==2.0.0 (from torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading triton-2.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.0 kB)\n",
            "Requirement already satisfied: JPype1==1.3.0 in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.3.0)\n",
            "Requirement already satisfied: yacs in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.1.8)\n",
            "Requirement already satisfied: anytree==2.8.0 in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.8.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (8.1.7)\n",
            "Requirement already satisfied: dill in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.3.8)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.1.4)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.26.4)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.3.2)\n",
            "Requirement already satisfied: openprompt==1.0.0 in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.0.0)\n",
            "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.19.2)\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.7.5)\n",
            "Requirement already satisfied: pprintpp in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.4.0)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (4.9.4)\n",
            "Requirement already satisfied: textdistance in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (4.6.3)\n",
            "Requirement already satisfied: ipywidgets==7.7.1 in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (7.7.1)\n",
            "Requirement already satisfied: ipykernel in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (5.5.6)\n",
            "Requirement already satisfied: enlighten in /usr/local/lib/python3.10/dist-packages (from deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.12.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 6)) (24.1)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 6)) (5.9.5)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 6)) (6.0.1)\n",
            "Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 6)) (0.32.1)\n",
            "Requirement already satisfied: safetensors in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 6)) (0.4.3)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (0.23.5)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (2019.8.19)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (2.32.3)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (0.13.3)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from openai==0.27.4->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10)) (3.10.0)\n",
            "Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.10/dist-packages (from openpyxl==3.1.2->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 11)) (1.1.0)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from anytree==2.8.0->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.16.0)\n",
            "Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.2.0)\n",
            "Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (5.7.1)\n",
            "Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.6.8)\n",
            "Requirement already satisfied: ipython>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (7.34.0)\n",
            "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.0.11)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (71.0.4)\n",
            "Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (0.43.0)\n",
            "Requirement already satisfied: sentencepiece==0.1.96 in /usr/local/lib/python3.10/dist-packages (from openprompt==1.0.0->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.1.96)\n",
            "Requirement already satisfied: tensorboardX in /usr/local/lib/python3.10/dist-packages (from openprompt==1.0.0->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.6.2.2)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from openprompt==1.0.0->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.6.3)\n",
            "Requirement already satisfied: rouge==1.0.0 in /usr/local/lib/python3.10/dist-packages (from openprompt==1.0.0->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.0.0)\n",
            "Requirement already satisfied: pyarrow in /usr/local/lib/python3.10/dist-packages (from openprompt==1.0.0->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (14.0.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from openprompt==1.0.0->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.13.1)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (3.30.1)\n",
            "Collecting lit (from triton==2.0.0->torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3))\n",
            "  Downloading lit-18.1.8-py3-none-any.whl.metadata (2.5 kB)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (2024.3.1)\n",
            "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.10/dist-packages (from ipykernel->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (6.1.12)\n",
            "Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.10/dist-packages (from ipykernel->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (6.3.3)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.31.0->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 9)) (2024.7.4)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.4->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10)) (2.3.4)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.4->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10)) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.4->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10)) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.4->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10)) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.4->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10)) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.4->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10)) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.4->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 10)) (4.0.3)\n",
            "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.6)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.4.1)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.70.16)\n",
            "Requirement already satisfied: blessed>=1.17.7 in /usr/local/lib/python3.10/dist-packages (from enlighten->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.20.0)\n",
            "Requirement already satisfied: prefixed>=0.3.2 in /usr/local/lib/python3.10/dist-packages (from enlighten->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.7.1)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (2.1.5)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2024.1)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2024.1)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.5.0)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.0.10)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.0.8)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (8.2.5)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.1.3)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.0.10)\n",
            "Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.4.1)\n",
            "Requirement already satisfied: typer<1.0.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.12.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.8.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.4.0)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.0.1->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 3)) (1.3.0)\n",
            "Requirement already satisfied: wcwidth>=0.1.4 in /usr/local/lib/python3.10/dist-packages (from blessed>=1.17.7->enlighten->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.2.13)\n",
            "Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.19.1)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (4.4.2)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.7.5)\n",
            "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.0.47)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.16.1)\n",
            "Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.2.0)\n",
            "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.1.7)\n",
            "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (4.9.0)\n",
            "Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.10/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.2.0)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.20.1)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.1.5)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.5.4)\n",
            "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (13.7.1)\n",
            "Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.18.1)\n",
            "Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (7.0.4)\n",
            "Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.10/dist-packages (from widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (6.5.5)\n",
            "Requirement already satisfied: jupyter-core>=4.6.0 in /usr/local/lib/python3.10/dist-packages (from jupyter-client->ipykernel->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (5.7.2)\n",
            "Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.10/dist-packages (from jupyter-client->ipykernel->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (24.0.1)\n",
            "Requirement already satisfied: protobuf>=3.20 in /usr/local/lib/python3.10/dist-packages (from tensorboardX->openprompt==1.0.0->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.20.3)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.8.4)\n",
            "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core>=4.6.0->jupyter-client->ipykernel->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (4.2.2)\n",
            "Requirement already satisfied: marisa-trie>=0.7.7 in /usr/local/lib/python3.10/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.2.0)\n",
            "Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (23.1.0)\n",
            "Requirement already satisfied: nbformat in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (5.10.4)\n",
            "Requirement already satisfied: nbconvert>=5 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (6.5.4)\n",
            "Requirement already satisfied: nest-asyncio>=1.5 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.6.0)\n",
            "Requirement already satisfied: Send2Trash>=1.8.0 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.8.3)\n",
            "Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.18.1)\n",
            "Requirement already satisfied: prometheus-client in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.20.0)\n",
            "Requirement already satisfied: nbclassic>=0.4.7 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.1.0)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->ipython>=4.0.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.7.0)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.0.0)\n",
            "Requirement already satisfied: wrapt in /usr/local/lib/python3.10/dist-packages (from smart-open<8.0.0,>=5.2.1->weasel<0.5.0,>=0.1.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.16.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.1.2)\n",
            "Requirement already satisfied: notebook-shim>=0.2.3 in /usr/local/lib/python3.10/dist-packages (from nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.2.4)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (4.12.3)\n",
            "Requirement already satisfied: bleach in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (6.1.0)\n",
            "Requirement already satisfied: defusedxml in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.7.1)\n",
            "Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.4)\n",
            "Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.3.0)\n",
            "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.8.4)\n",
            "Requirement already satisfied: nbclient>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.10.0)\n",
            "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.5.1)\n",
            "Requirement already satisfied: tinycss2 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.3.0)\n",
            "Requirement already satisfied: fastjsonschema>=2.15 in /usr/local/lib/python3.10/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.20.0)\n",
            "Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.10/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (4.23.0)\n",
            "Requirement already satisfied: argon2-cffi-bindings in /usr/local/lib/python3.10/dist-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (21.2.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2023.12.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.19.1)\n",
            "Requirement already satisfied: jupyter-server<3,>=1.8 in /usr/local/lib/python3.10/dist-packages (from notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.24.0)\n",
            "Requirement already satisfied: cffi>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.16.0)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.5)\n",
            "Requirement already satisfied: webencodings in /usr/local/lib/python3.10/dist-packages (from bleach->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (0.5.1)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (2.22)\n",
            "Requirement already satisfied: anyio<4,>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from jupyter-server<3,>=1.8->notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (3.7.1)\n",
            "Requirement already satisfied: websocket-client in /usr/local/lib/python3.10/dist-packages (from jupyter-server<3,>=1.8->notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.8.0)\n",
            "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.1.0->jupyter-server<3,>=1.8->notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.3.1)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.1.0->jupyter-server<3,>=1.8->notebook-shim>=0.2.3->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets==7.7.1->deeponto==0.8.7->-r /content/LM-ontology-concept-placement/requirements-LLM.txt (line 5)) (1.2.2)\n",
            "Downloading torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m619.9/619.9 MB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.5/78.5 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading deeponto-0.8.7-py3-none-any.whl (89.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m89.7/89.7 MB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading peft-0.4.0-py3-none-any.whl (72 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.9/72.9 kB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading bitsandbytes-0.40.2-py3-none-any.whl (92.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.5/92.5 MB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading trl-0.4.7-py3-none-any.whl (77 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.4/77.4 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading transformers-4.31.0-py3-none-any.whl (7.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m85.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading openai-0.27.4-py3-none-any.whl (70 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m70.3/70.3 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading openpyxl-3.1.2-py2.py3-none-any.whl (249 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m250.0/250.0 kB\u001b[0m \u001b[31m20.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 MB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.8/11.8 MB\u001b[0m \u001b[31m108.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m85.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m849.3/849.3 kB\u001b[0m \u001b[31m47.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m557.1/557.1 MB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m168.4/168.4 MB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.6/54.6 MB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.6/102.6 MB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m173.2/173.2 MB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m177.1/177.1 MB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.6/98.6 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading triton-2.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.3/63.3 MB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lit-18.1.8-py3-none-any.whl (96 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m96.4/96.4 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: lit, bitsandbytes, tqdm, openpyxl, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, nvidia-cusolver-cu11, nvidia-cudnn-cu11, transformers, openai, triton, torch, trl, peft, deeponto\n",
            "  Attempting uninstall: tqdm\n",
            "    Found existing installation: tqdm 4.66.1\n",
            "    Uninstalling tqdm-4.66.1:\n",
            "      Successfully uninstalled tqdm-4.66.1\n",
            "  Attempting uninstall: openpyxl\n",
            "    Found existing installation: openpyxl 3.1.5\n",
            "    Uninstalling openpyxl-3.1.5:\n",
            "      Successfully uninstalled openpyxl-3.1.5\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.29.0\n",
            "    Uninstalling transformers-4.29.0:\n",
            "      Successfully uninstalled transformers-4.29.0\n",
            "  Attempting uninstall: triton\n",
            "    Found existing installation: triton 2.3.1\n",
            "    Uninstalling triton-2.3.1:\n",
            "      Successfully uninstalled triton-2.3.1\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 1.11.0\n",
            "    Uninstalling torch-1.11.0:\n",
            "      Successfully uninstalled torch-1.11.0\n",
            "  Attempting uninstall: deeponto\n",
            "    Found existing installation: deeponto 0.8.8\n",
            "    Uninstalling deeponto-0.8.8:\n",
            "      Successfully uninstalled deeponto-0.8.8\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "torchaudio 2.3.1+cu121 requires torch==2.3.1, but you have torch 2.0.1 which is incompatible.\n",
            "torchtext 0.18.0 requires torch>=2.3.0, but you have torch 2.0.1 which is incompatible.\n",
            "torchvision 0.12.0 requires torch==1.11.0, but you have torch 2.0.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed bitsandbytes-0.40.2 deeponto-0.8.7 lit-18.1.8 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 openai-0.27.4 openpyxl-3.1.2 peft-0.4.0 torch-2.0.1 tqdm-4.64.1 transformers-4.31.0 triton-2.0.0 trl-0.4.7\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Example scripts for prompting and fine-tuning Large Language Models."
      ],
      "metadata": {
        "id": "3Paxt4xckAVW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# go to the prompting folder\n",
        "%cd blink/prompting\n",
        "\n",
        "# below scripts point to the python files for prompting and fine-tuning.\n",
        "# the scripts need to be configured with top_k_value, model and prompt format settings before running\n",
        "\n",
        "#!python chatGPT-prompting-edge.py\n",
        "#!python GPT_results_eval.py\n",
        "\n",
        "#!huggingface-cli login\n",
        "#!python Llama_2_finetuning.py\n",
        "#!python Llama_2_prompting.py\n",
        "#!python Llama_2_results_eval.py\n",
        "\n",
        "#!python FLAN_T5_prompting.py\n",
        "#!python FLAN_T5_results_eval.py\n"
      ],
      "metadata": {
        "id": "tmXD-oVda36R"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}