08_tfidf.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#default_exp tfidf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev.showdoc import *\n",
    "# all_flag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Search using TF-IDF and Elasticsearch\n",
    "> Summary: Information Retrieval, tf-idf, Elasticsearch, Text Matching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is TF-IDF?\n",
    "\n",
    "TF-IDF stands for \"Term Frequency — Inverse Document Frequency\". It is a statistical technique that quantifies the importance of a word in a document based on how often it appears in that document and a given collection of documents (corpus). The intuition for this measure is : If a word occurs frequently in a document, then it should be more important and relevant than other words that appear fewer times and we should give that word a high score (TF). But if a word appears many times in a document but also in too many other documents, it’s probably not a relevant and meaningful word, therefore we should assign a lower score to that word (IDF). The relevancy of a word is proportional to the amount of information that it gives about its context (a sentence, a document or a full dataset). The more relevant words help us better understand the entire document without reading it completely. The most relevant words are not necessary the most frequent words since **stopwords** like \"the\", \"of\" or \"a\" tend to occur very often in many documents, but do not give much information. TF-IDF method is widely used in Information Retrieval and Text Mining. The TF-IDF score of term $t$ in document $d$ with respect to corpus $D$ is:\n",
    "\n",
    "$$tfidf(t,d,D)=tf(t,d)\\times idf(d,D)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Term Frequency (TF) Score\n",
    "\n",
    "First we have to calculate the $tf(t,d)$, which is simply the number of times each word $t$ appeared in document $d$. While calculating $tf(t,d)$, we usually remove words like \"a\", \"as\", \"the\". These words are called stopwords and will not provide much information. Additionally, there could be many high frequency non-stopwords that do not provide much information in a given context (e.g., “Disney” in a collection of documents about “Disney\tWorld”), Therefore, we can filter them out too. Also, we normalize the term-frequency to make sure there is no bias for longer or shorter documents. Thus, we have:\n",
    "\n",
    "$$tf(t,d)=\\frac{f_{t,d}}{\\sum_{t'} f_{t',d}}$$\n",
    "\n",
    "where $f_{t,d}$ is the number of occurances of $t$ in $d$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inverse Document Frequency (IDF)\n",
    "\n",
    "It measures how rare a term $t$ is across the corpus $D$, meaning how much information it provides about a document it appears in. If  total number of documents in the corpus is $N=|D|$, and $n_t$ is the number of documents having $t$, then we have:\n",
    "\n",
    "$$idf(t,D)=\\log(\\frac{N}{n_t})$$\n",
    "\n",
    "The reason that we take the $\\log$ of IDF is that if we have a large corpus, the IDF values will become so large, therefore,  we use the $\\log$ value to decrease that effect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-IDF Example\n",
    "\n",
    "In order to fully understand how TF-IDF works, I will give you a concrete example. Let's assume that we have a collection of four documents as follows:\n",
    "\n",
    "- $d_1$: \"*The sky is blue.*\n",
    "\n",
    "- $d_2$: \"*The sun is bright today.*\"\n",
    "\n",
    "- $d_3$: \"*The sun in the sky is bright.*\"\n",
    "\n",
    "- $d_4$: \"*We can see the shining sun, the bright sun.*\"\n",
    "\n",
    "**Task:** Determine the tf-idf scores for each term in each document.\n",
    "\n",
    "- **Step1:** Filter out the stopwords. After removing the stopwords, we have \n",
    "\n",
    "    - $d_1$: \"*sky blue*\n",
    "\n",
    "    - $d_2$: \"*sun bright today*\"\n",
    "\n",
    "    - $d_3$: \"*sun sky bright*\"\n",
    "\n",
    "    - $d_4$: \"*can see shining sun bright sun*\"\n",
    "\n",
    "- **Step2:** Compute TF, therefore, we find document-word matrix and then normalize the rows to sum to 1.\n",
    "\n",
    "![](images/tfidf_ex1.png)\n",
    "*TF score computation. [[Image Source](http://www.cbrinton.net/ECE20875-2020-Spring/W10/ngrams.pdf)]*\n",
    "\n",
    "- **Step3:** Compute IDF: Find the number of documents in which each word occurs, then compute the formula:\n",
    "\n",
    "![](images/tfidf_ex2.png)\n",
    "*IDF score computation. [[Image Source](http://www.cbrinton.net/ECE20875-2020-Spring/W10/ngrams.pdf)]*\n",
    "\n",
    "\n",
    "- **Step4:** Compute TF-IDF: Multiply TF and IDF scores.\n",
    "\n",
    "![](images/tfidf_ex3.png)\n",
    "*TF-IDF score computation. [[Image Source](http://www.cbrinton.net/ECE20875-2020-Spring/W10/ngrams.pdf)]*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Application of tf-idf for Searching Text\n",
    "\n",
    "In order to understand how to use tf-idf, I am going to make use of this technique in a text searching application. I will use a dataset of [Python questions and answers from Stackoverflow](https://www.kaggle.com/stackoverflow/pythonquestions). The dataset contains all the questions (around 700,000) asked between August 2, 2008 and Ocotober 19, 2016. Please see the link for all the details about this dataset. For this application, I only use the Python questions. However, it would be an interesting exercise to create a question-answering application. Each question in this file contains `title` and `body` among other attributes. But I will merely use these two fields from each question in the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import time\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def preprocess(title, body=None):\n",
    "    \"\"\" Preprocess the input, i.e. lowercase, remove html tags, special character and digits.\"\"\"\n",
    "    text = ''\n",
    "    if body is None:\n",
    "        text = title\n",
    "    else:\n",
    "        text = title + body\n",
    "    # to lower case\n",
    "    text = text.lower()\n",
    "\n",
    "    # remove tags\n",
    "    text = re.sub(\"</?.*?>\",\" <> \", text)\n",
    "    \n",
    "    # remove special characters and digits\n",
    "    text = re.sub(\"(\\\\d|\\\\W)+\",\" \", text).strip()\n",
    "    return text\n",
    "    \n",
    "def create_tfidf_features(corpus, max_features=5000, max_df=0.95, min_df=2):\n",
    "    \"\"\" Creates a tf-idf matrix for the `corpus` using sklearn. \"\"\"\n",
    "    tfidf_vectorizor = TfidfVectorizer(decode_error='replace', strip_accents='unicode', analyzer='word', \n",
    "                                       stop_words='english', ngram_range=(1, 1), max_features=max_features, \n",
    "                                       norm='l2', use_idf=True, smooth_idf=True, sublinear_tf=True,\n",
    "                                       max_df=max_df, min_df=min_df)\n",
    "    X = tfidf_vectorizor.fit_transform(corpus)\n",
    "    print('tfidf matrix successfully created.')\n",
    "    return X, tfidf_vectorizor\n",
    "\n",
    "def calculate_similarity(X, vectorizor, query, top_k=5):\n",
    "    \"\"\" Vectorizes the `query` via `vectorizor` and calculates the cosine similarity of \n",
    "    the `query` and `X` (all the documents) and returns the `top_k` similar documents.\"\"\"\n",
    "    \n",
    "    # Vectorize the query to the same length as documents\n",
    "    query_vec = vectorizor.transform(query)\n",
    "    # Compute the cosine similarity between query_vec and all the documents\n",
    "    cosine_similarities = cosine_similarity(X,query_vec).flatten()\n",
    "    # Sort the similar documents from the most similar to less similar and return the indices\n",
    "    most_similar_doc_indices = np.argsort(cosine_similarities, axis=0)[:-top_k-1:-1]\n",
    "    return (most_similar_doc_indices, cosine_similarities)\n",
    "\n",
    "def show_similar_documents(df, cosine_similarities, similar_doc_indices):\n",
    "    \"\"\" Prints the most similar documents using indices in the `similar_doc_indices` vector.\"\"\"\n",
    "    counter = 1\n",
    "    for index in similar_doc_indices:\n",
    "        print('Top-{}, Similarity = {}'.format(counter, cosine_similarities[index]))\n",
    "        print('body: {}, '.format(df[index]))\n",
    "        print()\n",
    "        counter += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First step is to read and load the `questions.csv` file. Please note that it may take several seconds to fully load the file due to large number of records (607282 questions) in the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading the Questions file...\n",
      "done\n"
     ]
    }
   ],
   "source": [
    "# Reading the csv file of python Questions\n",
    "FILE_PATH = os.path.join('data','stackoverflow','Questions.csv')\n",
    "print('Reading the Questions file...')\n",
    "df = pd.read_csv(FILE_PATH, delimiter=',', encoding='ISO-8859-1')\n",
    "print('done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>OwnerUserId</th>\n",
       "      <th>CreationDate</th>\n",
       "      <th>Score</th>\n",
       "      <th>Title</th>\n",
       "      <th>Body</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>469</td>\n",
       "      <td>147.0</td>\n",
       "      <td>2008-08-02T15:11:16Z</td>\n",
       "      <td>21</td>\n",
       "      <td>How can I find the full path to a font from it...</td>\n",
       "      <td>&lt;p&gt;I am using the Photoshop's javascript API t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>502</td>\n",
       "      <td>147.0</td>\n",
       "      <td>2008-08-02T17:01:58Z</td>\n",
       "      <td>27</td>\n",
       "      <td>Get a preview JPEG of a PDF on Windows?</td>\n",
       "      <td>&lt;p&gt;I have a cross-platform (Python) applicatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>535</td>\n",
       "      <td>154.0</td>\n",
       "      <td>2008-08-02T18:43:54Z</td>\n",
       "      <td>40</td>\n",
       "      <td>Continuous Integration System for a Python Cod...</td>\n",
       "      <td>&lt;p&gt;I'm starting work on a hobby project with a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>594</td>\n",
       "      <td>116.0</td>\n",
       "      <td>2008-08-03T01:15:08Z</td>\n",
       "      <td>25</td>\n",
       "      <td>cx_Oracle: How do I iterate over a result set?</td>\n",
       "      <td>&lt;p&gt;There are several ways to iterate over a re...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>683</td>\n",
       "      <td>199.0</td>\n",
       "      <td>2008-08-03T13:19:16Z</td>\n",
       "      <td>28</td>\n",
       "      <td>Using 'in' to match an attribute of Python obj...</td>\n",
       "      <td>&lt;p&gt;I don't remember whether I was dreaming or ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Id  OwnerUserId          CreationDate  Score  \\\n",
       "0  469        147.0  2008-08-02T15:11:16Z     21   \n",
       "1  502        147.0  2008-08-02T17:01:58Z     27   \n",
       "2  535        154.0  2008-08-02T18:43:54Z     40   \n",
       "3  594        116.0  2008-08-03T01:15:08Z     25   \n",
       "4  683        199.0  2008-08-03T13:19:16Z     28   \n",
       "\n",
       "                                               Title  \\\n",
       "0  How can I find the full path to a font from it...   \n",
       "1            Get a preview JPEG of a PDF on Windows?   \n",
       "2  Continuous Integration System for a Python Cod...   \n",
       "3     cx_Oracle: How do I iterate over a result set?   \n",
       "4  Using 'in' to match an attribute of Python obj...   \n",
       "\n",
       "                                                Body  \n",
       "0  <p>I am using the Photoshop's javascript API t...  \n",
       "1  <p>I have a cross-platform (Python) applicatio...  \n",
       "2  <p>I'm starting work on a hobby project with a...  \n",
       "3  <p>There are several ways to iterate over a re...  \n",
       "4  <p>I don't remember whether I was dreaming or ...  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at a randomly selected question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title: Python: print() always writes to .ipynb making file too large, \n",
      "body: <p>To cut to the chase, I'd like to know how to make print statements display in ipython notebooks while simultaneously preventing those statements from saving to the .ipynb file. The purpose being to make a progress bar which doesn't make the file size ridiculously large.</p>\n",
      "\n",
      "<p>The background to this is that I've been writing a bit of python code which makes a bunch of png files so that I can eventually compile them into a GIF. While I was doing this I thought I'd be clever and print the progress of the task as it went using <code>print()</code> from <code>__future__</code> with carriage returns. Unfortunately though I've had two problem with my code, both of which I imagine are related to my implementation of this progress message.</p>\n",
      "\n",
      "<p>The first problem is with github's limit on file size:</p>\n",
      "\n",
      "<p>When I first tried to upload my file to github it prevented me from doing so because it exceeded their 100 MB limit. After investigating my .ipynb file I found that there was an obscene number of print statements which were being saved there. Initially I'd thought that including <code>'\\r'</code> to do carriage returns would prevent this, but clearly that's not the case.</p>\n",
      "\n",
      "<p>The second problem is probably related to this:</p>\n",
      "\n",
      "<p>Typically I don't have a problem creating the first few GIFs especially if I don't include that many frames, however beyond that my python notebook crashes. If this were a typical memory problem I'd imagine that it would just throw an error at me, but it doesn't, and instead promptly dies on me.</p>\n",
      "\n",
      "<p>Here's a sample of the sort of stuff that's bloating the .ipynb file:</p>\n",
      "\n",
      "<pre><code>{\n",
      "   \"output_type\": \"stream\",\n",
      "   \"stream\": \"stdout\",\n",
      "   \"text\": [\n",
      "    \"\\r\",\n",
      "    \"frame 1 -- 0.480% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.500% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.520% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.540% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.560% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.580% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.600% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.620% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.640% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.660% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.680% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.700% complete -- U_avg = -7.200000\\r\",\n",
      "    \"frame 1 -- 0.720% complete -- U_avg = -7.200000\\r\", ......\n",
      "</code></pre>\n",
      "\n",
      "<p>I've looked into how other people have implemented progress bars, however they don't appear to do anything special which would actually prevent the problem I'm having. If it comes down to it, I wouldn't mind importing something which would solve this problem in a black box, but at the same time if I run into this issue in a different context it would be useful to know how to reduce .ipynb file sizes by cutting out saved print statements.</p>\n",
      "\n",
      "<p>Thanks!</p>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sample_index = np.random.randint(len(df))\n",
    "sample = df.loc[sample_index,['Title', 'Body']]\n",
    "print('title: {}, \\nbody: {}'.format(sample['Title'],sample['Body']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preprocessing is one of the major steps when we are dealing with any kind of text models. As you can see above, the `body` of the question, it's true for all the questions, has plently of html tags and special characters. Therefore, we need to get rid of them as much as we can. The `preprocess()` function will carry out cleaning of the questions by removing html tags, special characters and digits. As usual, there is always room for improvements by adding more cleaning rules such as *stemming*, *lematization*, *stop words removal*, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocess the corpus\n",
    "data = [preprocess(title, body) for title, body in zip(df['Title'], df['Body'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we load and clean the data, it's time to create the `term-document` matrix. We can write simple functions for computing tf (term frequency) and idf (inverse document frequency). However, I leave this out as an interesting exercise. Instead I'll be using sklearn `TfidfVectorizer` to compute the word counts, idf and tf-idf values all at once. You can find all the details about `TfidfVectorizer` [here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer.transform). I would like to mention that in `create_tfidf_features()` function, I restrict the size of the vocabulary (i.e. number of features) to 5000 to make the computations cheaper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating tfidf matrix...\n",
      "tfidf matrix successfully created.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "5000"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('creating tfidf matrix...')\n",
    "# Learn vocabulary and idf, return term-document matrix\n",
    "X,v = create_tfidf_features(data)\n",
    "features = v.get_feature_names()\n",
    "len(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to test and see how our application works. We can ask an arbitrary question and see if the system can find the top-k most similar questions from the dataset to our question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "search time: 380.51 ms\n",
      "\n",
      "Top-1, Similarity = 0.6605864924688081\n",
      "body: loop through fixed number of files within a directory how can i loop through a fixed number of files within a directory with glob glob if there s more than x files within that directory i only want to loop through x and then exit the loop how do i do this, \n",
      "\n",
      "Top-2, Similarity = 0.6436681516641347\n",
      "body: how to list all files of a directory in python how can i list all files of a directory in python and add them to a list, \n",
      "\n",
      "Top-3, Similarity = 0.5968769006998226\n",
      "body: extract all zipped files to same directory using python i have a large amount of zipped files in a single directory that i would like to decompress and save them to the same directory and with the same name as the zipped file, \n",
      "\n",
      "Top-4, Similarity = 0.5377592889154766\n",
      "body: deleting all files in a directory with python i want to delete all files with the extension bak in a directory how can i do that in python, \n",
      "\n",
      "Top-5, Similarity = 0.5202741533554684\n",
      "body: move files from one directory to another based on a specific line i have some text files in a directory i would like to read all the files in this directory and if the file has the following line sample an integer of type decimal can be assembled by move that file with its all contents to another directory how can i do this, \n",
      "\n"
     ]
    }
   ],
   "source": [
    "user_question = ['how to loop over files in a directory']\n",
    "search_start = time.time()\n",
    "sim_vecs, cosine_similarities = calculate_similarity(X, v, user_question)\n",
    "search_time = time.time() - search_start\n",
    "print(\"search time: {:.2f} ms\".format(search_time * 1000))\n",
    "print()\n",
    "show_similar_documents(data, cosine_similarities, sim_vecs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to Use Elasticsearch for Indexing and Retrieving Text\n",
    "\n",
    "### What is Elasticsearch?\n",
    "\n",
    "[Elasticsearch](https://www.elastic.co/downloads/elasticsearch) is an open source distributed, RESTful search and analytics engine. Elasticsearch enables us to index, search, and analyze data at large scale. It provides real-time search and analytics for various types of data including structured or unstructured text, numerical data, or geospatial data. Elasticsearch can efficiently store and index it in a way that supports fast searches. In order to learn Elasticsearch please see the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-install.html). It is out of the scope of this tutorial, so I leave it as an exercise to understand and learn how Elasticsearch works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from elasticsearch import Elasticsearch\n",
    "from elasticsearch.helpers import bulk\n",
    "import json\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def create_index(es_client):\n",
    "    \"\"\" Creates an Elasticsearch index.\"\"\"\n",
    "    is_created = False\n",
    "    # Index settings\n",
    "    settings = {\n",
    "        \"settings\": {\n",
    "            \"number_of_shards\": 2,\n",
    "            \"number_of_replicas\": 1\n",
    "        },\n",
    "        \"mappings\": {\n",
    "            \"dynamic\": \"true\",\n",
    "            \"_source\": {\n",
    "            \"enabled\": \"true\"\n",
    "            },\n",
    "            \"properties\": {\n",
    "                \"body\": {\n",
    "                    \"type\": \"text\"\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "    print('Creating `Question` index...')\n",
    "    try:\n",
    "        if es_client.indices.exists(INDEX_NAME):\n",
    "            es_client.indices.delete(index=INDEX_NAME, ignore=[404])\n",
    "        es_client.indices.create(index=INDEX_NAME, body=settings)\n",
    "        is_created = True\n",
    "        print('index `Question` created successfully.')\n",
    "    except Exception as ex:\n",
    "        print(str(ex))\n",
    "    finally:\n",
    "        return is_created\n",
    "    return is_created\n",
    " \n",
    "    \n",
    "\n",
    "def index_data(es_client, data, BATCH_SIZE=100000):\n",
    "    \"\"\" Indexs all the rows in data (python questions).\"\"\"\n",
    "    docs = []\n",
    "    count = 0\n",
    "    for line in data:\n",
    "        js_object = {}\n",
    "        js_object['body'] = line\n",
    "        docs.append(js_object)\n",
    "        count += 1\n",
    "        \n",
    "        if count % BATCH_SIZE == 0:\n",
    "            index_batch(docs)\n",
    "            docs = []\n",
    "            print('Indexed {} documents.'.format(count))\n",
    "    if docs:\n",
    "        index_batch(docs)\n",
    "        print('Indexed {} documents.'.format(count))\n",
    "    \n",
    "    es_client.indices.refresh(index=INDEX_NAME)\n",
    "    print(\"Done indexing.\")\n",
    "\n",
    "    \n",
    "def index_batch(docs):\n",
    "    \"\"\" Indexes a batch of documents.\"\"\"\n",
    "    requests = []\n",
    "    for i, doc in enumerate(docs):\n",
    "        request = doc\n",
    "        request[\"_op_type\"] = \"index\"\n",
    "        request[\"_index\"] = INDEX_NAME\n",
    "        request[\"body\"] = doc['body']\n",
    "        requests.append(request)\n",
    "    bulk(es_client, requests)\n",
    "    \n",
    "def run_query_loop():\n",
    "    \"\"\" Asks user to enter a query to search.\"\"\"\n",
    "    while True:\n",
    "        try:\n",
    "            handle_query()\n",
    "        except KeyboardInterrupt:\n",
    "            break\n",
    "    return\n",
    "\n",
    "\n",
    "def handle_query():\n",
    "    \"\"\" Searches the user query and finds the best matches using elasticsearch.\"\"\"\n",
    "    query = input(\"Enter query: \")\n",
    "\n",
    "    search_start = time.time()\n",
    "    search = {\"size\": SEARCH_SIZE,\"query\": {\"match\": {\"body\": query}}}\n",
    "    print(search)\n",
    "    response = es_client.search(index=INDEX_NAME, body=json.dumps(search))\n",
    "    search_time = time.time() - search_start\n",
    "    print()\n",
    "    print(\"{} total hits.\".format(response[\"hits\"][\"total\"][\"value\"]))\n",
    "    print(\"search time: {:.2f} ms\".format(search_time * 1000))\n",
    "    for hit in response[\"hits\"][\"hits\"]:\n",
    "        print(\"id: {}, score: {}\".format(hit[\"_id\"], hit[\"_score\"]))\n",
    "        print(hit[\"_source\"])\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating `Question` index...\n",
      "index `Question` created successfully.\n",
      "Indexed 100000 documents.\n",
      "Indexed 200000 documents.\n",
      "Indexed 300000 documents.\n",
      "Indexed 400000 documents.\n",
      "Indexed 500000 documents.\n",
      "Indexed 600000 documents.\n",
      "Indexed 607282 documents.\n",
      "Done indexing.\n"
     ]
    }
   ],
   "source": [
    "INDEX_NAME = 'python_questions'\n",
    "es_client = Elasticsearch()\n",
    "create_index(es_client)\n",
    "index_data(es_client, data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter query:  how to loop over files in a directory\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'size': 3, 'query': {'match': {'body': 'how to loop over files in a directory'}}}\n",
      "\n",
      "10000 total hits.\n",
      "search time: 17.38 ms\n",
      "id: AnG2e3EBroreQxGKfg6x, score: 19.826128\n",
      "{'body': 'looping over filenames in python i have a zillion files in a directory i want a script to run on they all have a filename like prefix_foo_ _asdf_asdfasdf csv i know how to loop over files in a directory using a variable in the filename in shell but not python is there a corresponding way to do something like i for i lt process py prefix_foo_ i_ i endloop'}\n",
      "\n",
      "id: 8nW3e3EBroreQxGKGSnb, score: 19.142948\n",
      "{'body': 'iterate and delete the files in a directory i am trying to iterate over a few files from a directory then i am copying them in a group based on their initial name to a particular location and then deleting them from the current direcory but since i delete them after grouping them together i get a file not found exception when the loop move over to the next file which is deleted how can i resolve it here is my code import os import csv import glob fnmatch shutil time ftp_directory c cirp velocidata test ifind_location c cirp velocidata test getting the list of all files for root dirs files in os walk ftp_directory filtering for group names that are inprogress groups_beingworked for name in files getting the exception in the below line in the second iteration group name name lower find infile for loop in files if loop len group lower group groups_beingworked append loop for loop in groups_beingworked shutil copy os path join root loop ifind_location print deleting the file loop file removed from the ftp location through a ftp conn created elsewhere ftpconn delete os path join root loop list of sample files i am trying to go through is file new infile type file new infile type file old infile type file new infile type file old infile type file rec infile type'}\n",
      "\n",
      "id: fna3e3EBroreQxGKU39k, score: 18.82203\n",
      "{'body': 'python opening images in the fabio module by looping over a directory i m attempting to deal with cbf crystallographic binary format see below for link files in python i need a way of looping over all the files in the current directory example reading in first file in fabio dat raw_input please input required filename define the required filename as a string example input file cbf import fabio import fabio module for python img_ fabio open dat open image from defined filename this section of the code designed to open and display a file works perfectly fabio has a method of opening the next file available which is in this case of the format example img_ img_ next as i have already defined img_ in example this code would work how would i loop over all the files in the current directory without needing to execute the command in example for every file if there were files would it be something of the form example for i in range img_ i img_ i next how can i do this loop whilst also accounting for the leading zeros any help would be greatly appreciated thanks relevant information cbf files http www esrf eu computing forum imgcif cbf_definition html fabio module http pythonhosted org fabio getting_started html'}\n",
      "\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter query:  \n"
     ]
    }
   ],
   "source": [
    "SEARCH_SIZE = 3\n",
    "run_query_loop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra Resources\n",
    "\n",
    "[1] [Text Similarity Search for COVID-19 Dataset](https://colab.research.google.com/drive/1VJzTxN1QYXdc9LOjVO4DWqpV7nY4okmD#scrollTo=cSxeZmIZJ858)\n",
    "\n",
    "[2] [Text Embeddings in Elasticsearch](https://github.com/jtibshirani/text-embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}