Projeto_Web_Scraping_NBA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "267f5d33-7f70-4e54-860d-c9c6730ae7f8",
   "metadata": {},
   "source": [
    "# Web Scraping - Dados NBA\n",
    "\n",
    "## Objetivo\n",
    "\n",
    "> O objetivo com esse projeto é extrair dados do site [Sports Reference](https://www.sports-reference.com/). No caso, escolhi coletar dados do basquete masculino [NBA](https://www.basketball-reference.com/). \n",
    "Faremos um processo de ETL, depois analisaremos os dados, buscando informações que respondam nossas perguntas.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ec14cc63-6ba7-4834-93ff-01681a724c7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Carregar Bibliotecas\n",
    "\n",
    "from urllib.request import urlopen\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e67f6de9-2d26-43c6-98af-cc0818d1e771",
   "metadata": {},
   "source": [
    "### ETL"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e24a5e9-c5ff-4bc4-b386-7715159d8687",
   "metadata": {},
   "source": [
    "> Extrair, transformar e carregar (ETL) é o processo que as organizações orientadas a dados usam para coletar dados de várias fontes e reuni-los para dar suporte à descoberta, à geração de relatórios, à análise e à tomada de decisões.\n",
    "\n",
    "Nessa etapa faremos o processo de ETL, de maneira que nosso código fique otimizado, possamos carregar qualquer dataframe, dentro de 'year_range' ou seja de 2000 à 2022."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c070935f-9470-452f-9659-f6a72f76cb6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Range com temporadas de 2000 à 2022:\n",
    "from datetime import date\n",
    "start = date(2000, 1, 1)\n",
    "end = date(2022, 1, 1)\n",
    "\n",
    "year_range = [year for year in range(start.year, end.year + 1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "393a6fbb-d37e-467e-a35d-b8c0ef4c4c2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "names = ['Season_{}'.format(i) for i in year_range]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3a68afee-1b82-4c7d-b45a-13ae5de64775",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extraindo os dados e visualizando em dataframes\n",
    "for i in range(len(year_range)):\n",
    "    url = 'https://www.basketball-reference.com/leagues/NBA_{}_per_game.html'.format(year_range[i])\n",
    "    \n",
    "    #this is HTML from given URL\n",
    "    html = urlopen(url)\n",
    "    soup = BeautifulSoup(html)\n",
    "    headers = [th.get_text() for th in soup.find_all('tr', limit = 2)[0].find_all('th')]\n",
    "    headers = headers[1:]\n",
    "    \n",
    "    #avoid the first header row\n",
    "    rows = soup.find_all('tr')[1:]\n",
    "    stats = [[td.get_text() for td in rows[j].find_all('td')]\n",
    "                   for j in range(len(rows))]\n",
    "    locals()[names[i]] = pd.DataFrame(stats, columns = headers)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "daa96ec6-3896-49a9-ab0c-9aeec06ee4c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Player</th>\n",
       "      <th>Pos</th>\n",
       "      <th>Age</th>\n",
       "      <th>Tm</th>\n",
       "      <th>G</th>\n",
       "      <th>GS</th>\n",
       "      <th>MP</th>\n",
       "      <th>FG</th>\n",
       "      <th>FGA</th>\n",
       "      <th>FG%</th>\n",
       "      <th>...</th>\n",
       "      <th>FT%</th>\n",
       "      <th>ORB</th>\n",
       "      <th>DRB</th>\n",
       "      <th>TRB</th>\n",
       "      <th>AST</th>\n",
       "      <th>STL</th>\n",
       "      <th>BLK</th>\n",
       "      <th>TOV</th>\n",
       "      <th>PF</th>\n",
       "      <th>PTS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Tariq Abdul-Wahad</td>\n",
       "      <td>SG</td>\n",
       "      <td>25</td>\n",
       "      <td>TOT</td>\n",
       "      <td>61</td>\n",
       "      <td>56</td>\n",
       "      <td>25.9</td>\n",
       "      <td>4.5</td>\n",
       "      <td>10.6</td>\n",
       "      <td>.424</td>\n",
       "      <td>...</td>\n",
       "      <td>.756</td>\n",
       "      <td>1.7</td>\n",
       "      <td>3.1</td>\n",
       "      <td>4.8</td>\n",
       "      <td>1.6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.7</td>\n",
       "      <td>2.4</td>\n",
       "      <td>11.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tariq Abdul-Wahad</td>\n",
       "      <td>SG</td>\n",
       "      <td>25</td>\n",
       "      <td>ORL</td>\n",
       "      <td>46</td>\n",
       "      <td>46</td>\n",
       "      <td>26.2</td>\n",
       "      <td>4.8</td>\n",
       "      <td>11.2</td>\n",
       "      <td>.433</td>\n",
       "      <td>...</td>\n",
       "      <td>.762</td>\n",
       "      <td>1.7</td>\n",
       "      <td>3.5</td>\n",
       "      <td>5.2</td>\n",
       "      <td>1.6</td>\n",
       "      <td>1.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.9</td>\n",
       "      <td>2.5</td>\n",
       "      <td>12.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Tariq Abdul-Wahad</td>\n",
       "      <td>SG</td>\n",
       "      <td>25</td>\n",
       "      <td>DEN</td>\n",
       "      <td>15</td>\n",
       "      <td>10</td>\n",
       "      <td>24.9</td>\n",
       "      <td>3.4</td>\n",
       "      <td>8.7</td>\n",
       "      <td>.389</td>\n",
       "      <td>...</td>\n",
       "      <td>.738</td>\n",
       "      <td>1.6</td>\n",
       "      <td>1.9</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.7</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1.3</td>\n",
       "      <td>2.1</td>\n",
       "      <td>8.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shareef Abdur-Rahim</td>\n",
       "      <td>SF</td>\n",
       "      <td>23</td>\n",
       "      <td>VAN</td>\n",
       "      <td>82</td>\n",
       "      <td>82</td>\n",
       "      <td>39.3</td>\n",
       "      <td>7.2</td>\n",
       "      <td>15.6</td>\n",
       "      <td>.465</td>\n",
       "      <td>...</td>\n",
       "      <td>.809</td>\n",
       "      <td>2.7</td>\n",
       "      <td>7.4</td>\n",
       "      <td>10.1</td>\n",
       "      <td>3.3</td>\n",
       "      <td>1.1</td>\n",
       "      <td>1.1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>20.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cory Alexander</td>\n",
       "      <td>PG</td>\n",
       "      <td>26</td>\n",
       "      <td>DEN</td>\n",
       "      <td>29</td>\n",
       "      <td>2</td>\n",
       "      <td>11.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.4</td>\n",
       "      <td>.286</td>\n",
       "      <td>...</td>\n",
       "      <td>.773</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.2</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.3</td>\n",
       "      <td>2.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Player Pos Age   Tm   G  GS    MP   FG   FGA   FG%  ...   FT%  \\\n",
       "0    Tariq Abdul-Wahad  SG  25  TOT  61  56  25.9  4.5  10.6  .424  ...  .756   \n",
       "1    Tariq Abdul-Wahad  SG  25  ORL  46  46  26.2  4.8  11.2  .433  ...  .762   \n",
       "2    Tariq Abdul-Wahad  SG  25  DEN  15  10  24.9  3.4   8.7  .389  ...  .738   \n",
       "3  Shareef Abdur-Rahim  SF  23  VAN  82  82  39.3  7.2  15.6  .465  ...  .809   \n",
       "4       Cory Alexander  PG  26  DEN  29   2  11.3  1.0   3.4  .286  ...  .773   \n",
       "\n",
       "   ORB  DRB   TRB  AST  STL  BLK  TOV   PF   PTS  \n",
       "0  1.7  3.1   4.8  1.6  1.0  0.5  1.7  2.4  11.4  \n",
       "1  1.7  3.5   5.2  1.6  1.2  0.3  1.9  2.5  12.2  \n",
       "2  1.6  1.9   3.5  1.7  0.4  0.8  1.3  2.1   8.9  \n",
       "3  2.7  7.4  10.1  3.3  1.1  1.1  3.0  3.0  20.3  \n",
       "4  0.3  1.2   1.4  2.0  0.8  0.1  1.0  1.3   2.8  \n",
       "\n",
       "[5 rows x 29 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Temporada 2000\n",
    "Season_2000.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8bd7c547-e52c-4f92-8096-2625f0ddeb1f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Player</th>\n",
       "      <th>Pos</th>\n",
       "      <th>Age</th>\n",
       "      <th>Tm</th>\n",
       "      <th>G</th>\n",
       "      <th>GS</th>\n",
       "      <th>MP</th>\n",
       "      <th>FG</th>\n",
       "      <th>FGA</th>\n",
       "      <th>FG%</th>\n",
       "      <th>...</th>\n",
       "      <th>FT%</th>\n",
       "      <th>ORB</th>\n",
       "      <th>DRB</th>\n",
       "      <th>TRB</th>\n",
       "      <th>AST</th>\n",
       "      <th>STL</th>\n",
       "      <th>BLK</th>\n",
       "      <th>TOV</th>\n",
       "      <th>PF</th>\n",
       "      <th>PTS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Precious Achiuwa</td>\n",
       "      <td>C</td>\n",
       "      <td>22</td>\n",
       "      <td>TOR</td>\n",
       "      <td>73</td>\n",
       "      <td>28</td>\n",
       "      <td>23.6</td>\n",
       "      <td>3.6</td>\n",
       "      <td>8.3</td>\n",
       "      <td>.439</td>\n",
       "      <td>...</td>\n",
       "      <td>.595</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>6.5</td>\n",
       "      <td>1.1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.6</td>\n",
       "      <td>1.2</td>\n",
       "      <td>2.1</td>\n",
       "      <td>9.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Steven Adams</td>\n",
       "      <td>C</td>\n",
       "      <td>28</td>\n",
       "      <td>MEM</td>\n",
       "      <td>76</td>\n",
       "      <td>75</td>\n",
       "      <td>26.3</td>\n",
       "      <td>2.8</td>\n",
       "      <td>5.1</td>\n",
       "      <td>.547</td>\n",
       "      <td>...</td>\n",
       "      <td>.543</td>\n",
       "      <td>4.6</td>\n",
       "      <td>5.4</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.4</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Bam Adebayo</td>\n",
       "      <td>C</td>\n",
       "      <td>24</td>\n",
       "      <td>MIA</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>32.6</td>\n",
       "      <td>7.3</td>\n",
       "      <td>13.0</td>\n",
       "      <td>.557</td>\n",
       "      <td>...</td>\n",
       "      <td>.753</td>\n",
       "      <td>2.4</td>\n",
       "      <td>7.6</td>\n",
       "      <td>10.1</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>19.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Santi Aldama</td>\n",
       "      <td>PF</td>\n",
       "      <td>21</td>\n",
       "      <td>MEM</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>11.3</td>\n",
       "      <td>1.7</td>\n",
       "      <td>4.1</td>\n",
       "      <td>.402</td>\n",
       "      <td>...</td>\n",
       "      <td>.625</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.7</td>\n",
       "      <td>2.7</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.1</td>\n",
       "      <td>4.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LaMarcus Aldridge</td>\n",
       "      <td>C</td>\n",
       "      <td>36</td>\n",
       "      <td>BRK</td>\n",
       "      <td>47</td>\n",
       "      <td>12</td>\n",
       "      <td>22.3</td>\n",
       "      <td>5.4</td>\n",
       "      <td>9.7</td>\n",
       "      <td>.550</td>\n",
       "      <td>...</td>\n",
       "      <td>.873</td>\n",
       "      <td>1.6</td>\n",
       "      <td>3.9</td>\n",
       "      <td>5.5</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1.7</td>\n",
       "      <td>12.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              Player Pos Age   Tm   G  GS    MP   FG   FGA   FG%  ...   FT%  \\\n",
       "0   Precious Achiuwa   C  22  TOR  73  28  23.6  3.6   8.3  .439  ...  .595   \n",
       "1       Steven Adams   C  28  MEM  76  75  26.3  2.8   5.1  .547  ...  .543   \n",
       "2        Bam Adebayo   C  24  MIA  56  56  32.6  7.3  13.0  .557  ...  .753   \n",
       "3       Santi Aldama  PF  21  MEM  32   0  11.3  1.7   4.1  .402  ...  .625   \n",
       "4  LaMarcus Aldridge   C  36  BRK  47  12  22.3  5.4   9.7  .550  ...  .873   \n",
       "\n",
       "   ORB  DRB   TRB  AST  STL  BLK  TOV   PF   PTS  \n",
       "0  2.0  4.5   6.5  1.1  0.5  0.6  1.2  2.1   9.1  \n",
       "1  4.6  5.4  10.0  3.4  0.9  0.8  1.5  2.0   6.9  \n",
       "2  2.4  7.6  10.1  3.4  1.4  0.8  2.6  3.1  19.1  \n",
       "3  1.0  1.7   2.7  0.7  0.2  0.3  0.5  1.1   4.1  \n",
       "4  1.6  3.9   5.5  0.9  0.3  1.0  0.9  1.7  12.9  \n",
       "\n",
       "[5 rows x 29 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Temporada 2022\n",
    "Season_2022.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "7d114cef-ef8f-4a2c-927e-cf346040cba4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Salvando todos dataframes em CSV:\n",
    "\n",
    "Season_2000.to_csv('Season_2000.csv')\n",
    "Season_2001.to_csv('Season_2001.csv')\n",
    "Season_2002.to_csv('Season_2002.csv')\n",
    "Season_2003.to_csv('Season_2003.csv')\n",
    "Season_2004.to_csv('Season_2004.csv')\n",
    "Season_2005.to_csv('Season_2005.csv')\n",
    "Season_2006.to_csv('Season_2006.csv')\n",
    "Season_2007.to_csv('Season_2007.csv')\n",
    "Season_2008.to_csv('Season_2008.csv')\n",
    "Season_2009.to_csv('Season_2009.csv')\n",
    "Season_2010.to_csv('Season_2010.csv')\n",
    "Season_2011.to_csv('Season_2011.csv')\n",
    "Season_2012.to_csv('Season_2012.csv')\n",
    "Season_2013.to_csv('Season_2013.csv')\n",
    "Season_2014.to_csv('Season_2014.csv')\n",
    "Season_2015.to_csv('Season_2015.csv')\n",
    "Season_2016.to_csv('Season_2016.csv')\n",
    "Season_2017.to_csv('Season_2017.csv')\n",
    "Season_2018.to_csv('Season_2018.csv')\n",
    "Season_2019.to_csv('Season_2019.csv')\n",
    "Season_2020.to_csv('Season_2020.csv')\n",
    "Season_2021.to_csv('Season_2021.csv')\n",
    "Season_2022.to_csv('Season_2022.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}