1-data-collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b52b1d3a-4ded-43a1-a5a6-ecb90122dd00",
   "metadata": {},
   "source": [
    "![Data collection banner](./images/1_data_collection_banner.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "40efd49a-7cf0-4cc2-b87a-1834ae7db8a2",
   "metadata": {},
   "source": [
    "# 1. Data Collection\n",
    "\n",
    "Gather labeled data consisting of input features (independent variables) and their corresponding target labels or output variables (dependent variables). This labeled data is essential for training supervised learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfc9a354-14be-43f9-ad79-3bd433ca9a4e",
   "metadata": {},
   "source": [
    "## 1.1. Data Sources\n",
    "\n",
    "![Data sources](./images/1_data_collection_data_sources.png)\n",
    "\n",
    "### 1.1.1. Internal Data Sources\n",
    "\n",
    "- **Databases and Data Warehouses**: Many organizations have existing databases or data warehouses that store structured data from various business operations, transactions, or customer interactions. This data can be a valuable source for supervised learning problems, provided it contains relevant features and target variables.\n",
    "\n",
    "- **Log Files and Sensor Data**: For certain applications, log files from systems or sensor data from IoT devices can be used as data sources. These may require extensive preprocessing and feature engineering to extract meaningful information.\n",
    "\n",
    "### 1.1.2. External Data Sources\n",
    "\n",
    "- **Web Scraping**: Data can be extracted from websites, online forums, social media platforms, or other publicly available online sources through web scraping techniques. However, it's essential to ensure compliance with website terms of service and data privacy regulations.\n",
    "\n",
    "- **Crowdsourcing and Surveys**: Crowdsourcing platforms or online surveys can be used to collect labeled data from human participants. This approach is particularly useful when manual labeling or annotations are required, such as in computer vision or natural language processing tasks.\n",
    "\n",
    "- **Open Data Repositories**: Various organizations and communities maintain open data repositories containing datasets across different domains, such as healthcare, finance, or environmental sciences. These datasets can be used for supervised learning tasks, provided they align with the problem at hand.\n",
    "\n",
    "- **Data Marketplaces**: There are commercial data marketplaces where organizations or individuals can purchase datasets for specific use cases or domains. These datasets are often curated, cleaned, and labeled, but they can be expensive."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36195f76-f806-4514-a49d-63e9a1b87a26",
   "metadata": {},
   "source": [
    "\n",
    "-----\n",
    "\n",
    "## 1.2. Data Collection Considerations\n",
    "\n",
    "- **Data Quality**: Ensure that the collected data is accurate, complete, and free from errors or inconsistencies. Data quality issues can significantly impact the performance of supervised learning models.\n",
    "\n",
    "- **Data Relevance**: Carefully evaluate the relevance of the collected data to the problem you are trying to solve. Irrelevant or redundant features can introduce noise and negatively impact model performance.\n",
    "\n",
    "- **Data Representativeness**: The collected data should be representative of the real-world scenario or population you want to model. Biased or skewed data can lead to models that perform poorly on unseen data.\n",
    "\n",
    "- **Data Privacy and Ethics**: When collecting data, especially from individuals, it's crucial to consider data privacy regulations, such as GDPR or CCPA, and ensure that the data collection process adheres to ethical principles and guidelines.\n",
    "\n",
    "- **Data Labeling**: For supervised learning, the collected data must be labeled with the corresponding target variables or output classes. This labeling process can be time-consuming and may require domain expertise or crowdsourcing efforts.\n",
    "\n",
    "- **Data Versioning and Documentation**: Maintain clear documentation and versioning of the collected data, including metadata, data dictionaries, and any preprocessing or cleaning steps applied. This ensures reproducibility and transparency in the modeling process.\n",
    "\n",
    "By considering these factors and leveraging appropriate data collection sources, you can ensure that the collected data is suitable for training supervised learning models and can lead to accurate and reliable predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "296e9227-ece8-40dc-9d0a-f379ef532284",
   "metadata": {},
   "source": [
    "![Data sources](./images/1_data_collection_data_meme.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5687081d-89ac-4938-8d2c-9ff4fdf85663",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}