Added a CI/CD pipeline for code testing ✅

.github/workflows/code_quality.yml

@@ -29,7 +29,7 @@ jobs:
     - name: Auto-fix Linting Errors with Ruff
       run: ruff --fix .
     - name: Security Check with Bandit
-      run: bandit -r .
+      run: bandit -r . --exclude ./tests/
     - name: Commit & Push changes
       uses: actions-js/push@v1.4
       with:

.github/workflows/test_app.yml

@@ -0,0 +1,38 @@
+name: LLM RAG testing
+
+on:
+    push:
+      branches:
+        - main
+        - dev
+        - test-and-cicd-workflow
+    pull_request:
+      branches:
+        - main
+        - dev
+        - test-and-cicd-workflow
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.8'
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+        pip install pytest
+
+    - name: Run tests
+      env:
+        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+      run: |
+        pytest tests/

.gitignore

@@ -44,8 +44,10 @@ temp.txt
 # Secret files
 secrets/credentials.yml
 
-# generated text files
-data/processed/*.text
+# generated json files
+data/processed/*.json
+!data/processed/faiss_db_mock_markdown.json
 
 # generated bin files
-data/processed/*.bin
+data/processed/*.bin
+!data/processed/faiss_db_mock_markdown.bin

Makefile

@@ -47,11 +47,17 @@ jupy:
 	@echo ">>>>>> Jupyter Lab is running <<<<<<"
 
 app:
+	@$(VENV_ACTIVATE)
 	@python app.py
 
+test:
+	@$(VENV_ACTIVATE)
+	@pytest tests/
+
 # Display available make targets
 help:
 	@echo Available targets:
 	@echo   make jupy                                             - Activate the virtual environment and run Jupyter Lab
+	@echo   make test                                             - Tests the code using pytest
 	@echo   make app                                              - Runs the App
 	@echo Author: $(AUTHOR)

data/processed/faiss_db_mock_markdown.json

@@ -0,0 +1 @@
+["### diagnostics artificial intelligence (ai) in healthcare is revolutionizing the field with its ability to make faster and more accurate diagnoses. deep learning algorithms are now able to detect anomalies in medical imaging, such as mris and x-rays, which helps in early detection of diseases like cancer.", "### prognostics predictive analytics powered by ai is enabling healthcare providers to foresee patient outcomes more effectively. by analyzing large datasets, ai can identify patterns that human analysts might miss, leading to better patient care.", "### personalized medicine ai is the driving force behind personalized treatment plans. by taking into account a patient's genetic makeup, lifestyle, and other factors, ai systems can tailor treatments to individual needs, improving the efficacy of interventions.", "### algorithmic trading in the world of finance, ai excels at high-frequency trading. algorithms can execute trades at superhuman speeds, reacting to market changes instantaneously and optimizing portfolios in real-time.", "### fraud detection ai systems are skilled at identifying fraudulent activities. by analyzing spending patterns and comparing them against known fraudulent behaviors, ai can flag suspicious transactions with high accuracy.", "### customer service financial institutions are employing ai-powered chatbots and virtual assistants to provide 24/7 customer service. these systems are capable of handling a wide range of customer inquiries, from account balances to complex loan advice."]

data/raw/mock_markdown.md

@@ -1,205 +1,17 @@
-## Healthcare
+### Diagnostics
+Artificial intelligence (AI) in healthcare is revolutionizing the field with its ability to make faster and more accurate diagnoses. Deep learning algorithms are now able to detect anomalies in medical imaging, such as MRIs and X-rays, which helps in early detection of diseases like cancer.
 
-In healthcare, AI is being used to make more accurate diagnoses, predict patient outcomes, and personalize patient treatment plans.
+### Prognostics
+Predictive analytics powered by AI is enabling healthcare providers to foresee patient outcomes more effectively. By analyzing large datasets, AI can identify patterns that human analysts might miss, leading to better patient care.
 
-## Finance
+### Personalized Medicine
+AI is the driving force behind personalized treatment plans. By taking into account a patient's genetic makeup, lifestyle, and other factors, AI systems can tailor treatments to individual needs, improving the efficacy of interventions.
 
-The finance sector leverages AI for algorithmic trading, fraud detection, and customer service, enhancing efficiency and reducing risk.
+### Algorithmic Trading
+In the world of finance, AI excels at high-frequency trading. Algorithms can execute trades at superhuman speeds, reacting to market changes instantaneously and optimizing portfolios in real-time.
 
-# Ethical Considerations
+### Fraud Detection
+AI systems are skilled at identifying fraudulent activities. By analyzing spending patterns and comparing them against known fraudulent behaviors, AI can flag suspicious transactions with high accuracy.
 
-As AI continues to evolve, ethical considerations remain at the forefront of the technology's development and deployment.
-
-## Privacy and Surveillance
-
-With the increasing capability of AI in processing and analyzing large amounts of personal data, privacy and surveillance concerns are heightened.
-
-## Bias and Fairness
-
-AI systems can inherit biases present in their training data, leading to concerns about fairness and discrimination in AI decision-making.
-
-# Conclusion
-
-The rapid advancement of AI presents both opportunities and challenges. As we continue to explore the boundaries of what AI can achieve, it is crucial to address the ethical implications and ensure the technology is used for the betterment of society.
-
-# Introduction to Renewable Energy
-
-Renewable energy sources such as solar, wind, and hydro power offer sustainable alternatives to fossil fuels. They reduce greenhouse gas emissions and our dependence on non-renewable resources.
-
-## Solar Power
-
-Solar energy is harnessed from the sun using photovoltaic cells. It's a clean, abundant source of power that can significantly reduce electricity bills.
-
-### Global Solar Initiatives
-
-Countries around the world are investing in solar farms and technology to increase their share of energy production from solar sources.
-
-## Wind Energy
-
-Wind turbines convert kinetic energy from the wind into electrical power. Wind energy is a fast-growing sector, contributing to a significant portion of the global energy mix.
-
-### Offshore Wind Farms
-
-Offshore wind farms, built on the sea or large lakes, can generate more power than land-based turbines due to stronger and more consistent winds.
-
-# Introduction to AI
-
-Artificial intelligence (AI) has rapidly become a key technology in many industries, revolutionizing processes and efficiency.
-
-## History of AI
-
-The concept of artificial intelligence has been around for centuries, but it wasn't until the 20th century that it became a field of study. Alan Turing, a British mathematician and logician, laid the groundwork for modern computing and theorized about machines that could think.
-
-### Turing Test
-
-The Turing Test, proposed by Alan Turing in 1950, was designed to assess a machine's ability to exhibit intelligent behavior indistinguishable from that of a human.
-
-## Machine Learning
-
-Machine learning, a subset of AI, involves the development of algorithms that allow computers to learn from and make predictions or decisions based on data.
-
-### Deep Learning
-
-Deep learning is a machine learning technique that teaches computers to do what comes naturally to humans: learn by example. Deep learning is a key technology behind driverless cars, enabling them to recognize a stop sign or to distinguish a pedestrian from a lamppost.
-
-# Applications of AI
-
-AI technology is not just a scientific discipline, but a driver of high-tech innovation and real-world applications.
-
-# The Future of Transportation
-
-Advancements in technology are transforming how we move. Electric vehicles (EVs), autonomous driving, and high-speed rail systems represent the future of transportation.
-
-## Electric Vehicles
-
-EVs are gaining popularity as they become more affordable and infrastructure for charging stations expands.
-
-### Tesla's Innovations
-
-Tesla has been at the forefront of the electric vehicle revolution, pushing the boundaries of what's possible with EV technology.
-
-## Autonomous Vehicles
-
-Self-driving cars aim to make transportation safer and more efficient. Companies like Waymo and Uber are leading the way in autonomous vehicle technology.
-
-# Addressing Climate Change
-
-Climate change is one of the most pressing issues of our time. Actions to mitigate its impact range from international agreements like the Paris Accord to local sustainability initiatives.
-
-## Carbon Footprint Reduction
-
-Individuals and organizations alike are striving to reduce their carbon footprints by optimizing energy use and adopting green technologies.
-
-### Renewable Energy Adoption
-
-The transition to renewable energy sources is key to combating climate change. Governments and private sectors are incentivizing the shift to greener alternatives.
-
-# Cultural Shifts in the Digital Age
-
-The digital revolution has altered every aspect of our lives, from how we communicate to how we work and entertain ourselves.
-
-## The Rise of Social Media
-
-Platforms like Facebook, Twitter, and Instagram have changed the way we connect with others and consume information.
-
-### Influencer Culture
-
-Influencers on social media platforms have a significant impact on their followers' choices, shaping trends in everything from fashion to politics.
-
-## Remote Work Trends
-
-The COVID-19 pandemic accelerated the adoption of remote work, showcasing the potential for flexibility in how and where we work.
-
-### Coworking Spaces
-
-As remote work becomes more common, coworking spaces offer a solution for those seeking structure and community outside of the traditional office environment.
-
-# Comprehensive Guide on Health and Fitness
-
-Health and fitness have become central to modern lifestyle, emphasizing the importance of regular exercise, balanced diet, and mental well-being.
-
-## Benefits of Regular Exercise
-
-Engaging in regular physical activity can reduce the risk of chronic diseases, improve balance and coordination, enhance mental health, and extend lifespan.
-
-### Types of Physical Activities
-
-- **Aerobic Exercises**: Enhance heart and lung health and improve stamina.
-- **Strength Training**: Builds muscle strength and improves metabolism.
-- **Flexibility Exercises**: Increases range of motion and reduces injury risks.
-
-## Importance of a Balanced Diet
-
-A balanced diet provides the body with essential nutrients, supports immune system, and maintains healthy body weight.
-
-### Superfoods and Their Benefits
-
-- **Berries**: Rich in antioxidants.
-- **Leafy Greens**: High in vitamins and minerals.
-- **Nuts and Seeds**: Excellent sources of healthy fats and proteins.
-
-# The Fascinating World of Space Exploration
-
-Space exploration unlocks mysteries of the cosmos, from the origins of the universe to the potential for life on other planets.
-
-## Milestones in Space Exploration
-
-- **The Moon Landing (1969)**: Apollo 11 mission marked the first human moon landing.
-- **Voyager Probes**: Have traveled beyond the solar system, sending back invaluable data.
-
-### The Role of Satellites
-
-Satellites revolutionize communication, weather forecasting, and global navigation, playing a crucial role in modern life.
-
-## Future Missions to Mars
-
-The quest to land humans on Mars represents the next frontier for space exploration, with potential missions planned within the next decade.
-
-# Understanding Modern Art
-
-Modern art represents a radical break from traditional forms, embracing abstract and experimental expressions.
-
-## Key Movements in Modern Art
-
-- **Impressionism**: Focused on capturing light and natural forms.
-- **Cubism**: Introduced by Picasso, revolutionized perspective with abstract forms.
-
-### The Impact of Digital Art
-
-The digital age introduces new mediums for artistic expression, blending technology and creativity in innovative ways.
-
-# Global Environmental Challenges
-
-Addressing global environmental challenges requires international cooperation and sustainable practices to protect natural resources.
-
-## Climate Change and Global Warming
-
-The rising levels of greenhouse gases are leading to global warming, affecting weather patterns, sea levels, and biodiversity.
-
-### Renewable Energy Solutions
-
-Investing in solar, wind, hydro, and geothermal energy can significantly reduce carbon footprint and combat climate change.
-
-# Advances in Technology and Innovation
-
-Technological advances continue to reshape industries, from artificial intelligence and robotics to biotechnology and nanotechnology.
-
-## The Rise of Artificial Intelligence
-
-AI is transforming business, healthcare, and daily life, offering new possibilities in automation and smart technology.
-
-### The Potential of Quantum Computing
-
-Quantum computing promises to revolutionize data processing, with implications for cryptography, materials science, and complex system simulation.
-
-# Cultural Evolution in the 21st Century
-
-The 21st century witnesses significant cultural shifts influenced by globalization, digital connectivity, and social change.
-
-## The Influence of Social Media
-
-Social media platforms play a pivotal role in shaping public opinion, cultural trends, and personal identity.
-
-### The Power of Crowdsourcing
-
-Crowdsourcing leverages the collective intelligence of the internet, enabling innovation, problem-solving, and funding for creative projects.
+### Customer Service
+Financial institutions are employing AI-powered chatbots and virtual assistants to provide 24/7 customer service. These systems are capable of handling a wide range of customer inquiries, from account balances to complex loan advice.

src/utils/utils.py

@@ -1,16 +1,15 @@
 import yaml
+import os
 
 
-def load_credentials(credentials_path):
+def load_credentials(credentials_path="secrets/credentials.yml"):
     """
-    Load credentials from a YAML file.
-
-    Parameters:
-    credentials_path (str): The file path to the credentials YAML file.
-
-    Returns:
-    dict: The loaded credentials.
+    Load credentials from the environment or a YAML file. ( depends on the environment )
     """
+    openai_api_key = os.getenv('OPENAI_API_KEY')
+    if openai_api_key:
+        return {'OPENAI_CREDENTIALS': openai_api_key}
+
     with open(credentials_path, "r") as credentials_file:
         return yaml.safe_load(credentials_file)
 

tests/test_index_creation.py

@@ -0,0 +1,27 @@
+from src.pipelines.query_pipeline import QueryPipeline
+from src.utils.utils import load_models_config, load_credentials
+import os
+
+def test_index_creation(tmpdir):
+    output_directory = tmpdir.strpath
+    credentials = load_credentials("secrets/credentials.yml")
+    openai_api_key = credentials["OPENAI_CREDENTIALS"]
+    models_config = load_models_config("config/models_config.yml")
+
+    query_pipeline = QueryPipeline(openai_api_key, models_config)
+
+    total_cost = query_pipeline.setup_semantic_database(
+        markdown_path="data/raw/mock_markdown.md",
+        embedding_model="text-embedding-3-small",
+        save_index=True,
+        directory_path=output_directory,
+    )
+
+    # Check if the total cost is a float number
+    assert isinstance(total_cost, float)
+
+    # Check if the index file (.bin) and texts file (.json) are created in the directory
+    files = os.listdir(output_directory)
+    print(files)
+    assert any(os.path.splitext(f)[1] == ".bin" for f in files)
+    assert any(os.path.splitext(f)[1] == ".json" for f in files)