Malware-Detection-Using-Byte-Codes

Malware Classification using Convolutional Neural Networks (CNN) with PyTorch/Keras

Introduction:

This project aims to build a Convolutional Neural Network (CNN) for classifying malware into 9 distinct categories. We leverage the PyTorch/Keras framework for model development and evaluation. The dataset used is the Microsoft Malware Classification Challenge from Kaggle, preprocessed into 32x32 grayscale images with corresponding labels. The primary goal is to explore different techniques to improve the model's performance.

Objective:

The objective of this project is to develop an efficient malware classification model by:

1. Data Preparation: Loading the dataset and dividing it into training (80%), validation (10%), and testing (10%) sets.
2. CNN Architecture: Defining a parameterizable CNN model, equipped with Convolutional layers, a Fully Connected layer, and ReLU activation (except for the final Softmax layer) for classification.
3. Learning Rate Exploration: Training the model using Stochastic Gradient Descent (SGD) optimizer with fixed learning rates (0.01 and 0.001) for 20 epochs each. Implementing a learning rate scheduler to adjust the learning rate every 5 epochs and studying its impact on training and validation accuracy.
4. Regularization: Evaluating the effect of L2 regularization with different coefficients (ranging from 0 to 1.0). Identifying the optimal regularization coefficient through a detailed search and training the model accordingly.
5. Early Stopping: Implementing early stopping to halt training when validation accuracy fails to improve for three consecutive epochs.
6. Hyperparameter Tuning: Utilizing Keras Tuner or Ray Tune to optimize learning rate and regularization coefficients.
7. Dropout and Batch Normalization: Analyzing the effect of Dropout and Batch Normalization layers on the model's performance. Evaluating the combined impact of both layers on the model.
8. Transfer Learning: Utilizing the MobileNet-v2 model for transfer learning and fine-tuning the last layers for classification.
9. Model Conversion: Converting the model from PyTorch to Keras or vice versa and optimizing hyperparameters using Keras Tuner or Ray Tune.

Conclusion:

This GitHub project provides a comprehensive exploration of building a robust malware classification model using CNNs. Various techniques such as learning rate exploration, regularization, early stopping, hyperparameter tuning, and transfer learning are employed to enhance the model's accuracy and generalization. Developers interested in the field of malware classification can fork and contribute to this project to improve the model further. The final report and insights gained from each experiment are available in the repository, making it a valuable resource for machine learning enthusiasts.