Improving intrusion detection performance using bayesian hyperparameter optimization for supervised network traffic classification

Abstract

The rapid growth of networked systems has increased the complexity of network traffic and the risk of cyber-attacks, making intrusion detection more challenging. Machine learning approaches have been widely used to address this issue; however, their performance often depends on appropriate hyperparameter settings. This study examined the effect of Bayesian-based hyperparameter optimization on the performance of supervised machine learning models for network traffic classification. A publicly available dataset was used, consisting of various traffic-related features and labeled instances indicating normal or malicious activity. Several machine learning models, including Random Forest, Decision Tree, AdaBoost, Logistic Regression, Gradient Boosting, and Naïve Bayes, were evaluated. Each model was tested using default parameters and then optimized using Bayesian Optimization. The performance was assessed using accuracy, precision, recall, and F1-score. The results showed that ensemble-based models, particularly Gradient Boosting and Random Forest, achieved the best performance after optimization, with accuracy values above 89% and strong F1-scores. However, the findings also revealed a trade-off between precision and recall, where higher precision was often associated with lower detection of certain attack instances. In contrast, simpler models such as Logistic Regression showed lower performance, indicating their limitations in capturing complex patterns. Overall, the study demonstrated that Bayesian-based hyperparameter optimization contributed to improving model performance and provided a more reliable approach for network traffic classification.