Encrypted Remote Access Trojan Detection: A Machine Learning Approach with Real-World and Open Datasets

Abstract

The increasing use of encryption by cyber attackers to conceal Remote Access Trojans (RATs) challenges traditional signature-based detection systems, which struggle with encrypted traffic and leave security gaps. In this study, we propose a privacy-preserving, machine-learning-based framework that detects encrypted RATs without decrypting traffic. Instead, it analyzes behavioral indicators and metadata, including packet timing anomalies, TLS handshake irregularities, and persistent unidirectional flows. We evaluated our approach using two datasets: a public Kaggle dataset (177,482 labeled records, 85 features) and an anonymized internal dataset from Company X (40,000 samples, 27 features). Among four tested models—Logistic Regression, Decision Tree, Random Forest, and XGBoost—Random Forest performed best, achieving 74.83% and 72.11% accuracy on the Company X and Kaggle datasets, respectively, outperforming a baseline signature-based system (53.8% accuracy). Our model also showed strong generalization, with 80% correct predictions across sample-based evaluations, demonstrating its readiness for real-world deployment. By ensuring privacy and delivering improved detection, our framework offers a scalable, adaptive alternative to traditional cybersecurity methods.