Today, the majority of internet sessions are encrypted, with over half of all online traffic relying on TLS encryption for secure client-server communication. Services like Let's Encrypt have made it more accessible and cost-effective to implement HTTPS, ensuring our online privacy. However, this encryption also poses challenges, particularly for security professionals working to detect encrypted command-and-control (C&C) channels used by cyber attackers. In this blog post, we'll explore the importance of C&C detection, the role of machine learning in security, and how Vectra AI is leading the way in enhancing online security.
The attacker perspective
Attackers thrive on anonymity and seek ways to blend into the digital landscape. They have a variety of options at their disposal for establishing command-and-control (C&C) channels, as evidenced by the MITRE ATT&CK framework. Multistage channels, data encoding, and multi-hop proxies are among their favorites. Attackers can leverage standard encrypted protocols like TLS or even create custom protocols. They may also encrypt or encode data within their channels before sending it through an encrypted TLS tunnel.
The security team perspective
From the perspective of security analysts, uncovering an attacker's C&C channels is a critical task, even when traffic is encrypted. To achieve this, the assumption must be that encryption is in place, necessitating a reliance on accessible metadata. When everything is encrypted, the remaining data to examine involves the flow of information over time.
Fortunately, Vectra AI's flow engine offers detailed time-domain data for each tracked data flow, with sampling as granular as half-second intervals. This includes metrics such as bytes sent and received over time, providing insight into the dynamics of each interaction.
Machine Learning for C&C Detection
Vectra AI's journey in effectively utilizing time-domain data to detect command-and-control channels has seen various approaches, with a strong focus on supervised machine learning techniques. Several machine learning algorithms working in tandem offer both a coarse and granular view of attacker behaviors:
- Random Forests: Comprising multiple decision trees, random forests excel at delivering coarse-grained telemetry. They monitor time-series windows and track over 20 features, including client/server data ratios, data consistency, server break frequencies, and session length.
- Recurrent Neural Networks (RNNs): RNNs enable the representation of temporal behaviors, where one sequence influences the next, revealing unique human-driven features. Essentially, RNNs mimic human memory.
- Long Short-Term Memory (LSTM) Deep Learning Neural Networks: LSTM networks can learn long-range temporal dependencies and relationships, including the ability to forget. This approach is versatile and applicable across various use cases, including natural language processing. In security, it allows for tracking relevant attacker activity patterns over extended periods.
The Power of Convergence
The convergence of these machine learning methods, each meticulously developed and fine-tuned over the years, empowers Vectra AI to comprehend the behaviors of encrypted traffic and generate high-fidelity alerts for command-and-control detection. Success hinges on the careful application of these methods and the quality of training data used.
Vectra AI is fortunate to have an award-winning team of data scientists and security researchers dedicated to algorithm development, sample curation, and ongoing algorithm optimization for peak performance.
Conclusion
Detecting encrypted command-and-control channels with precision and minimal noise is a formidable challenge. It goes beyond the capabilities of a simple anomaly detector. Achieving this level of security requires the right data, the most suitable machine learning methods, and a dedicated team with expertise in building and fine-tuning models. Vectra AI stands at the forefront of this mission, contributing to a safer online environment for all.
▼ Read more on how Vectra AI uses AI to detect Command & Control channels