Unsupervised Progressive Anomaly Detection for Network Traffic

Today, almost any computing device is accessible over the Internet. As more and more computing devices get ubiquitous (interconnected), they cause more cybersecurity-related threats. The wide landscape and forms of networked computing devices and existing cyber threats make detection and prevention of cyber threats difficult. Differentiating between benign and malicious activities of networked devices, also known as anomaly detection, becomes even more challenging. In contrast to real-time network traffic monitoring, network traffic can be captured and analyzed later. But due to high-speed and increased network traffic volume, capturing, storing, and evaluating such network captures (PCAP) is expensive and complex. Conducting manual network traffic analysis on PCAP files is a very time-consuming task. Retrieving cyber threat detection-related information requires huge processing time as well as a set of skills from an analyst. In this thesis, we propose a cyber threat detection framework. We combine three open-source projects, Snort++, Surica, and Zeek, from the intrusion detection domain with file scanning tools, from malware and digital forensics domain. Additionally we intergrate two open source cyber threat intelligence datasources. For a detected anomaly in a given PCAP file, an anomaly score will be given. The anomaly score supports an analyst during PCAP file analysis by highlighting whether to investigate the detected threat futher or not. The result of our framework can be ingested into any external network monitoring application. As proof of concept, we integrate our solution into web-service NetCapVis, a web-based progressive visual analytics system for PCAP analysis developed by IGD Fraunhofer.