Cyber Anomalies in Real-Time

Warning This documentation is archived and is no longer maintained.

With an increasing number of cyber threats, most of the cyber security team doesn’t have the capacity to manually detect, monitor, and defend against all of them. Effective cyber threat management requires leveraging automation to inform decisions.

The Collibra DQ framework, provides organizations the ability to load and process diverse security data feeds at scale to detect network data anomalies. The DQ alerts enable network admins to respond to these events in timely manner.

The following scenario demonstrates how to detect anomalies with network traffic datasets.

  1. Perform IP address validation.
  2. Detect the unusual network traffic patterns based on locations.
  3. Identify the suspicious packets based on size.
  4. Detect the malicious activity based on source and destination IP addresses.

Infosec dataset preview

Data set contains Timestamp, Source Workgroup, Source IP, Source Port, Destination Workgroup, Destination IP, Destination Port, Application, Protocol and Packet size information.

IP Address format validation

Business Check Collibra DQ Feature Text
Is IP a valid format? RULE AUTO-IP detection
Is the IP address NULL or Missing? BEHAVIOR AUTO

Source and destination workgroups

Business Check Collibra DQ Feature Text
Is it usual network traffic based on locations? PATTERN Source_Workgroup -> Destination_Workgroup

Source and Destination IP Address validation

Business Check Collibra DQ Feature Text
Is it usual network traffic based on source and destination IP? PATTERN Source_IP -> Destination_IP

Packet Size

Business Check Collibra DQ Feature Text
Is the Packet Size NULL or Missing? BEHAVIOR AUTO
Packet Size within normal range? PATTERN Source_IP -> Packet_SizeB

Resulting DQ Check

Copy 
-f file:///home/danielrice/owl/bin/demos/infosec/ -d tab \
-fullfile -fq "select * from dataset" -encoding UTF-8 -ds infosecv2 \
-rd "2020-04-04" -dl -dlinc Destination_IP,Packet_SizeB,Source_IP \
-dlkey Source_IP -fpgon -fpginc Destination_Workgroup -fpgkey Source_Workgroup \
-df "yyyy-MM-dd" -loglevel INFO -h 10.142.0.29:5432/owltrunk -owluser admin \
-fpgsupport .000000001 -fpgconfidence 0.4 

Which components did we use?

DQ addresses the issue of efficient network traffic classification by performing unsupervised anomaly detection and uses this information to create dynamic rules that classify huge amounts of Infosec data in real time.

By providing Infosec datasets, along with anomaly records DQ outlier and pattern algorithms found the anomaly in the network traffic. It mainly detects the following anomalies:

  1. Traffic between Atlanta->Texas.
  2. The packet size extremely low between Atlanta->Texas.
  3. Atlanta source IP and Texas Destination IP.

Realtime DQ provides the alerts on network traffic anomalies, which can help network admins to do further deep analysis and takes preventative measure, which is a daunting task with huge amount of data.

Sample Dataset

infosec-anomaly.csv
Binary