An Introduction to Network Visibility

In our Introduction to Security monitoring post we introduced the use of sensors to gather monitoring data. That article is a good place to start if you are new to information security or security engineering. In this post we are going to expand on that topic, specifically on getting good data from your network. This an introductory-level post but will require some some basic networking knowledge (such as an understanding of the TCP/IP stack and the structure of IP packets) to follow everything.

Once you have a system in place to collect and process your monitoring data then you can start adding new sources to cast a wider detection net and monitoring your network should be high on your priority list. Packets move from point A to point B, within your perimeter and outside to the internet. It is unlikely that an application will not be sending and receiving some data and most modern applications are totally dependent on their network connection. We want to be able to see this activity, record it firstly to understand what normal looks like and then to start to hunt for the bad.

Before you start monitoring your network it is a black box and Network Visibility is an attempt see what is happening inside. You instrument your network by using of existing functionality in the network equipment itself or by deploying sensors.

Getting access to network data has always been a high priority for infrastructure Security Engineers. Your chances of detecting an attacker are greatly improved if you understand where your data is going, what services it’s being communicated over and even exactly what is being sent. It also creates an independent vantage point. It’s a capability that can be decoupled from your servers and applications; so if a system is compromised and you unable to trust that endpoint (imagine the attacker has been manipulating the processes or the logs), you still have something you can trust.

It’s possible to have a depth to your visibility depending on much engineering time you want to invest. In general terms a communication consists of metadata to route the communication to it’s intended recipient (e.g. an IP address or a telephone number) and the content of the communication itself (the payload of the IP packet or the actual phone call). It’s less expensive to pull the metadata from a communication than the content and while metadata analysis is very powerful we will term it ‘shallow visibility’ as you don’t know what exactly was contained within the communication.

If you currently don’t have a network visibility capability then flow data is a good starting point. A ‘flow’ is a unidirectional communication from source to destination. It can be expressed as 5-tuple: source IP & port, destination IP & port and protocol. We can use this to determine who originated the communication (source IP), who the recipient is (destination IP) and what service or application were they using to communicate (destination port). This is all in the header of the IP packet.

Collecting flow can have a low engineering cost as a lot of Enterprise grade networking equipment can already export it. If you have capable firewalls, routers or switches then you can turn it on now and have that device export flow to a collector (there are a number of different flow export formats depending on the vendor including Netflow, JFlow, sFlow and IPFIX).

So Flow is incredibly useful for detection but it does only give you shallow visibility. It is reporting on the metadata of the communication (specially from L2 and L3 in the OSI stack), there is much more data that can be inspected in the packet payload.

The next level of visibility involves looking at the content of the communication to determine exactly what the application is doing. We examine the payload of the packet by performing DPI (Deep Packet Inspection) to dig into the data that it contains. One common use case for using DPI is to match ‘signatures’ or ‘rules’ in traffic to identify malicious activity. Some Malware, for example, can be identified by a specific signature such as a file hash. Using a network intrusion detection system (NIDS) like snort you can generate a security events if you get a hit. Using DPI we can also determine what is really happening in that communication. Flow does tell you what port is being used and from that we can infer what that service is. For example it is common for ports 80 and 443 to be allowed outbound as they are for web traffic. But it is also common that those ports are abused and used for non-web related activity.

The deep visibility that you can get using DPI does come with both an engineering and a hardware cost. Depending on the size of your network links you will likely need dedicated hardware and to make changes in your infrastructure (such as ‘taps’ or span ports) to deliver the traffic for analysis. For high traffic links (10G+) this did used to mean vendor appliances with expensive hardware (such as dedicated ASICs or FPGAs). But now we can get really great results from COTS hardware and open source projects such as Suricata, snort and Bro.

Bro in particular is something I would recommend to anyone who really wants to understand what is happening inside their network, it’s a technology that has changed network traffic analysis and arguably is where we derive the term network visibility from. Bro provides you with a framework to understand at a protocol level what is travelling across your links; turning the protocols it understands into logs which can be analysed and crossed referenced. It also has it’s own scripting language, allowing you to add in any unusual protocols you have in your environment.

Depending on how you have implemented DPI it won’t allow you to go back and perform detailed forensics after an incident. Full packet capture is a very low level of visibility where you record your raw network traffic to disk. Tools such as AOL maloch allow you to record and store all of your packets. This can be the most expensive visibility tool as the amount of disk space required can be significant depending on the size of your links and length of time you want to keep.

This post has been an introduction to Network visibility and the technologies that you can use to implement it. If you’re interested in this topic or any of the technologies mentioned here then contact us.

Originally published on the Cadency blog