Chaosreader
A open source tool to trace TCP/UDP/... sessions and fetch application data
from snoop or tcpdump logs. This is a type of "any-snarf" program, as it will
fetch telnet sessions, FTP files, HTTP transfers (HTML, GIF, JPEG, ...),
SMTP emails, ... from the captured data inside network traffic logs.
A html index file is created that links to all the session details, including
realtime replay programs for telnet, rlogin, IRC, X11 or VNC sessions; and
reports such as image reports and HTTP GET/POST content reports.
Chaosreader can also run in standalone mode - where it invokes tcpdump or snoop
(if they are available) to create the log files and then processes them.
For a look at the command line options see the
readme printed by the program
(chaosreader --help2).
Screenshots
- Chaosreader example output
In this example, a snoop file was created while I loaded a website,
used telnet to login, and ftp to transfer files; Chaosreader has managed
to extract the HTTP sections, follow the telnet session, grab the FTP
files, and create an Image Report
from the snoop log. It has also created a
replay program to
playback the telnet session.
- Chaosreader example output 2
This demonstrates Chaosreader processing IPv6. A few telnet
sessions were captured that used site local IPv6 addresses.
- Chaosreader example output 3
This is a large example to demonstrate the standalone mode of
chaosreader - where it does not require a log file, instead running
tcpdump or snoop and generating it's own logs. In this example it was run
using "-S 1,5" which generates five samples of one minute each. (While this
runs, reports can be viewed by reloading the index.html file - which
is updated as new samples complete). I was viewing websites and using telnet
and ftp while chaosreader collected and processed it's five samples.
This is the final index.html generated, which links to all the collected
data.
- Chaosreader example output 4
This demonstrates Chaosreader capturing HTTP GET and POST queries,
and presenting them in the GET/POST Report.
The processed log file captured a search on google, slashdot, and imdb.
This reporting feature could be used to highlight potential
vunerabilities.
- Chaosreader example output 5
Here the options "-ve" were used, where it extracts everything it can
from a log. This includes
hex dumps and files
containing the raw traffic (usually three - ".raw" is 2-way, ".raw1" is A to B,
".raw2" is B to A).
- Chaosreader example output 6
This demonstrates processing X11 traffic - currently an experimental feature.
Here I run "xeyes", "xclock" and "xterm" - and run a few commands from the
xterm. The following is created from the X11 "xterm" session: a red/blue
html report
containing keystrokes and displayed text from the session;
a text playback
perl program to replay the displayed text;
and an X11 playback
perl program to redisplay the X11 session in realtime or by
a factor - although the most interesting feature it is currently the least
robust (this example requires a 24 bit display).
See troubleshooting below.
- Chaosreader example output 7
This demonstrates processing VNC traffic.
Here I connect to a VNC session and type in a few commands. Chaosreader
creates the following:
a html report
containing the keystrokes; a
keystroke playback
program. But neither are too interesting as they
lack the reply text (it's been pixelated). What is interesting is the
VNC playback
perl program that redisplays the VNC session in realtime or by
a factor. The program is used in conjunction with vncviewer and is
experimental.
This example requires 24 bit colour support, an 8 bit example is
here.
See troubleshooting below.
- Chaosreader example output 8
This demonstrates processing SSH traffic.
Three connections are captured: an SSH command line session,
an X11 session forwarded over SSH, and an SCP copy.
From the command session a
html report, a
playback program, and a
keydata file has
been created. The encryption is not broken, instead we focus on the data that
is available - transfer size and timing.
See sshanalysis for a demonstration of
further analysis.
Downloads
chaosreader - latest version (github)
chaosreader0.94 - version 0.94, stable, 29-May-2004
This started life as a small handy perl program, and as it grew I was
reluctant to break it into libraries. It still is a handy perl program
although the code is massive. It currently remains a single program for
ease of use.
Sites
Operating Systems
This has been tested on Solaris 9, RedHat 9 Linux and
Windows 98. Perl version 5.6 is required.
Feature List
(New features are in red)
* Reads Solaris snoop logs and four versions of tcpdump/libpcap logs
* Standalone mode generates a series of logs and then processes those
* Processes HTTP, FTP, telnet, SMTP, IRC, ... application protocols
* Processes any TCP and UDP traffic
* Processes 802.11b wireless traffic
* Processes PPPoE traffic, tun device traffic
* Retrieves transferred files from FTP and HTTP traffic
* Creates HTML and text reports to list contents of the log
* Creates realtime replay programs for telnet or IRC sessions
* Creates red/blue coloured HTML reports for 2-way sessions such as
telnet and FTP
* Creates red/blue coloured HTML reports for any TCP, UDP or ICMP traffic
* Creates image reports from HTTP, FTP transfers
* Creates HTTP GET and POST reports from queries
* Creates red/blue coloured HTML hex dumps for any TCP, UDP or ICMP traffic
* Creates plain text hex dumps for any TCP, UDP or ICMP traffic
* Creates HTTP proxy logs based on observed HTTP traffic, using the squid
log format
* Creates X11 realtime replay programs to playback an X11 session.
(experimental).
* Creates red/blue coloured HTML reports for X11 text and keystrokes.
* Creates realtime replay programs for X11 text communication.
* Creates VNC realtime replay programs to playback a VNC session.
(experimental).
* Creates HTML reports for VNC keystrokes.
* Creates realtime replay programs for VNC keystrokes.
* SSH content analysis. reports, replays and keystroke delay data files.
* Creates raw data files from TCP or UDP transfers
* Supports TCP out of sequence number delivery
* Supports IP fragmentation
* Supports IPv4 and IPv6
* Processes ICMP and ICMPv6
* Very configurable (including filtering on IPs and ports)
* Can sort data based on time, size, type or IP.
* Can skip sessions smaller than a min size.
* Runs on Solaris, RedHat, Windows, ...
I plan to add many more protocols, especially freeware/opensource protocols.
Current News
This temporary news section is here to explain what may appear
in future versions.
I'm currently working on wireless protocols,
reducing memory usage, and usability features.
You could help by emailing a small sample snoop/tcpdump dump of wireless
(or any odd)
traffic to brendan.gregg@tpg.com.au .
Troubleshooting
Just as a note, the most reliable client I've used to playback X11
has been RedHat 9 Linux;
The most reliable client to playback VNC has been Solaris 9;
The most reliable playbacks in general are
those on the same OS and hardware as the original.
1. X11 playbacks, although these are experimental here are some
things to check,
- Nothing Happens. - Did you capture the start of the connection?
This is necessary to draw the initial windows.
- Too big? - Large X11 sessions are problematic. Experiment by capturing
and playing back something simple first, eg an "xterm", "xclock" or "xeyes".
Check the size of the replay script, I've yet to see one larger than 500 Kb
actually work. Check for newer versions of chaosreader that may improve
on this.
- Authorised to connect? - Check your xhost settings on your target
server. eg, "xhost +hostname" can authorise the hostname.
- Colour depth ok? - A 24 bit colour playback won't display on an 8 bit
screen, and visa versa. Read the text the playback script prints during
connection to see both the server and the playback depth. It will help
if you can change your screen depth to match. I use an extra
VNC window as the target screen, eg "/usr/local/bin/Xvnc :2 -geometry 700x600
-depth 16 & DISPLAY=mars:2 dtwm -display :2 &" and then direct the
script to port 6002 ("-p 6002") while vncviewer is displaying it.
- Can't connect to server? - try using the options "-d" and "-p", eg,
"./session_0001.X11.replay -d mars -p 6000"
2. X11 red/blue html report, this displays the communicated text,
- Red text garbeled? - If the red text
(keystrokes) are garbeled, then the playback contains a strange keymap.
You can view the keymap using "xmodmap -pke" on the client host. Check
for newer versions of chaosreader.
3. VNC Playbacks, these are experimental (although more robust than
X11 playbacks)
- vncviewer prompts for a password? - Just hit 1 character (anything) and
then enter. The playback will continue.
- the colours are a little odd - Sometimes the playback will appear to have
a tinge of green or blue, this usually happens when a session from one
desktop is replayed on another with different video settings. This is
normal (so far).
- The colours are VERY odd - if you just get a mess of ranbow colours
that slowly write from top down,
you are probably playing back a 8 bit or 24 bit depth session on a video
display with a different depth. One solution can be to change your display
depth. Or try forcing the depth in vncviewer using "-depth", new versions
have extra options such as "-8bit" (for 8 bit playbacks) which can
definately help. If you are on a unix style vncviewer, you can read the
requested and delivered colour depth values during the connection.
- vncviewer crashes - "Rect too large". I found this was a depth
negotiation problem, as written above.
- The screen draws okay then blinks white - This usually happens when the
vnc session window is larger than your current screen (eg replaying a
1024x768 session on a 800x600 screen). Try increasing your screen
resolution.
- vncviewer crashes - "Runtime Error". Try upgrading to the
latest version of vncviewer. You may have tried to display a newer
session on an older client.
- "Please Wait - initial screen loading" - is all I see, it never seems
to load. I've found this was due to a newer version of vncviewer
autonegotiating "Format and Encoding" incorrectly. In the vncviewer options,
disable "Auto select" and manually try encoding, eg "Hextile".
4. Out of Memory - Chaosreader currently uses around 7x the log file
size in memory. Future versions will be far more memory efficient.
- Try running chaosreader without the extra report options such as "-ve",
as these use much more memory to process.
5. index.html is too long, and is mostly small connections that
are not interesting.
- Try the minsize option, eg "-m 1k" to only save sessions that are
at least 1 Kb.
- Try to exclude data you are not interested in using the include/exclude
port options or the include/exclude IP address options. See
"chaosreader --help".
Back to Brendan Gregg's Homepage
Last updated: 09-Apr-2014 (previous update: 02-May-2004)