Friday, 28 December 2012

Timeline Pivot Points with the Malware Domain List

I thought as its the end of the year it would be a good opportunity to briefly break away from the SANS Forensic Artifact posts I've been writing. In my own time I've been playing around with some code that parses a Timeline file for any URL discovered within and then compares that with the URLs listed in the Malware Domain List (MDL).

If a match is found it lists the malicious URL from MDL and the description which explains why that URL has been listed on MDL. I'm creating this for greater ability to find "Pivot Points" which both Rob Lee and Harlan Carvey mention which serve as an anchor for our investigations. Pivot Points can come in a variety of forms both verbally and technically and will hopefully assist us with a starting point or area of focus. The less time we can spend poking around an image the more time we can spend providing value to our customers or employers.

So to get started I first downloaded a copy of the Malware Domain List. You can get yourself a copy at the following location -> Once you have the list I proceeded to create an SQLlite database and imported the MDL list within it. You can easily install the Firefox addon SQLlite Manager which is the method I've used.
  1. Create a new database in the same directory as the script called malwaredomainlist.sqlite
  2. Import the MDL from CSV into new table called mdomain
  3. See screenshot below for appropriate field names to use for the table

Above are some basic steps to get you up and running and if you review the screenshot you'll see the table and field names I've used. If you decide to use the tool that I post you will want to ensure that your  filename, table name and your field names are the same as mine otherwise you'll generate some errors.

I had a few attempts at tackling how I would compare the domains discovered within my timeline to the ones within the MDL. I felt the only way to do this accurately would be to reduce both URLs down to their domain name including the suffix / tld / gtld. I had a few attempts at coding this but always found that some domain would break the script at a point. In the end I went with a pre packaged module -> To install the module type the following command from command prompt while ensuring you've obviously installed Perl in the first place. Below is the command plus the output

 ppm install Domain::PublicSuffix  
 Downloading Domain-PublicSuffix-0.07...done  
 Downloading Data-Validate-Domain-0.10...done  
 Downloading Net-Domain-TLD-1.69...done  
 Unpacking Domain-PublicSuffix-0.07...done  
 Unpacking Data-Validate-Domain-0.10...done  
 Unpacking Net-Domain-TLD-1.69...done  
 Generating HTML for Domain-PublicSuffix-0.07...done  
 Generating HTML for Data-Validate-Domain-0.10...done  
 Generating HTML for Net-Domain-TLD-1.69...done  
 Updating files in site area...done  
  11 files installed  

The above module make use of a Firefox dat file which it uses to identify the TLD / suffix on the domain. So in order for the script to work you'll need to also download this dat file which you can find at the following -> and save it within the same directory at the script.

Now that we have our database sorted I've created the following script. At this point its still a work in progress and I haven't commented it very well. As always my code is taken "as is" and I provide no additional support or responsibility for the output it provides. I'm no coding guru and always appreciate feedback on a better or more efficient way of doing things so feel free to shout out.

 #! c:\perl\bin\perl.exe   
 use Domain::PublicSuffix;  
 use DBI;   
 use strict;   
 use Getopt::Long;   
 use Regexp::Common qw /URI/;   
 use URI;  
 use List::MoreUtils qw/ uniq /;  
 my %config = ();   
 GetOptions(\%config, qw(file|f=s system|s=s user|u=s help|?|h));   
 if ($config{help} || ! %config) {   
   exit 1;   
 die "You must enter a path.\n" unless ($config{file});   
 #die "File not found.\n" unless (-e $config{file} && -f $config{file});   
 my $file = $config{file};   
 my @uniq_domains;  
 my $suffix = new Domain::PublicSuffix ({  
   'data_file' => 'effective_tld_names.dat'  
 open( my $fh, '<', $file ) or die "Can't open $file: $!";  
 while ( my $line = <$fh> ) {  
      my @url = $line =~ m/($RE{URI}{HTTP}{-scheme => qr(https?)})/g;   
           my $temp_domain = URI->new( $url[0] );  
           my $domain = $temp_domain->host;  
           my $domain1 = getDomain($domain);  
      close $fh;  
      my @unique = uniq @uniq_domains;  
           foreach ( @unique ) {       
                     if($_) {  
                          my $db = DBI->connect("dbi:SQLite:dbname=malwaredomainlist.sqlite","","") || die( "Unable to connect to database\n" );   
                          my $all = $db->selectall_arrayref("SELECT domain,description from mdomain where domain LIKE '%$_%'");   
                          foreach my $row (@$all) {   
                               my ($maldomain,$description) = @$row;        
                               my @splitdomain = split('/',$maldomain);  
                               my @splitdomain = split(':',$splitdomain[0]);                                
                               my $tempmdomain = getDomain($splitdomain[0]);                      
                               if($_ eq $tempmdomain) {  
                                    print $_.",".$maldomain.",".$description."\n";   
 sub getDomain {  
 my $root = $suffix->get_root_domain($_[0]);  
 return $root;  
  sub _syntax {   
  print<< "EOT";   
  Produce list of malware domain hits from timeline output   
  -f file..................path to timeline file   
  -h ......................Help (print this information)   
  **All times printed as GMT/UTC   
  copyright 2012 Sploit   

At this point if you run a command such as the following:

 malwaredomainlist -f timeline.csv > output.txt  

You'll be presented with output in csv format (assuming my instructions made sense) where the fields presented are the domain in question, the complete malware domain url and the description/comments. Here is a sample output:,|,RFI,,RFI,,RFI,,RFI,,Mebroot calls home,,Rogue,,compromised site directs to exploits  

As you can see from the above there are some URLs which will consistently generate false positives such as My script grabs the unique URLs listed within a timeline and typically you'll almost always have listed.

At this point I'm not sure the value in this tool. Its fairly quick to run and if you find yourself with a massive timeline file and you're not sure where to start then potentially this might be your next best bet. While i'm tweaking the code I haven't created the executable version of it yet however I have uploaded the  following code to my Google code repository to save you any issues with copying the source code above.

Hopefully you get some value out of the tool please let me know if you have any success with using it. In the meantime I'll continue to tweak and update the code. At this point it would be nice to have an option to download a fresh MDL and update the database. Overall this wouldn't take long to do manually however it would be nice for it to be automatic.

Thursday, 27 December 2012

SANS Forensic Artifact 6: UserAssist

I'm a little late to say this but firstly Happy Christmas to my readers out there. I've been fortunate enough to have a little time off but still find myself working the Christmas / New Year period. I hope some of you have more time off and can catch up on some of those tasks you've been avoiding.

For today we're moving onto the new category which I think everybody will find of interest which is Program Execution. There have been a huge number of posts on these artifacts and just how valuable they can be. Once again we'll attempt to create a few of the artifacts in different ways and see how that results when using our tools.

I still haven't forgotten about the artifacts we've missed so far and I'm currently working on some posts to cover those so that I have a complete series.

GUI-based programs launched from the desktop are tracked in the launcher on a Windows System.
All values are ROT-13 Encoded
  • GUID for XP 
    • 75048700 Active Desktop 
  • GUID for Win7 
    • CEBFF5CD Executable File Execution
    • F4E57C4B Shortcut File Execution
  • Program Locations for Win7 Userassist
    • ProgramFilesX64 6D809377-…
    • ProgramFilesX86 7C5A40EF-…
    • System 1AC14E77-…
    • SystemX86 D65231B0-…
    • Desktop B4BFCC3A-…
    • Documents FDD39AD0-…
    • Downloads 374DE290-…
    • UserProfiles 0762D272-…
Lets firstly take a look at what we see in my UserAssist registry key so we understand what our tool must export and parse and to be able to understand  which applications have launched and from where. I browsed  to the following "NTUSER.DAT\Software\Microsoft\Windows\Currentversion\Explorer\UserAssist" and found this

Within each of the Count keys listed a number of values which as mentioned above are ROT13 encoded. To the human eye they don't make much sense but once we decode them we'll easily see what the values mean. To give you a feel for what the values look like compared to the decoded values see the following output. I have just grabbed some sample values from my own computer where the first value is the ROT13 value and the second value is the decoded value.

 P:\Cebtenz Svyrf (k86)\Zbmvyyn Sversbk\bzav.wn  
 C:\Program Files (x86)\Mozilla Firefox\omni.ja  

You get the picture of what we are dealing with and as mentioned above these are just a few samples of what I have in mine. You'll notice that there are a number of values with UEME prefixing a word. These can also add context to how an applications may have been run. I've attempted to find a full list of each of these for both Windows 7 and Windows XP however I've only been able to find bits and pieces. The following list is taken from Didier Stevens blog at the following location (here).
In Windows 7 they've significantly reduced the amount as you can see below in the comparison. Many of the following are self explanatory and I won't be going into each for this particular tutorial.

 Windows 7  
 XP DLL (version 6.00.2900.3157):  

So lets try to generate some of our own values and see how that shows within the output of RegRipper. To get started I began by running 'procexp.exe' from the system internals suite. I picked this application because it was GUI based and it would be easy for me to copy it to different locations on my computer. I'd then once again use a combination of HoboCopy (to rip my active registry hive) and RegRipper to rip the userassist registry key and examine the contents. I ran procexp.exe in four different places which were Desktop, root of my username folder, Documents and finally from within the x64 Program Files location.

I  ran the following command for HoboCopy

 HoboCopy.exe c:\Users\username c:\tmp\ ntuser.dat  

Then the following for RegRipper

 rip.exe -r c:\tmp\ntuser.dat -p userassist2 > c:\tmp\userassist.txt  

The above commands produced the following output

 Thu Dec 27 07:31:20 2012 Z  
  {6D809377-6AF0-444B-8957-A3773F02200E}\procexp.exe (1)  
 Thu Dec 27 07:30:57 2012 Z  
  C:\Users\username\Documents\procexp.exe (1)  
 Thu Dec 27 07:30:37 2012 Z  
  C:\Users\username\procexp.exe (1)  
 Thu Dec 27 07:30:11 2012 Z  
  C:\Users\username\Desktop\procexp.exe (1)  

As you can see from above most of them make sense apart from the one where we ran from within our x64 Program Files. I grabbed the code highlighted in red and Googled the code. I found the following Microsoft site which explained each of the codes.

If you don't want to use the list I've posted above you can also do a find from within regedit and that will also find the code.

I decoded some of the values that I had listed in my output and placed them in the categories identified in the Microsoft article

           {1AC14E77-02E7-4E5D-B744-2EB1AE5198B7}\NOTEPAD.EXE (19)  
           {1AC14E77-02E7-4E5D-B744-2EB1AE5198B7}\cmd.exe (5)  
           {F38BF404-1D43-42F2-9305-67DE0B28FC23}\regedit.exe (1)  
           {7C5A40EF-A0FB-4BFC-874A-C0F2E0B9FA8E}\Notepad++\notepad++.exe (1)  
           {7C5A40EF-A0FB-4BFC-874A-C0F2E0B9FA8E}\Microsoft Office\Office12\OUTLOOK.EXE (11)  

Hopefully I've explained the artifact and you can take a better understanding away. This artifact has had countless articles written about it and the importance to your investigations. If you're not reviewing it then you should get started with it and make sure its part of all your investigations.

Below are some key references that I've found while researching this artifact and you might find some value.


Monday, 3 December 2012

SANS Forensic Artifact 5: Downloads.sqlite

I thought I'd get through this next artifact fairly quickly as again I've done some work prior with my Firefox script which has the option available to parse the information out of the Downloads.sqlite database.

Please note that the last category should have been posted as Artifact 4, I've adjusted that, and therefore this makes Artifact number 5 on the poster.

SANS lists the following information within the poster within their File Download Category

Firefox has a built-in download manager application which keeps a history of every file downloaded by the user. This browser artifact can provide excellent information about what sites a user has been visiting and what kinds of files they have been downloading from them.

Location: Firefox
XP %userprofile%\Application Data\Mozilla\ Firefox\Profiles\<random text>.default\downloads.sqlite
Win7 %userprofile%\AppData\Roaming\Mozilla\ Firefox\Profiles\<random text>.default\downloads.sqlite
Downloads.sqlite will include:
• Filename, Size, and Type
• Download from and Referring Page
• File Save Location
• Application Used to Open File
• Download Start and End Times

While we are on this topic I thought it might be timely to touch on a recent post by Patrick Olsen over at the System Forensics blog. Patrick posted this week about the creation of a new tool that he'd been working upon named BARFF which stands for Browser Artifact Recovery Forensic Framework. This tool is beneficial for both my last and current post but in particular the SANS poster category of "Browser Forensics". I haven't had the chance to download a copy myself as yet but I encourage anyone to give it a go and provide him with your feedback.

In terms of the structure of the Downloads.sqlite database and any of the databases associated with Firefox David Koepi has an excellent resource available here which will provide a strong resource for those wanting to get started on browser forensics. I thought it would be beneficial to first download a number of applications through Firefox and then using SQLite Manager, a plugin for firefox, we can run an initial query and take a look at what we see.

From the above screenshot there are a number of items we can use from a forensic perspective:
  • The name which contains the name of the executable
  • The source which contains the source of where the file was downloaded from.
  • The target file path
  • Start and end time which is what we'll use within our timelines
  • The state of the download as mentioned by David Koepi
    • "0"  in the state object indicates download is in progress
    • "1" in the state object indicates download is successful
    • "3" indicates download is cancelled
    • "4" indicates download is paused
  • We have a referer  field for the referring site
  •  Although not shown well in the above screenshot two important fields are preferredApplication and preferredAction which show the default application for opening 
    • "0" states that the file has been saved
    • "4" I believe states that it was open with a preferred application but more testing is required
    • In my tests i was unable to populate the preferredApplication field and again  some further testing is required
  • Lastly the currBytes and MaxBytes which can be used for a comparison between how large the file is in comparison to what has actually been downloaded.
In my example in the screenshot above I cancelled the download of FTK 4.1 and that is reflected by the state of 3 and MaxBytes lists it as -1. Important to note that this database is updated to reflect the same view as the one viewed in the graphical downloads window. Should a user delete all of the entries or remove individual downloads then this will also remove it from the database. As well as the tool mentioned above I've also created a number of Perl scripts or their converted executable to parse this information. Lets take a look at how we'd run those tools and compare the output.

To run the command you can run something like the following and obviously be aware that you're profile will be in a different location to mine.

 firefox.exe -d -p C:/Documents and Settings/username/Application Data/Mozilla/Firefox/Profiles/fd9zh9ag.default -s WORKSTATION -u USERNAME > c:\temp\events.txt  

Again this parses to Harlan's TLN timeline format and you can then convert it with the script that Harlan provides and turn this into a spreadsheet for your analysis.The output is the following.

 1353362936|FIREFOX|WORKSTATION|USERNAME|dl:winscp511setup.exe src: cB:4854080 mB:4854080  
 1353364779|FIREFOX|WORKSTATION|USERNAME|dl:sav32sfx(1).exe src: cB:72805712 mB:72805712  
 1353364806|FIREFOX|WORKSTATION|USERNAME| src: cB:3656900 mB:3656900  
 1354486120|FIREFOX|WORKSTATION|USERNAME|dl:googletalk-setup.exe src: cB:1606064 mB:1606064  
 1354493698|FIREFOX|WORKSTATION|USERNAME|dl:FTK 4.1.0 Intl.iso src: cB:0 mB:-1  

Although Chrome is not specifically mentioned I felt it was of equal importance in this category and therefore it was best I showed examples for both. Again with these examples its important that when testing these tools you note the time that you download each of the files and confirm in the output, as we did in the last post, that your timeline produces the correct time while at the same time understanding any conversions required from UTC to local time.

Again I opened the database, the History file, using SQLite Manager

In this case we don't have as much detail in the downloads table as we do with the downloads database within firefox. Once again I ran the command using a similar command to the one we used above however this time using my chrome script

 chrome -d -p "C:\Documents and Settings\username\Local Settings\Application Data\Google\Chrome\User Data\Default" -s WORKSTATION -u USERNAME > c:\temp\chrome_events.txt  

The output is the following.

 1347264951|CHROME|WORKSTATION|USERNAME|dl:C:\Documents and Settings\username\My Documents\Downloads\ChromeSetup (1).exe src: cB:739808 mB:739808  
 1354584249|CHROME|WORKSTATION|USERNAME|dl:C:\Documents and Settings\username\My Documents\Downloads\sav32sfx (1).exe src: cB:72805712 mB:72805712  
 1354584279|CHROME|WORKSTATION|USERNAME|dl:C:\Documents and Settings\username\My Documents\Downloads\googletalk-setup (1).exe src: cB:1606064 mB:1606064  
 1354584297|CHROME|WORKSTATION|USERNAME|dl:C:\Documents and Settings\username\My Documents\Downloads\sav32sfx (2).exe src: cB:72805712 mB:72805712  

Well I think I've discussed this topic enough. Again not overly complex but like everything you'll have a return on your tools if you understand what they do and have some assurance that the output is expected and once again you're aware of any adjustments you may need to make to ensure you're looking at the correct local time if required. If you're looking for the tools mentioned above that I've written you can find them at the following location