"Robust" is a set of tools to leverage the HTTPS REST API of the McAfee Advanced Threat Detection 3.8 - 4.x appliance.
robust
: basic cli submission of a single piece of malware to a MATD server.robust-profiles
: list the available MATD profilesrobust-search
: search MATD via MD5 for a report.robust-watchdog
: monitor a directory for files and submitmultithreaded
to MATDrobust-convict
: submitmultithreaded
a directory filled with samples and sort into malicious, clean, error, etc.robust-reporter
: parse offline the json files returned during large batch submissions.robust-version-checker
: Check the MATD Server Version
This is not a supported or official application of McAfee. This work is based off of publicly available published documentation for integrating with the McAfee ATD REST API 3.6.x to 4.x
Official API Documentation is available here:
McAfee ATD is a commercial grade enterprise security sandbox analysis appliance. It main function is to provide advanced detection for stealthy, zero-day malware. McAfee Advanced Threat Defense is available as an on-premises appliance or a virtual form factor, with support for both private and public cloud with availability in the Azure Marketplace.
- https://www.mcafee.com/us/products/advanced-threat-defense.aspx
- https://www.mcafee.com/us/resources/data-sheets/ds-advanced-threat-defense.pdf
Req: Python 2.7.x.
pip install robust-atd
For Development it is best to
git clone [....] pip install -e .
Note: Python 3.x is not supported.
Note: python setup.py install
will attempt to install dependencies from the internet via pip
.
For offline runtime installation, please download the pip packages listed in the requirements.txt
.
It is recommended to install virtualenv & virtualenvwrapper via Virtualenv Burrito
.
See: README_PYTHON_UP.md
$> mkvirtualenv robust
$> workon robust
$(robust)> wget https://github.com/shadowbq/robust-atd/archive/master.zip
$(robust)> unzip master.zip
$(robust)> cd master
$(robust)> python setup.py install
-or-
$> mkvirtualenv --python=python2.7 robust
$> workon robust
$(robust)> pip install robust-atd
Robust will use the ~\.robust
configuration file to load defaults into the scripts.
The configuration file is broken into multiple sections. If you use a section you must define all the settings in that section.
- [auth]
- [connection]
- [convict]
It is recommended to set the file to read-write
only for the current user, and remove all world (-)rwx
permissions.
Authentication Section [auth]
:
$(robust)> cat ~/.robust
[auth]
user: admin
password: password.
Connection Detail Section [connection]
:
$(robust)> cat ~/.robust
[connection]
ip: atd.localhost.localdomain
skipssl: true
maxthreads: 15
Data Storage Section [storage]
:
Note: Datastorage locations will be created if they do not exist.
$(robust)> cat ~/.robust
[storage]
severity: 3
cleandir: ~/robust/clean
dirtydir: ~/robust/dirty
reportdir: ~/robust/reports
errordir: ~/robust/errors
This file is expanded via the os
module is compliant with windows user directories.
Using robust
for submitting samples.
usage: robust.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -s
FILE_TO_UPLOAD -a ANALYZER_PROFILE [-v] [--version]
Robust McAfee ATD Python CLI tool
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output verbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)
Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: admin)
-p PASSWORD (p)assword for username
(default: password.)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: False)
Sample parameters:
-s FILE_TO_UPLOAD (s)ample or file to be analyzed
(default: None)
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)
Examples:
robust.py -u admin -p password. -i atd.localhost.localdomain -s /usr/local/bin/file_to_scan -a 1
A sample can be submitted via cli with full flags, .robust
configuration file, or interrupt passwords.
$(robust)> robust.py -u admin -p password. -i atd.localhost.localdomain -s /home/malware/non-malicious-container/putty_upx_7.exe
Using interrupt (interactive) passwords:
$(robust)> robust-profiles.py -n -l
Password: <input password>
ATD profiles: 1
Profile id: 26
Name: Win XP Down Select (Online)
OS: winXPsp3
Run all down selects?: Off
******************
Using System Return codes with -q
Quiet output flag. When the quiet flag is
used for submitting samples or searching reports the severity of the application
is returned as a system exit/return code. Negative return codes indicate faults
or failure during submission.
(robust)>$ robust.py -n -a 26 -s ./.samples/Sample.exe -q
(robust)>$ echo $?
2
Common Fault codes:
-1 ---> Error connecting to the ATD Server
-2 ---> Error uploading file to the ATD Server
-3 ---> Analysis failed
-4 ---> Error getting report
-5 ---> Error Obtaining vmprofilelist
Malware ranking: (If the severity level of the sample is 3 and above it is generally regarded a threat)
N/A -> Sample did not run
-1 --> Sample is white listed
0 ---> No malicious activity detected (None)
1 ---> Sample is slightly suspicious (Low)
2 ---> Sample is somewhat/probably is suspicious
3 ---> Sample is malicious (Medium)
4 ---> Sample is malicious
5 ---> Sample is malicious (Very High)
You can quickly test your connection settings in the CLI.
$(robust)> robust-version-checker.py -u robust -p password. -i atd.example.com -n
Connection successful...
Session Value: g7aenj99pfp0gbrogfbqsd9085
User ID: 57
ATD ver: 4.2.2.16
ATD Box heartbeat: 1519939175
A tool designed to pull the Analyzer Profile policy list available to a specific user.
Pulling the Policy List - In order to submit a sample using robust
you must identify the Analyzer Profile ID. robust-profiles
assists in identifying the available profiles your user can submit samples to.
$(robust)> robust-profiles.py -n -l
ATD profiles: 10
Profile id: 1
Name: Android
OS: android
Run All Selected?: Off
******************
Profile id: 26
Name: Win XP Down Select (Online)
OS: winXPsp3
Run All Selected?: Off
******************
Profile id: 25
Name: Windows XP Full Run (Offline)
OS: winXPsp3
Run All Selected?: On
******************
Profile id: 24
Name: Windows XP Full Run (Online)
OS: winXPsp3
Run All Selected?: On
******************
A tool designed to search and return reports for a specific md5 hash.
(robust)$> $ robust-search.py -m 2F7568342339CDB8321B52FF7BEBE661 -n
(Sample.exe:2F7568342339CDB8321B52FF7BEBE661) = 2: "Sample probably is suspicious"
robust-search
has the options -w
and -t
to collect the proper report on the submission.
usage: robust-search.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -m MD5
[-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}]
[-w FILENAME] [--version] [-v | -q]
Robust McAfee ATD Python CLI tool
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)
Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: admin)
-p PASSWORD (p)assword for username
(default: password.)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: False)
Search parameters:
-m MD5 (m)d5 32bit hash of the sample to search
(default: None)
Reporting parameters:
-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}
(t)ype of report requested
(default: None)
-w FILENAME (w)rite filename for saving the requested report
(default: None)
A tool that watches a directory recursively for any new files to submit.
Example CLI
usage: robust-watchdog.py [-h] -u USER [-p PASSWORD] -i ATD IP [-n] -a
ANALYZER_PROFILE -d DIRECTORY [-e] [-j MAXTHREADS]
[--version] [-v | -q]
Robust McAfee ATD Python CLI tool
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)
Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: None)
-p PASSWORD (p)assword for username
(default: None)
-i ATD IP (i)p or hostname address of ATD
(default: None)
-n do (n)ot verify the SSL certificate for the communications
(default: False)
Watch parameters:
-f (f)ollow and watch the directory for new files to submit
(default: True)
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)
-d DIRECTORY (d)irectory to watch for events
(default: None)
-e (e)xisting files in directory will be submitted
(default: False)
-j MAXTHREADS (j) max number of threads
(default: 1)
Let it run in a shell and open another one or the file browser to create files in the /path/to/directory. Since the handler is printing the results, the output will reflect the flags chosen similar to robust.py
:
The -e
flag can be passed to cause all existing files in the directory (recurisively) to be submitted upon start.
(robust)$> robust-watchdog.py -a 26 -d ./ -n -e
.
...
.
.....
robust-convict
is a tool designed like robust-watchdog
but its purpose is to help sort large directories of malware samples into directories, while downloading their corresponding reports.
Example Usage
robust-convict.py -n -a 26 -c ./tmp/clean/ -x ./tmp/dirty/ -r ./tmp/reports/ -z ./tmp/errors/ -d ./tmp/preprocess -j 10 -t zip -q
Options
usage: robust-convict.py [-h] [-u USER] [-p PASSWORD] [-i ATD IP] [-n] -a
ANALYZER_PROFILE -d DIRECTORY [-e] [-y SEVERITY]
[-c CLEANDIR] [-x DIRTYDIR] [-r REPORTDIR]
[-z ERRORDIR]
[-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}]
[-j MAXTHREADS] [--version] [-v | -q]
Robust McAfee ATD Python CLI tool
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)
Authentication parameters:
-u USER (u)sername for the API of the ATD
(default: robust)
-p PASSWORD (p)assword for username
(default: ****<.robust>*****)
-i ATD IP (i)p or hostname address of ATD
(default: atd.localhost.localdomain)
-n do (n)ot verify the SSL certificate for the communications
(default: True)
Watch parameters:
-f (f)ollow and watch the directory for new files to submit
(default: False)
-a ANALYZER_PROFILE (a)nalyzer profile id to be used during analysis
(default: None)
-d DIRECTORY (d)irectory to watch for events
(default: None)
-e (e)xisting files in directory will be submitted
(default: False)
-j MAXTHREADS (j) max number of threads
(default: 1)
Convict parameters:
-y SEVERITY (y) treat sample as dirty with this severity [0-5] or higher
(default: 3)
-c CLEANDIR (c) move clean files to this directory
(default: ~/robust/clean/)
-x DIRTYDIR (x) move processed dirty files to this directory
(default: ~/robust/malware/)
-r REPORTDIR (r) save reports to this directory
(default: ~/robust/reports/)
-z ERRORDIR (z) move error or skip files to this directory
(default: ~/robust/errors/)
-t {html,txt,xml,zip,json,ioc,stix,pdf,sample}
(t)ype of report requested
(default: None)
robust-reporter
is a tool designed to quickly summarize the downloaded *.json
files in your 'reports' directory.
Options
usage: robust-reporter.py [-h] [-r REPORTDIR] [--version] [-v | -q]
Robust McAfee ATD Python CLI tool
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v, --verbosity increase output (v)erbosity
(default: None)
-q, --quiet (q)uiet all output
(default: False)
Reporter parameters:
-r REPORTDIR (r) reports are processed or stored using this directory
(default: ~/robust/reports/)
Sample Run
$ robust-reporter.py
82344C9864B0F1D120C0D1AB7F7C54C3 (---) : Somewhat/probably is suspicious(Low-Medium) - 24sec
D012492123E4CF0CFB3A017A2E92C077 (Malware.Dynamic) : Malicious(High) - 194sec
DB273A97C54E3E23F411EA7C9B5A82DA (Malware.Dynamic) : Malicious (Medium) - 53sec
165A36C02B3FAAF4DE38F93A3DCB821B (---) : Somewhat/probably is suspicious(Low-Medium) - 36sec
D10195670651A40C46C22972CD839E89 (Artemis!D10195670651) : Malicious (Very High) - 32sec
8271093E0E78574428BBDDDA6F34A980 (Malware.Dynamic) : Malicious(High) - 192sec
86DAFA0262BF217F5344A3B057C0DB06 (Malware.Dynamic) : Malicious(High) - 193sec
8DA4CDC3E2EE16021F237EA7A043DA8E (Malware.Dynamic) : Malicious(High) - 191sec
The inotify(7) subsystem has three important tunings that impact robust's directory watching.
/proc/sys/fs/inotify/max_user_instances impacts how many different root dirs you can watch.
/proc/sys/fs/inotify/max_user_watches impacts how many dirs you can watch across all watched roots.
/proc/sys/fs/inotify/max_queued_events impacts how likely it is that your system will experience a notification overflow.
You obviously need to ensure that max_user_instances
and max_user_watches
are set so that the system is capable of keeping track of your files.
max_queued_events
is important to size correctly; if it is too small, the kernel will drop events and robust won't be able to report on them. Making this value bigger reduces the risk of this happening.
$(robust)> pip install -r devel-requirements.txt
(robust)$> invoke -l
Available tasks:
build Build the setup.py
clean Clean up docs, bytecode, and extras
codestats Run flake8 PeP8 tests for code stats
release ``version`` should be a string like '0.4' or '1.0'.
smell Run flake8 PeP8 tests
test Run Unit tests
Nose is run via invoke test
Clearing rm -rf build
Clearing rm -rf dist
Clearing rm -rf *.egg-info
Clearing rm -rf pyclient.log
Clearing rm -rf **/**/*.pyc
Clearing rm -rf **/*.pyc
Clearing rm -rf ./*.pyc
...................
-----------------------------------------------------------------------------
19 tests run in 0.3 seconds (19 tests passed)
117 E501 line too long (97 > 79 characters)
7 F401 'ratd' imported but unused
1 F841 local variable 'rb_rtnv' is assigned to but never used
1 N802 function name should be lowercase
5 W601 .has_key() is deprecated, use 'in'
A modified Fork of atdcli.py
(Carlos Munoz - 2014).
https://pypi.python.org/pypi/atd
There is a fully operational Xubuntu 14.04 liveCD that includes:
- robust - https://github.com/shadowbq/robust-atd
- maltrieve - https://github.com/shadowbq/maltrieve
- vxcage - https://github.com/shadowbq/vxcage
It also includes
- hexeditors
- static analysis tools
- google chrome
- vmtools
- etc..
xubuntu-14.04.4-desktop-x86_64-VX-Workshop-0.4.iso (~ 1.2 GB)
Available to download with READMEs here: https://goo.gl/flcvew