HTTPS SSH

Rekha: Evaluating Effectiveness of Free Android App Security Analysis Tools in Detecting Known Vulnerabilities

This repository contains results of evaluating freely available tools related to Android security. The evaluation considered 64 tools but evaluated only tools that could be used off-the-shelf to detect known vulnerabilities (malicious behavior) with no or minimal configuration. Consequently, 19 tools were evaluated. Considered Tools section lists the tool that were considered and their details.

The 19 tools selected for evaluation were evaluated against Ghera -- An open source repository of Android app vulnerability benchmarks. Each benchmark has a Benign app that captures a specific vulnerability, a Malicious app that exploits Benign, and a Secure app that does not exhibit the vulnerability captured by Benign. The 14 vulnerability detection tools out of the 19 selected tools were executed against Benign and Secure apps of the benchmarks. The reports and logs from these executions can be found in the vulevals and secevals folders, respectively. The remaining 5 malicious behavior detection tools were executed against Malicious apps of the benchmarks. The reports and logs for these tools can be found in the malevals folder.

The manuscript is available here.

Considered Tools

The following table lists the tools were considered for the evaluation. Each tool is accompanied with a link to the repository, a one line description of the tool, and a brief reason why it was rejected for this evaluation (if it was rejected).

Tool Description Reason for rejection
Amandroid A static analysis framework with built-in checkers to look for vulnerabilities in Android apps
AndroBugs A vulnerability scanner for Android apps
AndroTotal A free service to scan malicious apps
AndroWarn A tool that detects malicious Android apps via static analysis Not actively maintained; Fails to process apks that target API levels below 19
Android Hooker An opensource project to help dynamic analyses of Android applications Enables analysis; Not a detection tool
Android Malware Analysis Toolkit A Linux distro focused on Mobile Malware Analysis for Android Enables analysis; Not a detection tool
Android Tamer A virtual environment to enable security analysis Enables analysis; Not a detection tool
Androl4b A Virtual Machine For Assessing Android applications, Reverse Engineering and Malware Analysis Enables analysis; Not a detection tool
ApkAnalyser A static, virtual analysis tool for examining the source code of Android apps Enables analysis; Not a detection tool
ApkCombiner A tool to combine multiple apks into one apk Enables analysis; Not a detection tool
ApkInspector A GUI tool for analysts to analyze the Android applications Enables analysis; Not a detection tool
AppAudit Detects data leaks The scan feature does not work
AppCritique An online tool that detects several vulnerabilities in Android apps
AppGuard A tool to manage app permissions Tool documentation in German
AppRay A tool to identify vulenrabilities automatically Paid tool
Aquifer An OS framework to prevent accidental data disclosure Enforces security; Not a detection tool
ASM A programmable interface for defining new reference monitors for Android Enforces security; Not a detection tool
Aurasium An automatic apk repackaging and security policy enforcement tool Enforces security; Not a detection tool; Not applicable Android API 19 and above
AutoCog tool to assess how well an Android application description reveal the security-concerned permission in semantics level Enables analysis; Not a detection tool
BlueSeal A tool to detect malware using multi-flows and API patterns Source code unavailable at the time of evaluation
CFGScanDroid A utility for comparing CFG signatures to CFGs of Android methods Enables analysis; Not a detection tool
ConDroid Performs concolic execution to generate test inputs Enables analysis; Not a detection tool
CopperDroid An online tool that performs automatic dynamic analysis to detect malicious behavior The APK upload service does not work
Covert A tool for compositional verification of Android inter-application vulnerabilities
CuckooDroid A tool to perform automated Malware analysis of Android apps Fails to complete analysis; forcefully timed out after 30 mins
DECAF A binary analysis platform Enables analysis; Not a detection tool
DIALDroid A tool to detect inter-app vulnerabilities
DIDFAIL A static taint analyzer for Android apps Fails to build based on the instruction on their webpage
DRACO A tool to classify apps as Malicious and Benign No documentation about usage
DeepDroid No information provided about the tool except source code No documentation about usage
DevKnox An Android Studio Plugin that detects vulnerabilities in Android apps
DroidLegacy No information provided about the tool except source code No documentation about usage
DroidSafe A static information flow analysis system to detect and remove malicious code from Android apps Analysis fails to terminate; forcefully timed out after running on a 64 GB ubuntu machine for 15 mins
DroidSearch/DroidSIFT An online classification service for Android malware Unavailability of online service
DroidSec/HickWall A tool that detects malicious behavior based on deep learning techniques
Drozer A Security Assessment Framework to analyze Android apps Enables analysis; Not a detection tool
FLowDroid A static taint analysis tool for Android apps
FixDroid An Android Studio plugin that provides warning to developers about potential vulnerabilities
FlaskDroid A generic security architecture for the Android OS that can serve as a flexible and effective ecosystem to instantiate different security solutions A generic security architecture for the Android OS; Not a detection tool
Gran'n Run A Java Library to help implement secure dynamic code loading in Android apps Enables secure development; Not a detection tool
HornDroid A static analysis tool that detects sensitive information leak in Android apps
IccTA A tool for inter-component Taint analysis in Android Fails to analyze apps due to logical failures
JAADS A static analysis that detects vulnerabilities in Android apps
LetterBomb A tool to automatically generate exploit for Android apps Not applicable to Ghera benchmarks
MARA A toolkit that puts together commonly used mobile application reverse engineering and analysis tools to assist in testing mobile applications Enables analysis; Not a detection tool
Maldrolyzer A framework to extract "actionable" data from Android malware (C&Cs, phone numbers etc.)
Mallodroid A tool to detect broken SSL certificate valiadation in Android apps
Marvin-SF An Android app vulnerability scanner
MobSF An automated pen-testing framework capable of performing static analysis and dynamic analysis to uncover vulnerabilities in Android apps
NVISO APK Scan An online service that scans malicious apps
PScout A tool to analyse permission usage in Android apps Not applicable to Ghera benchmarks
ProbeDroid A dynamic Java code instrumentation kit for Android apps Enables analysis; Not a detection tool
Qark A tool to look for several security related Android application vulnerabilities
SAAF A static analysis framework for Android apps Enables analysis; Not a detection tool
SMV_Hunter A tool-set for performing large-scale automated detection of SSL/TLS man-in-the-middle vulnerabilities in Android apps Targets API 18 and below while Ghera targets API 19 and higher
SPARTA A tool to verify that apps are not malware Cannot be used off-the-shelf; Apps need to be annotated with security types
ScanDroid A tool that scans Android apps for inconcsistent data flows Not maintained actively
SmaliSCA Static Code Analysis for Smali files Enables analysis; Not a detection tool
StaDyna A tool to address the problem dynamic code updates in Android apps Not applicable to Ghera benchmarks
TaintDroid A realtime monitoring tool that analyses how private information is obtained and released by Android apps Not applicable Android API 19 and above
TraceDroid A tool records the behavior of an Android app via automaed dynamic analysis Fails to process apks that target API levels below 19
TreeDroid A security policy specification and enforcement framework for Android Enforces security; Not a detection tool
VirusTotalTotal An online service that scans malicious apps
WeChecker A tool to check for Privilege Escalation Not maintained actively; No documentation for using the tool