This repository documents an experiment to measure how well the lean vulnerability benchmarks in Ghera repository represent real world Android apps in terms of Android API involved in the creation of vulnerabilities.

We say a groups of apps is representative of real world apps if the use of Android API in the group of apps is similar or conservative when compared to the use of Android API in real world apps. By Android API, we mean

  1. published methods and fields of Android SDK used to program Android apps and
  2. elements and attributes (including their values) used in the manifest file of Android apps.

There are many nuances to consider while measuring similar API usage, e.g., consider code structure, consider data and control flows between API uses in an app. In this regard, given a specific part p of an API, we measure similar or conservative use of p as the difference between the proportion of apps in a group Pg that use p and the proportion of real world apps Pr that use p. If this difference dP = Pg - Pr is negative or zero, then we consider the use of p in group of apps to be similar or conservative to the use of p in real world apps.

What do we consider?

  1. Only lean benchmarks because we are interested in the representativeness of lean benchmarks.
  2. Only benign apps from lean benchmarks because we are interested in representativeness in terms of APIs likely involved in the creation of vulnerabilities. (This exercise can be trivially extended for both malicious and secure apps in the benchmarks.)
  3. Only the parts of Android API used in lean benchmark apps. Since lean benchmarks are minimalistic, they will use far fewer features of Android API than real world apps. Specifically, we considered
    1. fields and methods referenced in an APK (as reported by apkanalyzer tool) that occur in the namespace begining with any of the following prefixes.
      • android
      • java
      • org
    2. aspects of app manifests
      1. value of attributes of specific elements in an app manifest
        • intent-filter/action@name
        • intent-filter/category@name
        • permission@protectionLevel
        • uses-library@name
        • uses-permission@name
        • uses-permission-sdk-23@name
        • uses-permission-sdk-23@exported
      2. presence of elements in an app manifest
        • action
        • category
        • permission-tree
        • uses-permission
        • uses-library
        • grant-uri-permission
      3. presence of attributes of specific elements in an app manifest
        • permission@protectionLevel
        • activity@allowTaskReparenting
        • application@allowTaskReparenting
        • activity@launchMode
        • activity@permission
        • service@permission
        • receiver@permission
        • provider@permission
        • activity@taskAffinity
        • application@taskAffinity
        • application@allowBackup
        • application@debuggable
        • data@scheme
        • data@path
        • uses-permission@path
        • data@pathPrefix
        • uses-permission@pathPrefix
        • intent-filter@priority
        • path-permission@readPermission
        • provider@readPermission
        • path-permission@writePermission
        • provider@writePermission
        • activity@exported
        • service@exported
        • receiver@exported
        • provider@exported
  4. Only the use and non-use of various parts of Android API in apps as we are interested in the existence of a vulnerability in apps and not the number of instances of a vulnerability in apps.


How do we measure representativeness?

  1. Identify the parts of Android API used by the lean benign benchmark apps.
  2. For an API part X,
    1. From AndroZoo repository, draw random samples of APKs with replacement where each sample has 40 data points and each APK has minSDK >= 14 and targetSDK >= 19 and calculate the sampling proportion Pr(X) of APKs that use X.
    2. Draw 40 random samples with replacement from lean benign benchmarks apps and calculate the sampling proportion Pg(X) of apps that use X. .
    3. Compare the proportions Pg(X) and Pr(X) with
      • confidence interval: 0.95
      • null hypothesis: Pg(X) <= Pr(X)
      • alternative hypothesis: Pg(X) > Pr(X)
    4. Interpretation:
      • p-value < 0.05: the use of X in lean benign benchmark apps is probably not representative of its use in real world Android apps.
      • p-value >= 0.05: the use of X in lean benign benchmark apps may be representative of its use in real world Android apps.

Given the large population of real world Android apps, we measure representativeness at sample size of 40 by considering only apps with *minSDK

= 14 and targetSDK >= 19*.

Given the number of Android API versions, we measure representativeness for API versions 19, 21, 22, 23, 24, and 25 at sample size of 40. Given an API version K, we consider only real world apps with 14 <= app.minSDK <= K <= app.targetSDK.

Required Software

  1. R v3.4.4
    • Required libraries:
      • readr v1.1.1
      • docopt v0.4.5
      • ggplot2 v2.2.1
      • parallel v3.4.4
  2. Groovy v2.4.13


  1. Open a terminal and change folder to root folder of this repo.
  2. Place the APKs of apps to be used to calculate Ghera's representativeness in a folder in input folder and update REAL_WORLD_APKS variable in scripts/masterScriptsh.
    • We extracted a random selection of APKs from AndroZoo repository and placed them in this folder. The SHA hashes of our selection is available in input/androzoo/sha.txt.
  3. Place apks that will provide baseline APIs in relevance calculation in input/baseline folder.
  4. Place (benign|malicious|secure) apps from Ghera benchmarks in input/ghera/(benign|malicious|secure) folders.
    • We used Ghera APKs tagged with RekhaEval-3 in Ghera repo.
  5. Set ANDROID_HOME variable in scripts/
  6. Set OUTPUT_FOLDER variable in scripts/
  7. Execute
    • At the start of execution, you will be prompted to answer two questions.

If you want to rerun the script on the data in the repository, then unzip output-androzoo/real-world*bz2 files before running


Please refer to output-androzoo/ for description of generated files.


Copyright (c) 2018, Kansas State University

Licensed under BSD 3-clause "New" or "Revised" License


  1. Joydeep Mitra
  2. Venkatesh-Prasad Ranganath