Wiki

Clone wiki

Recon-ng / Development Guide

Contents


Development Notes

  • The "recon/utils" folder is for small 3rd party dependencies not available through the Python Package Index (PyPI). Place needed modules here and import as normal into modules e.g. from recon.utils import stuff.
  • The "modules" folder is crawled at runtime to establish the module tree from which all modules are loaded. Place new modules where it makes logical sense, or create a new folder to expand the module tree.
  • During module development, developers need to repeatedly reload framework modules to test code changes without restarting the framework. The "reload" command provides the capability to reload framework modules within the global context while maintaining command history and global options settings.

Pull Requests

  • Make all contributions using the "fork and pull" method to the "staging" branch.
  • Squash all commits into a single commit before submitting pull requests. This can be done from the command line by checking out the source fork with the git checkout <fork> command and issuing the git rebase -i command. Choose "squash" for all of your commits, except the first one, and consolidate the commit messages to a single message that summarizes the pull request. Push the modified fork to the remote repository with the git push -f command. This technique will automatically update existing pull requests from the affected source fork.

Module Template

# module required for framework integration
from recon.core.module import BaseModule
# mixins for desired functionality
from recon.mixins.resolver import ResolverMixin
from recon.mixins.threads import ThreadingMixin
# module specific imports
import os

class Module(BaseModule, ResolverMixin, ThreadingMixin):

    # modules are defined and configured by the "meta" class variable
    # "meta" is a dictionary that contains information about the module, ranging from basic information, to input that affects how the module functions
    # below is an example "meta" declaration that contains all of the possible definitions
    # all items are optional and may be omitted

    meta = {
        'name': 'Hostname Resolver',
        'author': 'Tim Tomes (@LaNMaSteR53)',
        'version': 'v0.0.1',
        'description': 'Resolves IP addresses to hosts and updates the database with the results.',
        'comments': (
            'Note: Nameserver must be in IP form.',
            '\te.g. 1.2.3.4',
        ),
        'query': 'SELECT DISTINCT host FROM hosts WHERE host IS NOT NULL',
        'options': (
            ('nameserver', '8.8.8.8', 'yes', 'ip address of a valid nameserver'),
        ),
    }

    # "query" is optional and determines the "default" source of input
    # the "SOURCE" option is only available if "query" is defined

    # "options" expects a tuple of tuples containing 4 elements:
    # 1. the name of the option
    # 2. the default value of the option (strings, integers and boolean values are allowed)
    # 3. a boolean value (True or False) for whether or not the option is mandatory
    # 4. a description of the option

    # optional method
    def module_pre(self):
        # override this method to execute code prior to calling the "module_run" method
        # returned values are passed to the "module_run" method and must be captured in a parameter
        return value

    # mandatory method
    # the second parameter is required to capture the result of the "SOURCE" option, which means that it is only required if "query" is defined within "meta"
    # the third parameter is required if a value is returned from the "module_pre" method
    def module_run(self, hosts, value):
        # do something leveraging the api methods discussed below
        # local option values can be accessed via self.options['name']
        # use the "self.workspace" class property to access the workspace location
        # threading can be used anywhere with the module through the usage of the "self.thread" api call
        # the "self.thread" api call requires a "module_thread" method which acts as the worker for each item in a queue
        # "self.thread" takes at least one argument
        # the first argument must be an iterable that contains all of the items to fill the queue
        # all other arguments get blindly passed to the "module_thread" method where they can be accessed at the thread level
        self.thread(hosts, url, headers)

    # optional method
    # the first received parameter is required to capture an item from the queue
    # all other parameters passed in to "self.thread" must be accounted for
    def module_thread(self, host, url, headers):
        # never catch KeyboardInterrupt exceptions in the "module_thread" method as threads don't see them
        # do something leveraging the api methods discussed below

Output Methods

Recon-ng enables the consistent display of output to the user in multiple styles. The following methods can be called anywhere within a module to present a consistent interface to the user.

  • Format and display exceptions.
self.print_exception([line='<string>']):
  • Format and display errors.
self.error('<string>')
  • Format and display normal output.
self.output('<string>')
  • Format and display emphasized output.
self.alert('<string>')
  • Format and display output if in verbose mode.
self.verbose('<string>')
  • Format and display output if in debug mode.
self.debug('<string>')
  • Format and display a heading.
    • level is the style of heading. Currently there are only two style options: 0 and 1.
self.heading('<string>'[, level=1])
  • Build, display, and store an ASCII table of given data.
    • tdata is the table data as a two dimensional list (list of lists), with each list representing a row of data. Each list must be the same length.
    • header (optional) is a list containing the header values. By default, tables are displayed without a header row.
    • title (optional) is the title of the table. By default, tables are displayed without a title.
self.table(tdata[, header=[]][, title=''])

Examples:

self.error('This is an error.')
'[!] This is an error.         # all red'

self.output('This is normal output.')
'[*] This is normal output.    # blue highlights'

self.alert('This is important output!')
'[*] This is important output! # green highlights'

self.verbose('This is verbose output.')
'[*] This is verbose output.   # blue highlights'

self.debug('This is debug output.')
'[*] This is debug output.     # blue highlights'

self.heading('This is a level 0 heading', 0)
"""
-------------------------
THIS IS A LEVEL 0 HEADING
-------------------------
"""

self.heading('This is a level 1 heading')
"""
  This Is A Level 1 Heading
  -------------------------
"""

self.table(table_data, header=['host', 'ip_address'], title='Hosts')
"""
  +-------------------------------------+
  |                Hosts                |
  +-------------------------------------+
  |        host        |   ip_address   |
  +-------------------------------------+
  | www.apache.org     | 192.87.106.229 |
  | www.google.com     | 74.125.225.176 |
  | www.twitter.com    | 199.59.148.10  |
  | www.whitehouse.gov | 157.238.74.67  |
  +-------------------------------------+
"""

Database Interaction Methods

Interacting with the database is easy in Recon-ng. There are core database interaction methods which allow the addition of records to the default tables in the database. There is also a generic query method for all other queries. In each case, the work of setting up the connection, extracting the results, and closing the connection is done for you.

  • Add a host to the database and return the affected row count.
    • host is the FQDN of the host as a string.
    • ip_address (optional) is the IP address of the host as a string.
    • region (optional) is the city, state or region of where the host is located.
    • country (optional) is the country of where the host is located.
    • latitude (optional) is the latitude of where the host is located.
    • longitude (optional) is the longitude of where the host is located.
self.add_hosts(host[, ip_address=None][, region=None][, country=None][, latitude=None][, longitude=None])
  • The above is one example of a series of methods that exist to add records to the default tables in the database. The methods are named with the following naming scheme: self.add_<table>(<parameters>). Table names are available via the "show schema" command. See the source code for each of the methods in the "recon/core/framework.py" module for parameter details.
  • Query the database and return the results as a list (SELECT) or affected rowcount if no results are returned (INSERT, DELETE, UPDATE, etc.). A FrameworkException is raised if an error is encountered during the query process.
    • query is the SQL statement to process.
    • values (optional) is a tuple of values to use with the query string for parameter substitution (parameterized queries).
self.query(query[, values=()])

API Key Management Methods

Some Recon-ng modules require the use of an API key, OAuth Token, etc. To prevent users from having to continually input keys and regenerate tokens, Recon-ng provides methods which assist in storing, managing and accessing these items.

  • Fetch a key from the local key store.
    • name is the unique name for the key. A FrameworkException is raised if no such key exists.
self.get_key(name)
  • Store a key in the local key store.
    • name is the unique name for the key. If not unique, the existing key will be overwritten.
    • value is the key string to store.
self.add_key(name, value)
  • Delete a key from the local key store.
    • name is the unique name for the key.
self.delete_key(name)

Third Party API Methods

Some Recon-ng modules may require the use of popular search engines and social media sites with complex OAuth authentication schemes. Recon-ng provides developers with an easy way to create OAuth tokens for the LinkedIn and Twitter APIs, and interface with the Google, Bing, and Shodan search APIs.

  • Return an OAuth token for use with Twitter API requests.
self.get_twitter_oauth_token()
  • Search Twitter using the Twitter API and return a list of the results.
    • payload is the search string submitted to the search API.
self.search_twitter_api(payload)
  • Search Shodan using the Shodan Search API and return a list of the results.
    • query is the search string submitted to the search API.
    • limit (optional) is the maximum number of results pages to return. A value of "0" returns all results.
self.search_shodan_api(query[, limit=0])
  • Search Bing using the Bing Azure Search API and return a list of the results.
    • query is the search string submitted to the search API.
    • limit (optional) is the maximum number of results pages to return. A value of "0" returns all results.
self.search_bing_api(query[, limit=0])
  • Search Google using the Google Custom Search Engine (CSE) API and return a list of the results.
    • query is the search string submitted to the search API.
    • limit (optional) is the maximum number of results pages to return. A value of "0" returns all results.
self.search_google_api(query[, limit=0])

Web Request Methods

The most important capability of a tool which specializes in web based reconnaissance is the ability to make web requests. Recon-ng relieves the burden of complicated request building logic by providing a custom method for handling web requests.

  • Make a web request and return a response object.
    • url is the base URL of the request.
    • method (optional) is the method of the request. Currently, only "GET" or "POST" are available.
    • timeout (optional) is an integer representing the socket timeout for the request. If not set, the socket timeout defaults to the global option setting.
    • payload (optional) is a dictionary of name:value pairs to be encoded as request parameters. payload should be used for "GET" and "POST" methods as the request method will encode and build the request as needed for the given method.
    • headers (optional) is a dictionary of name:value pairs to be added as request headers.
    • cookiejar (optional) is an instance of the cookielib.CookieJar class to manage cookies between requests.
    • auth (optional) is a tuple "(username, password)" containing the username and password to use with basic authentication.
    • content (optional) is a string indicating the content subtype of the POST payload. By default, the standard for a URL encoded POST payload is applied. Currently, only the default and "JSON" subtypes are available. Submitting a content subtype for any method other than a POST request raises a RequestException.
    • redirect (optional) is a boolean value which determines if redirects are followed.
    • agent (optional) is a string which overwrites the global user-agent setting.
self.request(url[, method='GET'][, timeout=None][, payload={}][, headers={}][, cookiejar=None][, auth=()][, content=''][, redirect=True][, agent=None])

The resulting response object after a successful request allows quick access to all of the information required for further action.

Example:

resp = self.request(...)
resp.url         # The full request url as a string.
resp.status_code # The response status code as an integer.
resp.headers     # A dictionary of the response headers.
resp.encoding    # A string representing the encoding used in the response.
resp.cookiejar   # The updated cookielib.CookieJar object.
resp.raw         # The raw response body.
resp.text        # The response body decoded on-the-fly using the "resp.encoding" charset.
resp.json        # A JSON object if the decoded "resp.text" is valid JSON.
resp.xml         # A XML object (xml.etree.ElementTree) if the decoded "resp.text" is valid XML.

Recon-ng attempts to auto detect the charset encoding by analyzing the response headers. If not found, 'utf-8' is used as the default encoding. The encoding property is writable and will effect how the "resp.text" property is decoded. WARNING: This will also effect the ability of Recon-ng to build JSON and XML objects out of "resp.text".

  • Build and return a cookielib.Cookie object for use with a cookielib.CookieJar object.
    • name is the name of the cookie.
    • value is the value of the cookie.
    • domain is the domain to which the the cookie is bound.
    • path (optional) is the path within the domain to which the cookie is bound.
self.make_cookie(name, value, domain[, path='/'])

Mixins

Some web content is just too difficult to parse as a string. Therefore, Recon-ng includes the Mechanize python library to provide access to a powerful web content parsing engine.

  • Build and return a mechanize browser object configured with the framework's global options.
from recon.mixins.browser import BrowserMixin
...
br = self.get_browser()
  • Build and return a dnspython default resolver object configured with the framework's global options.
from recon.mixins.resolver import ResolverMixin
...
br = self.get_resolver()

Threading connections to internet resources drastically decreases time wasted due to network latency. Recon-ng speeds things up by providing developers with the ability to introduce threading into their modules.

  • Execute module functionality within threads.
from recon.mixins.threads import ThreadingMixin
...
    def module_run(self, elements):
        var1 = something
        var2 = something_else
        self.thread(elements, var1, var2)

    # worker function
    def module_thread(self, element, var1, var2):
        ...

RPC Interface

Recon-ng comes equipped with an RPC interface, ./recon-rpc.py, to provide 3rd party tools with a standardized protocol for accessing the data gathered and stored by the framework. Both JSONRPC and XMLRPC protocols are supported, as well as MultiCall objects, and session IDs are used to provide a multi-user environment. Use ./recon-rpc.py -h for information on runtime options.

Example RPC clients:

# XMLRPC
import xmlrpclib
client = xmlrpclib.Server('http://localhost:4141')
sid = client.init()
client.use('recon/profiles-profiles/profiler', sid)
client.set('source', 'lanmaster53', sid)
results = client.run(sid)
print results

# JSONRPC
import jsonrpclib
client = jsonrpclib.Server('http://localhost:4141')
sid = client.init()
client.use('recon/profiles-profiles/profiler', sid)
client.set('source', 'lanmaster53', sid)
results = client.run(sid)
print results

Updated