Bing Reverse IP Lookup Module

Issue #6 resolved
Tim Tomes
repo owner created an issue

Build a module which leverages Bing's IP lookup capability to conduct a reverse lookup of a hosts IPs, returning all hosts which share the same IP.

Comments (11)

  1. thrapt

    Should this be a module by itself? It is possible to accomplish this by adding one option to the current module and adding/modifying about 5 lines of code...

  2. thrapt

    I think I found a problem with the approach used for bing.

    Excluding results from the search concatenating previous results with the '-site:' keyword generates huge payloads and Bing is returning 404 when I send a big enough request. Can anyone confirm that?

    In the case I tested:

    Payload size: 63 bytes - 1st request for results - OK

    Payload size: 1524 bytes - 2nd request for results - OK

    Payload size: 2912 bytes - 3rd request for results - 404

  3. Tim Tomes reporter

    I did some testing on this a long time ago when I wrote the host name enumeration module. I thought I accounted for it. Do you encounter this error with that module too?

    Each search engine reacts differently to long query strings. Some have a limit, and some react unexpectedly. Some things to try:

    1. Try using the POST method and see if they are allowing method interchange.
    2. Try using the Bing API.
    3. Figure out the max length and stop the script prematurely.
    4. Handle the 404 error and stop the script.
  4. thrapt

    Yes, bing module suffers from the same problem.

    Method interchange doesn't seem to work.

    I don't think it's wise to stop the script, specially doing a IP lookup. In certain cases it'll leave too many hosts out.

    Parsing the results by hand isn't something pretty either.

    Is the API the only solution here?

  5. Tim Tomes reporter

    Possibly, but I've seen the API have problems too. Also, in my experience, the search engine APIs don't seem to return as many results. What options are you using to generate the error? Is it repeatable? I'd like to check it out.

  6. thrapt

    For testing purposes I'm doing a search for a top level domain so the results are huge. The script fails after the 3rd request, when the length of my payload exceeds ~2900 bytes and Bing responds with a 404. No matter how many times I try the results are the same.

  7. Tim Tomes reporter

    Take a look at this. It's the original recon-ng script. I did some testing when developing this and it looks like i discovered that "typical URI max length is 2048 (starts after top level domain)". It doesn't look like i took that into account when building the module for the framework. If you get some time, do some testing and see if that helps things.

  8. kc57

    I have already created a fully functional module for bing ip searches that can seach by ip or cidr range and return a list of unique domains. It uses the bing api and works great. It returns plenty of results if there are enough available. I can get the code in when I get back home in a few hours.

  9. kc57

    I copied in the code i used to parse CIDR ranges from https://github.com/bsterne/bsterne-tools/blob/master/cidr/cidr.py but the rest of the code is all my own. I'll have to take a look at your updates, i had no idea you had already worked on this :(

    import framework
    # unique to module
    
    import requests
    import json
    import sys
    import re
    from urlparse import urlparse
    
    class Module(framework.module):
    
        URL = 'https://api.datamarket.azure.com/Data.ashx/Bing/Search/v1/Web?Query=%%27%(query)s%%27&Market=%%27en-US%%27&$top=50&$format=json'
    
        def __init__(self, params):
            framework.module.__init__(self, params)
            self.register_option('CIDR', '', 'yes', 'The CIDR range to search')
            self.register_option('verbose', self.goptions['verbose']['value'], 'yes', self.goptions['verbose']['desc'])
            self.info = {
                         'Name': 'Bing IP Enumerator',
                         'Author': 'Rob Simon (@_Kc57)',
                         'Description': 'Harvests hosts from Bing.com by using the \'IP\' search operator. This module updates the \'hosts\' table of the database with the results.',
                         'Comments': []
                         }
    
        def do_run(self, params):
            if not self.validate_options(): return
            # === begin here ===
            self.API_KEY = self.manage_key('bing_key', 'Bing API Key')
            if not self.API_KEY: return
            self.ipSearch()
    
    
        def remove_dups(self,seq, idfun=None): 
            # order preserving
            if idfun is None:
                def idfun(x): return x
            seen = {}
            result = []
            for item in seq:
                marker = idfun(item)
                # in old Python versions:
                # if seen.has_key(marker)
                # but in new ones:
                if marker in seen: continue
                seen[marker] = 1
                result.append(item)
            return result
    
        def search(self,query, **params):
            r = requests.get(self.URL % {'query': query}, auth=(self.API_KEY, self.API_KEY))
            return json.loads(r.content)
    
        def searchUrl(self,url):
            r = requests.get(url, auth=(self.API_KEY, self.API_KEY))
            return json.loads(r.content)
    
        def getDomains(self,results):
            domains = []
            for result in results['d']['results']:
                domain = urlparse(result['Url']).hostname
                domains.append(domain)
            return domains
    
        def ipSearch(self):
            cidr = self.options['cidr']['value']
            if self.validateCIDRBlock(cidr):
                ips = self.getCIDR(self.options['cidr']['value'])
            else:
                ips = []
                ips.append(cidr)
            domains = [['host','ip_address']]
            for ip in ips:
                nextUrl = self.URL % {'query': 'ip:' + ip}
                self.output('searching %s' % ip)
                tmp = []
                while True:
                    # Perform the search
                    results = self.searchUrl(nextUrl)
                    # Get the next search URL if available
                    nextUrl = results.get('d').get('__next')
                    # If the next URL is not empty add the JSON format parameter
                    if nextUrl is not None:
                        nextUrl += '&$format=json'
                    # Get the list of domains from the results
                    tmp += self.getDomains(results)
                    if nextUrl is None:
                        break
                tmp = self.remove_dups(tmp)
                tmp.sort()
                for host in tmp:
                    domains.append([host,ip])
                    self.add_host(host,ip)
    
    
            self.table(domains, True)
    
            #for domain in domains:
            #    self.output('Domain: %s' % domain)
    
    
        # convert an IP address from its dotted-quad format to its
        # 32 binary digit representation
        def ip2bin(self,ip):
            b = ""
            inQuads = ip.split(".")
            outQuads = 4
            for q in inQuads:
                if q != "":
                    b += self.dec2bin(int(q),8)
                    outQuads -= 1
            while outQuads > 0:
                b += "00000000"
                outQuads -= 1
            return b
    
        # convert a decimal number to binary representation
        # if d is specified, left-pad the binary number with 0s to that length
        def dec2bin(self,n,d=None):
            s = ""
            while n>0:
                if n&1:
                    s = "1"+s
                else:
                    s = "0"+s
                n >>= 1
            if d is not None:
                while len(s)<d:
                    s = "0"+s
            if s == "": s = "0"
            return s
    
        # convert a binary string into an IP address
        def bin2ip(self,b):
            ip = ""
            for i in range(0,len(b),8):
                ip += str(int(b[i:i+8],2))+"."
            return ip[:-1]
    
        # get a list of IP addresses based on the CIDR block specified
        def getCIDR(self,c):
            ips = []
            parts = c.split("/")
            baseIP = self.ip2bin(parts[0])
            subnet = int(parts[1])
            # Python string-slicing weirdness:
            # "myString"[:-1] -> "myStrin" but "myString"[:0] -> ""
            # if a subnet of 32 was specified simply print the single IP
            if subnet == 32:
                ips.append(bin2ip(baseIP))
            # for any other size subnet, print a list of IP addresses by concatenating
            # the prefix with each of the suffixes in the subnet
            else:
                ipPrefix = baseIP[:-(32-subnet)]
                for i in range(2**(32-subnet)):
                    ips.append(self.bin2ip(ipPrefix+self.dec2bin(i, (32-subnet))))
            return ips
    
        # input validation routine for the CIDR block specified
        def validateCIDRBlock(self,b):
            # appropriate format for CIDR block ($prefix/$subnet)
            p = re.compile("^([0-9]{1,3}\.){0,3}[0-9]{1,3}(/[0-9]{1,2}){1}$")
            if not p.match(b):
                return False
            # extract prefix and subnet size
            prefix, subnet = b.split("/")
            # each quad has an appropriate value (1-255)
            quads = prefix.split(".")
            for q in quads:
                if (int(q) < 0) or (int(q) > 255):
                    return False
            # subnet is an appropriate value (1-32)
            if (int(subnet) < 1) or (int(subnet) > 32):
                return False
            # passed all checks -> return True
            return True
    
  10. Log in to comment