Using Python to implement domain asset monitoring

Required prefix skills

Python simple use of Linux operating system simple use of DNS principle

steps

Perform port scanning for IP addresses

The IP address is found through the above steps. The next step is to perform port scanning for the IP address. However, before scanning, we need to do one more step, need to determine whether the current IP machine is alive.



You may immediately think of a ping method, but in the security world, many companies require servers to block ICMP. In other words, the ping method may return a timeout, but the machine is not necessarily not enabled, but actively blocked the request for you.



Here, we use the -spn parameter of Nmap for scanning. Take the IP address 47.119.137.0 corresponding to www.mengwa.store as an example:







We then implement this logic in code, using the python-nmap package, as follows:

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple/ --trusted-host pypi.tuna.tsinghua.edu.cn python-nmap
Copy the code

The code is as follows:

def check_ip_reachable(ip: str) :
    nm = nmap.PortScanner()
    xx = nm.scan(hosts=ip, arguments='-sP') # -spn also works
    if 'scan' in xx:
        i = xx.__getitem__('scan')
        if ip in i:
            j = i.__getitem__(ip)
            if 'status' in j:
                status = j['status']
                if 'state' in status:
                    state = status['state']
                    print(ip, state)
                    return state
    print(ip, 'down')
    return 'down'
Copy the code

The prerequisite for this step is that you have NMAP installed on your computer. Judging from this, if the machine where the IP resides is enabled, then we can conduct port scanning for it. The code is as follows:

Implement the port scan and return the result. Use NMAP
def port_scan(ip: str) :
    second = check_ip_reachable(ip)
    if second is not None and second == 'up':
        nm = nmap.PortScanner()
        xx = nm.scan(ip)
        print(nm.command_line())
        ans = []
        if 'scan' in xx:
            i = xx.__getitem__('scan')
            if ip in i:
                j = i.__getitem__(ip)
                if 'tcp' in j:
                    tcp = j['tcp']
                    for port in tcp:
                        ans.append(port)
        ans = sorted(ans)
        return ans
    else:
        write_to_file('output/' + time_now + '/ips.txt', ip + 'Unreachable \n')
    return []
Copy the code

The nm.command_line() method prints out the currently executed command line. (This package is essentially a command line call package that is then parsed)



Taking 47.119.137.0 as an example, the results are as follows:







In this method, common Ports scanned by NMAP are used. If necessary, you can add Ports parameters to the Scan method to customize the Ports to be scanned, or even all Ports.

Subdomain discovery

If my master domain asset has many secondary domain names, and I do not know which subdomains I have, then I need to do subdomain discovery. Then call the previous method for each subdomain to find its corresponding IP open port. Generally speaking, there are three ways to discover the subdomain name: the first way is to crawl the main domain name website to resolve the subdomain name from the content; The second way is to obtain results from external query websites, such as CRT, virustotal, fOFa, etc. The third way is to obtain the result through dictionary blasting. The DNS server is constantly polled to check whether it has the value recorded by A. If so, it indicates that the subdomain name exists; if not, it is temporarily considered that the subdomain name does not exist. The first method is to implement a crawler and extract the subdomain name in the website by regular expression after getting the content of the website. The extraction method is as follows:

def get_subdomains(value: str, domain: str) :
    x = f'(? P<subdomain>[a-zA-Z0-9.]+{domain}) '
    pattern = re.compile(x)
    y = pattern.findall(value)
    ans = set(a)for i in y:
        ans.add(i)
    print(ans)
    return ans
Copy the code

The second method, using the interface call, can see the related API interface. The third method is subdomain blasting. The core code is as follows

answers = await self.resolvers[j].query(cur_domain, 'A') Find A record
If the subdomain name exists, the subdomain name exists
Copy the code

Once you have completed the above steps and de-duplicated, you can use these subdomains as input to the previous steps. In this way, for the user needs to monitor a master domain name, the scope of monitoring is improved a lot. Do not use it to monitor non-existing domain names. The loss caused has nothing to do with me. The above test data are my existing domain name and IP.