How to get response status of large list of subdomains?

I have been trying to check the status of all these subdomains all at once and I have tried multiple techniques even grequests and faster than requests wasn't much helpful and then I started using asyncio with aiohttp and it is slower than normal requests library now. Also I checked that it wasn't actually sending the requests asynchronously rather it was sending one after another.

I know that "await resp.status" has issues because resp.status does not support await but I tried removing it and it's still the same.

import aiohttp
import asyncio
import time

start_time = time.time()


async def main():
#List of 1000 subdomains , Some subdomains do not exist 

    data = [ "LIST OF 1000 SUBDOMAINS" ]

    async with aiohttp.ClientSession() as session:
        for url in data:
            pokemon_url = f'{url}'
            try:
                async with session.get(pokemon_url, ssl=False) as resp:
                    pokemon = await resp.status
               #If subdomain exists then print the status
                    print(pokemon)
            except:
               #else print the subdomain which does not exist or cannot be reached

                print(url)

asyncio.run(main())
print("--- %s seconds ---" % (time.time() - start_time))


Comments

Popular posts from this blog

Spring Elasticsearch Operations

Network Error and Timeout on Authorize.net JS

Object oriented programming concepts (OOPs)