How to resolve "coroutine 'Launcher.killChrome' was never awaited" error?
I am trying to automate searching on a website. I am using requests_html in headless mode for it.
I have written the following code in ipython. This works fine if k<=20 otherwise it just hangs up. I neither understand the cause of this problem nor how to debug it.
k=20
import asyncio
from requests_html import AsyncHTMLSession
from bs4 import BeautifulSoup
async def asyncurl(url):
r = await asession.get('https://pricehistoryapp.com/')
script = """
() => {
setTimeout(function(){
var element = document.getElementsByTagName("input");
document.getElementsByClassName("px-3 py-2 outline-none rounded-none disabled:bg-gray-200")[0].setAttribute("value", "%s");
document.getElementsByClassName("px-3 py-2 outline-none rounded-none disabled:bg-gray-200")[0].click();
document.getElementsByClassName("px-3 py-2 outline-none rounded-none disabled:bg-gray-200")[0].dispatchEvent(new Event('change', { 'bubbles': true }));
document.getElementsByClassName('bg-secondary-600 text-white py-1 font-bold shadow-lg')[0].click();
}, 1190);
}
"""%(url)
await r.html.arender(sleep=7, keep_page=True, scrolldown=1,timeout=105,script=script)
return r.html.html
results=[]
start=time.time()
ii=0
#urls=list of urls
for i in range(1,5,1):
if __name__ == '__main__':
try:
asession = AsyncHTMLSession()
list_of_lambdas = [lambda url=url: asyncurl(url) for url in urls[k*(i-1):k*(i)]]
results.append(asession.run(*list_of_lambdas))
asession.close()
ii+=1
print("Completed records till "+str(i*k)+" itteration: "+str(ii)+" records in "+str(time.time()-start)+" seconds")
except:
continue
name=[BeautifulSoup(p,'lxml').find('h1',{"class":True}).text for pp in results for p in pp]
Please guide me toward a workaround to this problem. Thanks in advance.
When I try to get out of hang by ^+C it gives below error repeatedly but remains hung.
<ipython-input-3-76d514892a35>:11: RuntimeWarning: coroutine 'Launcher.killChrome' was never awaited
continue
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
^C^CTraceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pyppeteer/launcher.py", line 153, in _close_process
self._loop.run_until_complete(self.killChrome())
File "/usr/lib/python3.8/asyncio/base_events.py", line 592, in run_until_complete
self._check_running()
File "/usr/lib/python3.8/asyncio/base_events.py", line 552, in _check_running
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
Also, any other methods for speeding up automated searching (like running the code inside for loop in multi-threads) would be appreciated.
Comments
Post a Comment