How do I wait for this website to load using selenium in Python?
Currently I'm using beautifulSoup in my python web scraping project, however in one of the pages I need to scrape, I need to interact with a javascript element, so I'm being forced to used selenium (which I am not that familiar with). This is my code so far:
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
opts = Options()
opts.add_argument('--headless')
seleniumDriver = Firefox(options=opts, executable_path = 'D:\Programs\Python\Scripts\geckodriver.exe')
seleniumDriver.get("https://www.thecompleteuniversityguide.co.uk/courses/details/computing-bsc/57997898")
driverWait = WebDriverWait(seleniumDriver, 10)
driverWait.until(EC.invisibility_of_element_located((By.ID, "mainBody")))
moduleButton = seleniumDriver.find_element_by_xpath("//div[@class='mdldv']")#.find_element_by_tag_name("span")
print("MODULE BUTTON:", moduleButton)
moduleButton.click()
seleniumDriver.close()
Currently, I am getting a timeout error, however I'm certain that the mainBody ID element does exist. (I don't know how to use the By class, so I have no idea how it will work). Error Message:
Traceback (most recent call last):
File "D:/Web Scraping/selenium tests.py", line 12, in <module>
driverWait.until(EC.invisibility_of_element_located((By.ID, "mainBody")))
File "D:\Programs\Python\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
from Recent Questions - Stack Overflow https://ift.tt/3aL6L4E
https://ift.tt/eA8V8J
Comments
Post a Comment