Wait Until Page Is Loaded With Selenium WebDriver For Python

Selenium is an automation tool or a web framework used mainly in testing web applications across various browsers. Apart from testing web applications, we can also perform various tasks with selenium. With the help of selenium, we can also perform various web related tasks such as web scraping, web element interaction ( this includes form filling, clicking buttons and many more such things) etc. It also allows user to write scripts in various languages such as Python, Java, C# etc.

In this article, we are going to explore how we can perform "Wait until page is loaded with selenium WebDriver" task. We will explore many ways to achieve this task with clear and concise examples along with their respective explanations.

Understanding Page Load in Selenium

Page loading in selenium generally refers to the process where Selenium WebDriver waits for the web page to be fully loaded before we can perform any actions on it. If the page is not fully loaded, we might not perform our desired tasks as some of the web elements are not fully loaded. This can run us into errors. We can overcome this issue with the help of with the help of two methods, i.e., WebDriverWait method and page_load_strategy options. We will briefly discuss both the methods.

Using WebDriverWait

In Selenium, we can use WebDriverWait to wait for a specific condition to be true. If , there is any error or condition cannot be true we can simply throw an error or display the appropriate message to the user. This can generally used, when some elements might end up taking more time to load than expected. In such cases, this method is proved to be a lifesaver. We can always use this method, when we are not sure about how much time a particular element will end up taking to load on which we have decided to perform actions.

Example:

In this code, we have utilized WebDriverWait function to perform our task. We have created a user-defined function pageLoad where we used try and except block to handle any unpredictable errors. We have set the wait time seconds, if the page fails to load then control flows to the except block. We have also used By class of selenium to locate for a specific web element. If we found the element, then we will perform our actions and close the browser. Otherwise, we will throw the use appropriate message.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException


def pageLoad(url):
    dr = webdriver.Chrome()
    dr.get(pageUrl)
    wait = 5
    try:
      	#Checking our desired element is loaded or not
        element = WebDriverWait(dr, wait).until(
            ec.presence_of_element_located((By.CSS_SELECTOR, 
                                            '.HomePageTopicCard_homePageTopicCard__eePhS.row2')))
        print("You can procced to scrape the data.....\n")

        #We are now performing actions with our desired element
        c=1
        print("Our Web Dev Courses :-")
        for i in dr.find_elements(By.CSS_SELECTOR,
                                  '.HomePageTopicCard_homePageTopicCard__eePhS.row2 '):
          print(str(c)+". ",i.text)
          c += 1
    except TimeoutException:
      	#If our desired element is not found
        print("An ERROR Occured!!!!")

    dr.quit()

if __name__ == "__main__":
    
    #our url in which we want to visit and perform actions
    pageUrl = "https://www.geeksforgeeks.org"
    pageLoad(pageUrl)

Output:

Using page_load_strategy

In selenium, page_load_strategy is used define how the selenium WebDriver will handles the loading of a web pages. We can choose the specific behavior of page load i.e. NORMAL, EAGER or NONE. By default, it is NORMAL but we can change this to EAGER of NONE for faster execution

• NORMAL : In this, driver waits for the entire page to load with all its web elements before performing any actions.
• EAGER : In this, driver waits for only for the initial HTML document to be fully loaded but, it do not wait for any other web elements like images, subframes to load.
• NONE : In this, driver do not wait at all. We can start performing any actions after the get() method is called.

Example:

In this code, we have used page_load_strategy option to achieve our task. In this, we have defined our url and passed it to our user defined function pageLoad(). In this, we have set the page the page strategy as 'normal', you can set it to eager or none according to your use. As we have used normal, our driver waits for the page to be completely loaded before performing any actions on it.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options

def pageLoad(url):
    options = Options()
    
    # we initially set the page load stategy to normal 
    # which means it wait for the complete web pages to load.
    options.page_load_strategy = 'normal'  
    dr = webdriver.Chrome(options=options)
    dr.get(url)
    
    #we are performing action here
    print(dr.title)

    #closing the browser
    dr.quit()

if __name__ == "__main__":
    
    #Our url is defined here
    pageUrl = "https://www.geeksforgeeks.org"
    pageLoad(pageUrl)

Output:

Handling Asynchronous Content

In selenium, handling asynchronous content means waiting for the dynamic elements of an web-page to load. We can handle this asynchronous content using WebDriverWait function along with ExpectedConditions (elementToBeClickable, presenceOfElementLocated, or visibilityOf), such as waiting for elements to be clickable or visible or waiting for a class element to loaded fully. Through this we can ensure that, we will not run into any unexpected errors. This will help us to ensure that our automation scripts don't fail due to trying to interact with elements before asynchronous content has finished loading.

Conclusion

Selenium is a great automation tool we used mainly for testing web applications. But this is not the only use of selenium , we can also perform tasks such as web scrapping or web element interactions ( such as clicking buttons or filling forms). Page load is a process where selenium WebDriver waits for the web page to be fully loaded before performing any actions. We have covered two methods here to achieve our task, they are WebDriverWait method and page_load_strategy options. We have discussed all the mentioned points and tried to elaborate them accurately, concisely, and clearly.