A piece of code can’t be considered good if it doesn’t work properly.
One of the questions that arises from the previous sentence is, “So how can I know that my code is actually doing what it’s supposed to be doing?” The quick and easy answer is: Tests.
But how can you decide which types of tests to write and which testing suite to use? Well there is no single best answer to these questions. Here at Wuerth Phoenix for example, we decided to put our trust in Selenium Tests for testing our UI.
The Selenium test suite is a group of tools that allow you to automate web browsers across many platforms. These operations are highly flexible, allowing many options for locating UI elements and comparing expected test results against actual application behavior.
As good as it sounds, sadly I cannot forget to mention that among all the characteristics that Selenium has, there is one that actually hinders its functioning. It’s a well-known downside of using Selenium and is commonly referred to as “flakiness”. The word is used to describe the tendency of Selenium tests to randomly fail when run on an environment with high load.
Many attempts have been made world-wide to try and solve, or at least mitigate, the issue. Wuerth Phoenix’s R&D Team has tried, too.
It revolves around the concept of retrying failed tests, but we also take it one step further.
Just re-running tests would seem to work most of the time, but when the load is higher, it’s also the case that due to sheer chance, the probability that a test will randomly fail is also higher.
Digging deeper into the problem, our team discovered that many errors were caused either by waiting times, or by the fact that a series of methods that needed more time to be executed was run consecutively, and therefore overlapping each other or else not waiting for the other to finish. This problem arises specifically within those methods that imply the location of a certain element in the UI, and then either clicking on that element or submitting a value.
More precisely, the problem arises in between retrieving the reference to the element and the action performed on it. At some point the reference becomes stale because the DOM is somehow being refreshed. And so we are unable to access the desired element via that reference anymore.
We decided to create specific code blocks for find&click/find&Submit operations in which we could put a wrapper around the operation of getting the reference to the element, and then perform some action on it. This is done to make the operations as atomic as possible and make them run one after another.
An example of an atomic operation can be seen in this code snippet in which the two operations of fetching the reference to an element and clicking on it are performed. The method showed is called: waitForElementClickableAndClick().
def waitForElementClickable_normal(self, text=None, css=None, xpath=None, name=None, msg=None):
if(text!=None):
locator = (By.PARTIAL_LINK_TEXT, text)
elif(css != None):
locator = (By.CSS_SELECTOR, css)
elif(xpath!=None):
locator = (By.XPATH, xpath)
elif(name!=None):
locator = (By.NAME, name)
error = "Element is not found"
if(msg!=None):
error = msg
#link = WebDriverWait(self.driver,
WAIT).until(EC.element_to_be_clickable(locator()), message=error)
return WebDriverWait(self.driver, WAIT).until(EC.element_to_be_clickable(locator), message=error)
def waitForElementClickableAndClick_normal(self, text=None, css=None, xpath=None, name=None, msg=None):
element = self.waitForElementClickable_normal(text=text, css=css, xpath=xpath, name=name, msg=msg)
element.click()
return element;
def waitForElementClickableAndClick(self, text=None, css=None, xpath=None, name=None, msg=None):
return
self.retryMethod(self.waitForElementClickableAndClick_normal,[],{"text":text, "css":css, "xpath":xpath, "name":name, "msg":msg},retryCounter=RETRY,caller=sys._getframe().f_back.f_code.co_name)We then retry using this wrapper up to a maximum of 5 times until it works without problems.
The wrapper is run multiple times thanks to the newly implemented method: retryMethod().
Here’s a short snippet of code that describes it:
retryMethod(self, methodName, positional_arguments, dict_arguments, retryCounter=1, caller=None)methodName is the name of the method to retry, positional_arguments are arguments that don’t have a value-name pair and are passed as values, dict_arguments are arguments that have a name and a value, retryCounter is the number of times to retry when failure is encountered, and caller is the name of the caller to retryMethod().
To fully understand how the implementation of these wrapper methods affects our code, here is a snippet that compares before and after refactoring.
Before
def test_02_add_to_dashboard_interface(self):
locator = (By.CSS_SELECTOR, '#col1 #grafana-iframe')
WebDriverWait(self.driver, 30).until(
EC.frame_to_be_available_and_switch_to_it(locator)
)
self.driver.switch_to_default_content()
self.hover_iframe()
# The block below adds a dashboard by clicking one of the dropdown items
# But it fails randomly because between hovering over the menu item, waiting for dropdown to appear and clicking on the drop down item, something could go wrong.
# So we have to put the code block below in a retry
self.waitAndHover(css="#col1 .controls .tabs .dropdown-nav-item")
self.common.waitForVisible(xpath='//*[@class="dropdown-nav-item"]/ul/li/a[@class="analytics-add-to-dashboard icon-dashboard"]')
self.common.waitForElementClickableAndClick_normal(xpath='//*[@class="dropdown-nav-item"]/ul/li/a[@class="analytics-add-to-dashboard icon-dashboard"]')
self.common.waitForElementLocation(text='New Dashlet')
newDashletTitle = self.common.waitForElementLocation(css='#col1 .content h1', err="The New Dashlet title is not visible.")
self.assertTrue(newDashletTitle.text == 'Add Dashlet To Dashboard', "The New Dashlet page is not correctly loaded.")After
# Wrapper for the set of instructions to retry in succession
def add_to_dashboard(self):
# It's best to try to reset the conditions before a retry after a failure
# This hover is only put here to reset the hover cursor so that the second hover gets triggered correctly
self.waitAndHover(css="#col1 .controls .tabs li:first-child") # new reset line
self.waitAndHover(css="#col1 .controls .tabs .dropdown-nav-item")
self.common.waitForVisible(xpath='//*[@class="dropdown-nav-item"]/ul/li/a[@class="analytics-add-to-dashboard icon-dashboard"]')
self.common.waitForElementClickableAndClick_normal(xpath='//*[@class="dropdown-nav-item"]/ul/li/a[@class="analytics-add-to-dashboard icon-dashboard"]')
def test_02_add_to_dashboard_interface(self):
locator = (By.CSS_SELECTOR, '#col1 #grafana-iframe')
WebDriverWait(self.driver, 30).until(
EC.frame_to_be_available_and_switch_to_it(locator)
)
self.driver.switch_to_default_content()
self.hover_iframe()
# Retry method used on the new add_to_dashboard method
# Retries the add_to_dashboard up to 5 times upon encountering any type of exception
self.common.retryMethod(self.add_to_dashboard,[],{},retryCounter=5)
self.common.waitForElementLocation(text='New Dashlet')
newDashletTitle = self.common.waitForElementLocation(css='#col1 .content h1', err="The New Dashlet title is not visible.")
self.assertTrue(newDashletTitle.text == 'Add Dashlet To Dashboard', "The New Dashlet page is not correctly loaded.")Afterwards, we discovered that for some specific actions performed on the UI, this approach does not work because the number of operations needed for those actions exceeds the maximum number that an operation should perform in order to be considered atomic (which is two). This specific case is once again solved by re-wrapping the operations it needs.
This didn’t just solve our problems related to waiting times, it also allowed us to create specific methods to use in common cases such as: (1) finding an element and clicking on it, (2) waiting for the element to appear and clicking on it, (3) waiting for an element to be clickable and finally clicking on it, (4) finding an element and submitting its value, etc.
Angelo Rosace
Author
