Home ›

Automated frontend testing for the lazy. With Firefox, Selenium, Webdriver and Hudson/Jenkins. Now with 50% less programming.

Nico Heid's picture
by Nico Heid on Wed, 12/07/2011 - 11:40

introduction

We all know: Programmers are lazy. And that is probably why we like automation so much. This is why implemented some frontend tests lately at +inovex

Let's imagine we have a (pre)live system where we want to check that our ultra cool form is working as intended. Because sometimes we forget a dependency or our integration tests didn't pick it up. Testing the frontend can be a helpful way of seeing that the whole chain, from backend to frontend, is working properly. At least that's what we hope. So please don't forget to unit test your code too. ;)

So let's record frontend usage with Firefox and SeleniumIDE, export our recordings into a Webdriver JUnit test and let our headless Hudson run it periodically with the help of xvfb.

Google "Find Chuck Norris" Selenium Test

So which form is more fun than testing if the old chuck norris easteregg is still working in google's "I'm Feeling Lucky" search.

recording the frontend test

Start you Firefox and open SeleniumIDE (Tools -> SeleniumIDE). Now go to http://google.com, enter "Find Chuck Norris" in the search bar and hit "I'm Feeling Lucky". We should end up on a site telling us, that Google won't search for Chuck. Even a giant like Google is scared of someone who could actually roundhouse kick the s*** out of them counted to infinity, twice.

Now export the test as JUnit Webdriver test case (File -> Export Test Case As .. -> JUnit4 (WebDriver) and you should end up with something similar to this:

  1. package com.example.tests;
  2.  
  3. import java.util.regex.Pattern;
  4. import java.util.concurrent.TimeUnit;
  5. import org.junit.*;
  6. import static org.junit.Assert.*;
  7. import static org.hamcrest.CoreMatchers.*;
  8. import org.openqa.selenium.*;
  9. import org.openqa.selenium.firefox.FirefoxDriver;
  10. import org.openqa.selenium.support.ui.Select;
  11.  
  12. public class Findchuck {
  13.         private WebDriver driver;
  14.         private String baseUrl="";
  15.         private StringBuffer verificationErrors = new StringBuffer();
  16.         @Before
  17.         public void setUp() throws Exception {
  18.                 driver = new FirefoxDriver();
  19.                 driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
  20.         }
  21.  
  22.         @Test
  23.         public void testFindchuck() throws Exception {
  24.                 driver.get("/");
  25.                 driver.findElement(By.id("lst-ib")).clear();
  26.                 driver.findElement(By.id("lst-ib")).sendKeys("find chuck norris");
  27.                 driver.findElement(By.name("btnI")).click();
  28.         }
  29.  
  30.         @After
  31.         public void tearDown() throws Exception {
  32.                 driver.quit();
  33.                 String verificationErrorString = verificationErrors.toString();
  34.                 if (!"".equals(verificationErrorString)) {
  35.                         fail(verificationErrorString);
  36.                 }
  37.         }
  38.  
  39.         private boolean isElementPresent(By by) {
  40.                 try {
  41.                         driver.findElement(by);
  42.                         return true;
  43.                 } catch (NoSuchElementException e) {
  44.                         return false;
  45.                 }
  46.         }
  47. }

finetuning the test

This is not quite what we need. So let's finetune it a bit. You can find the result on github.

First of all, I need a build tool. This needs to run on hudson later, so I personally mostly use maven for this. So I set up a simple maven project and added the dependencies needed.

We're going to verify the following pageflow:

  • go to the google search page
  • enter "find chuck norris"
  • click I'm Feeling Luck
  • find the message "Google won't search for Chuck Norris because it knows you don't find Chuck Norris, he finds you" on the result page

I ran into a few things I didn't like with the initial export. There seems to be some javascript that actually brings you to a different search layout once you start entering text. So I decided to disable javascript. This can be done via the profile class. So our test setup looks like this:

  1.  @Before
  2.     public void setUp() throws Exception {
  3.         FirefoxProfile profile = new FirefoxProfile();
  4.         profile.setPreference("javascript.enabled", false);
  5.         driver = new FirefoxDriver(profile);
  6.         driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
  7.     }

As we can not expect to find Chuck immediately (he can only be found if he wants to be), we have to wait a bit to make sure we're on the result page. We're going to use .findElement to look for an item on the result page. This will wait for the element to appear, discarding NoSuchElementExceptions until we run in a timeout. In this setup after 30 seconds.

  1.  @Test
  2.     public void testFindchuck() throws Exception {
  3.         driver.get("http://google.com");
  4.         driver.findElement(By.id("lst-ib")).sendKeys("find chuck norris");
  5.         driver.findElement(By.name("btnI")).click();
  6.         driver.findElement(By.id("t1a"));
  7.         assertTrue(StringUtils.contains(driver.getPageSource().toString(),
  8.                 "Google won't search for <b>Chuck Norris</b> because it knows you don't find <b>Chuck Norris</b>, he finds you"));
  9.     }

headless frontend testing with hudson and xvfb

hudson settings
You will need Firefox and xvfb (virtual framebuffer) on your CI machine. There are good tutorials out there, so I won't repeat this part. You might want to read: integrating selenium and hudson. Basically we'll just install xvfb and firefox on our headless linux machine, no further configuration necessary.

Then we'll set up a new hudson job. Take the free-style software project as we will run a couple shell commands. There are already xvfb plugins for Hudson and Jenkins, but I didn't have the time and privileges to test them. So this might come later. If you have some time, try them, they might solve some problems and make the setup easier. Especially starting framebuffers on different ports and storing away the reports for every run. At the moment you'll find the output for the latest build in the workspace.
But this part now is quick and dirty version until I get to look for a better setup.

Well use the parameterized build to set the location of our framebuffer. The name is DISPLAY, value is :99.0 (see image to the right). In the build itself we'll first run a shell command to start the framebuffer:

  1. Xvfb :99 -ac -screen 0 1024x768x16 &

after that we run the maven job:
  1. maven clean test

This will run our frontend test. The CI machine will start Firefox with webdriver and run everything in the framebuffer. No need to have a running X instance or a logged in user on your CI machine.

After the job is done the frambuffer will terminate.

And that's all you need to get roundhouse kicked by Chuck fully automated.

conclusion

There might be better setups, especially with appropriate hudson/jenkins plugins. This was my first step and it already helped to find errors after we changed something in the backend or the forms themselves. You don't have to manually test the frontend every time. Let the CI server do it after a commit or periodically. The overhead for firefox and the framebuffer is fairly small.

With every change in the code, you have to check the appropriate tests. If you're running unit tests, you have to adapt them after a code change that breaks them or the other way around if you're doing test driven development. It's the same with the selenium tests. So there is a little extra work once you change the behavior. So I recommend writing the tests after your forms are in the final version or implementing them in the beginning in a test first approach and let them break till you're done with the frontend. This of course voids the use of Selenium IDE.

Alltogether I'm glad we started frontend testing. Especially for a couple of cases, where integration tests would have required a lot of mocking or test system. Also we can know that our live system is working properly.

Comments

Anonymous's picture
Jenkins , selenium and linux issue in launching firefox
Anonymous (not verified) - Wed, 04/18/2012 - 13:44

i am setting up to the jenkins (hudson) on linux environment .to run my selenium script .

So i configure a job in jenkins from which i am fetching code from svn and then i am starting selenium server and then invoking ant to run my scripts. but when i am running this job my selenium server get started inline but firefox is not getting launched.
So i checked on the google they were saying to use Xvfb. I have installed Xvfb on Linux machine and download a xvfb plugin in jenkins
And set up its configuration on Jenkins still my firefox is not getting launched.

could you please help me how we use to configure Xvfb in jenkins and How this firefox launch problem can be resolved