Web Test Framework API

I've looked at different kinds of testing tools (API) to use them as runners for CubicTest. The various tools have their own API that provides different kinds of features. They also access the browser in different ways; either through JavaScript like Selenium, taking over the browser like WebDriver and Watir or being the browser it self like JWebUnit which relays on HtmlUnit to run.

I think that there should be a basic set of features that a web test tool should have. There might even be a common set of test to pass in order to support the basics. An example could be the JWebUnit-Commons-Test that ensures that every API that JWebUnit uses meats their requirements. It is a natural thing for test tool.

There are two parts of testing web applications in addition to setup (connecting to the website) and teardown. First is the assertion or lookup for elements on the page. The other part is performing some sort of action on the elements. This is pretty simple. Ones should be able to support most kind events. For example an onclick event should be fired with a click action. Some events should not be supported. I guess events that are not implemented on most modern browser. Also events that don’t represent user actions like onload, should be omitted.

I personally think that the system should be object oriented (at least it makes sense to me as a Java and JavaScript programmer). This means that in order to perform actions on objects one should look the up first. This also applies to asserting elements.

Ways of looking up elements are:

• By id or name – this is most common. It is supported by pretty much all modern browsers. However it isn’t very handy for element that doesn’t have an id and for element where its id may changes rapidly due to the web framework in use. Example code (Java):

  Element element = document.get(ELEMENT.id("myId"));
  assertThat(element, notNull());
  

• By other attributes present – there are different ways to do this. Selenium got support for CSS style lookup. A more common way of doing it is by using XPath. I personally prefer XPath to CSS, but for most people it’s a mater of taste. I think that just being able to look for more than one attribute is as powerful. Example code (Java):

  Element element = document.get(ELEMENT.id("myId")
       .class("myCSSClass"));
  assertThat(element, notNull());
  

• By parts of attribute – One should be able to look up elements that have an attribute that is equal to, begins with, ends with or contains a certain value. Even logic like and/or might be useful. Example code (Java):

  Element element = document.get( ELEMENT.id("myId")
       .class(contains("myCSSClass").
             or(contains("otherCSSClass"))));
  assertThat(element, notNull());
  

• By probability – Another way to ensure that stuff doesn’t break is to add probability. I guess that I would like to lookup elements that should have an attribute like one value and can have another attribute like another value. One should be able to find the element (or elements) that are the best fit. To push this even further we may use edit distance to handle misspelling. I guess that this might return the wrong element. So the returned element should know how well it fits the element looked for. An element that only fits 65% might be wrong. This raises the question of what attributes the element found actually got. The strength of this is the ability to adjust the lookup attributes to better fit the reality of the web application. I guess this improves robustness and enables tests that run with a “warning” to complete instead of just throwing an exception. Example code (Java):

  TableRow row = document.get(TR.id( SHOULD, equal("myId")
       .class(MUST, contain("myCSSClass")
             .or(contains("otherCSSClass"))));
  assertThat(row, notNull());
  if(row.getScore() < 100){
   println("Element might be wrong"); 
   println("id: " + row.getId());
   println("class: " + row.getClass());
   println("Contains button: " + row.get(BUTTON));
   println("Contains text: " + row.contains(
       text("mySuper product")));
  }
  assertThat(row.getScore(), greaterThan(65));
  

• By negative - One should not be constrained to one attribute or the only asking for positive. Should not is equally valid. Example code (Java):

  TableRow row = document.get(TR.id( SHOULD, equal("myId").class(MUST, 
          contain("myCSSClass"). or(contains("otherCSSClass")))
          .class(MUST_NOT, contain("alianCSSClass"))
          .contains(BUTTON, text("mySuper product")));
  assertThat(row, notNull());
  if(row.getScore() < 100){
   println("Element might be wrong"); 
   //... figure out what really happende
  }
  assertThat(row.getScore(), greaterThan(65));
  

• When testing web shop application’s product overview I like to be able to say that I’m looking for a table row (which is tend to be a lot of in a table) that contains a buy button (which typically is found in every row) and some text describing the product I’m looking for. This text is only found in the row that I’m looking for. This enables us to create less fragile tests. If I was to look for a row with a given attribute (like id) it would make my test fragile in quite a few web frameworks as it tends to change. I could have said that it should be the fifth row but that can also change quite rapidly. I guess the same “pattern” for testing can be used in different settings. It might be that one shouldn’t have to know what kind of container the button and the text is within. It probably should not be tied to know that it is table row. Example code (Java):

  TableRow row = document.get(TR.contains(BUTTON, 
       text("mySuper product")));
  assertThat(row, notNull());
  // to actually get the button to click:
  row.get(BUTTON).click();
  

I think that by adding these features it should be possible to create robust test. I also think that this kind of code is quite readable, at least for Java programmers that don’t know CSS or XPath very well.

I hope that some framework will add this functionality. We – the CubicTest developers – would sure love to have it.