Все сразу и в одном проекте

Прекрасная статья с http://binil.wordpress.com

Automated Smoke Tests With Selenium, Cargo, TestNG and Maven

In the project I am working on right now, we have decent amount of unit tests. Some of the unit tests launch an in-memory HSQL DB and run the tests against them – this helps us catch bugs in schema, Hibernate mapping etc. We use a Continuous Integration server which checks out code from the repository every hour and runs the tests. This setup helps us catch some kind of bugs early, but some other errors still can happen – for example, errors in web.xml, javascript errors etc.
Also, we frequently deploy newer releases to QA. Before making a release, we do a “smoke test” where one of us runs through the major functional areas of the application. Sometimes we encounter bugs at this stage. The development web server is Jetty, but all deployments from QA upwards happen on Tomcat – minor incompatibilities between Jetty and Tomcat are identified very late, usually when the smoke test is run right after a deployment.
I devised a mechanism to run an automated smoke test, and I will describe the process here.
The tools we use for this are:

1. Maven: The build tool we use
2. Selenium Remote Control: A package which lets one programatically control a browser, mimicking the actions of a real user
3. Cargo: A package which lets one programatically control a server
4. TestNG: A test framework

All the above tools are available free and open source. This is not an exhaustive tutorial on any of these tools. All I intend to do is show how these can be used together to write a smoke test process. I have broken down the integration of these tools into eight steps. This tgz bundle contains the files needed for each step – kindly rename the file to steps.tar.gz (without the .pdf extension) and uncompress it to your local disk.
Note:I do not discuss creating the database schema for the test.
Step 0: Application under test
The web application we are going to test is a very simple one. On the first page there is a link, and when a user clicks on it she is taken to a form. The form is a simple Celsius-Fahrenheit converter; on submitting the form the user is shown the result of the conversion, and she is shown the form again so that she can try another conversion. There is a link at the bottom of the form which takes the user back to the index page. The application is developed using WebWork.
To run this app, get the file, uncompress it to your local disk, run:

steps/step0/mywebapp> mvn jetty:run

and open http://localhost:8080/mywebapp/index.html on your browser.
Note: If you have not worked on Maven before, setting it up is real easy. Download the package from the project website, unzip it somewhere on your local disk, add a system variable MAVEN_HOME pointing to where you have installed Maven, and add $MAVEN_HOME/bin to the PATH. You can verify the installation by running mvn -v, which is expected to print the Maven version on the console.
Step 1: Configure an integration test suite
We create a new module, functest, to hold our functional tests. We configure for the following in the functest/pom.xml:

1. test sources are within src/it
2. tests are to be compiled with JDK5
3. tests are to be run during the integration-test phase of the build

Also note that the packaging is set to pom.

<project ..>
  ..
  <packaging>pom</packaging>
  ..
  <build>
    <testSourceDirectory>src/it</testSourceDirectory>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <source>1.5</source>
          <target>1.5</target>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <executions>
          <execution>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

To verify that our setup works, run:

steps/step1/functest> mvn integration-test
[INFO] Scanning for projects...
[INFO] ----------------------------------------------------------------------------
[INFO] Building functional-tests
[INFO]    task-segment: [integration-test]
[INFO] ----------------------------------------------------------------------------
[INFO] [compiler:testCompile {execution: default}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [compiler:testCompile {execution: default}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [site:attach-descriptor]
[INFO] [surefire:test {execution: default}]
[INFO] No tests to run.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9 seconds
[INFO] Finished at: Thu Dec 07 22:49:54 IST 2006
[INFO] Final Memory: 4M/10M
[INFO] ------------------------------------------------------------------------

Step 2: Configure TestNG
Next let us configure TestNG to work well with Maven. We change the functest pom.xml to:

1. add TestNG as a test-scoped dependency
2. specify the 2.8-SNAPSHOT version of surefire plugin, because that is the one that works well with TestNG
3. add TestNG-specific configuration for surefire plugin

<project ..>
  ..
  <dependencies>

    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>5.1</version>
      <scope>test</scope>
      <classifier>jdk15</classifier>
    </dependency>
  </dependencies>

  <build>
    <testSourceDirectory>src/it</testSourceDirectory>
    <plugins>
    ..
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        ..
        <configuration>
          <suiteXmlFiles>
            <suiteXmlFile>src/it/testng.xml</suiteXmlFile>
          </suiteXmlFiles>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

We also add a test, com.mycompany.app.functest.FirstTest, which is a simple TestNG test. Notice:

1. the use of Java5 static imports
2. the @Test annotation that marks a method as a test

package com.mycompany.app.functest;

import static org.testng.Assert.*;
import org.testng.annotations.Test;

public class FirstTest {

  @Test
  public void shouldPass() {
    assertTrue(true);
  }
}

Also notice the functest/src/it/testng.xml file which groups all tests in com.mycompany.app.functest package as functionalTests.

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="seleniumTests">
  <test name="functionalTests">
    <packages>
      <package name="com.mycompany.app.functest" />
    </packages>
  </test>
</suite>

Let us verify that our setup is good; if everything is configured properly you should notice that your simple TestNG test has passed.

steps/step2/functest> mvn integration-test
..
Running functionalTests
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.137 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
..

Step 3: Configure Selenium RC
Next let us configure our tests to use Selenium RC. We add the Selenium RC client driver as a dependency in the functest/pom.xml.

<project ..>
  ..
  <dependencies>
    <dependency>
      <groupId>org.openqa.selenium.client-drivers</groupId>
      <artifactId>selenium-java-client-driver</artifactId>
      <version>0.9.0</version>
      <scope>test</scope>
    </dependency>
  ..
  </dependencies>
  ..
</project>

The Selenium RC client driver is a package that provides an abstraction for a browser window. We change our earlier test to launch a firefox browser window using Selenium RC.

@Test
public void shouldPass() {
  Selenium selenium = new DefaultSelenium("localhost", 4444, "*firefox", "http://localhost:8080/");
  selenium.start(); // launch a brower window
  selenium.stop(); // close the browser window
}

To run this test, we need to first start the Selenium RC server. The Selenium RC client driver communicates with the server; the server performs the actual task of launching of the browser and controlling it. Download Selenium RC server 0.9.0 version from OpenQA repository and launch the jar:

> java -jar selenium-server-0.9.0.jar
application/xhtml+xml
Dec 7, 2006 9:56:01 PM org.mortbay.http.HttpServer doStart
INFO: Version Jetty/0.9.0
Dec 7, 2006 9:56:01 PM org.mortbay.util.Container start
INFO: Started HttpContext[/,/]
Dec 7, 2006 9:56:01 PM org.mortbay.util.Container start
INFO: Started HttpContext[/selenium-server,/selenium-server]
Dec 7, 2006 9:56:01 PM org.mortbay.util.Container start
INFO: Started HttpContext[/selenium-server/driver,/selenium-server/driver]
Dec 7, 2006 9:56:01 PM org.mortbay.http.SocketListener start
INFO: Started SocketListener on 0.0.0.0:4444
Dec 7, 2006 9:56:01 PM org.mortbay.util.Container start
INFO: Started org.mortbay.jetty.Server@6672d6

This will launch the Selenium RC server on port 4444.
Note: Make sure that firefox executable is on the PATH of the command window from where Selenium RC server is launched.
Now, in another command window, run:

steps/step3/functest> mvn integration-test

If everything is set up fine, you would be seeing a Firefox browser open and close.
Step 4: Test for the title of the index page
So far we got the browser to launch, but we have not tested if the application is working fine. Now let us write a test which launches a browser, opens the index page of the application, and verifies the title of the index page.

selenium.start();
selenium.open("http://localhost:8080/mywebapp/index.html");
selenium.waitForPageToLoad("5000");
assertEquals(selenium.getTitle(), "My WebWork Application");

Notice the use of Selenium RC client driver API to direct the browser to a particular URL, wait for the page to load, and geting the browser title string.
To run this test we need to start the application. On a new command window, run:

steps/step4/mywebapp> mvn jetty:run

In another command window, run:

steps/step4/functest> mvn integration-test

If everything is fine, you should see a browser open, load the index page, and close. You should also notice that the test passes. To test that this assertion really works, change the assertion, run the test again, and verify that the test fails.
Note: Make sure that the Selenium RC server from step 3 is running.
Step 5: Adding one more test
Now we add another test to click on the link on the index page of the app, and verify that the page with the form comes up. We also refactored the opening and closing of the browser into seperate methods. We annotate these methods – openBrowser & closeBrowser – such that they are run before and after the tests respectively.

@BeforeTest
public void openBrowser() {
  selenium = new DefaultSelenium("localhost", 4444, "*firefox", "http://localhost:8080/");
  selenium.start();
}

@AfterTest
public void closeBrowser() {
  selenium.stop();
}

@Test
public void indexPageShouldComeUp() {
  ..
}

@Test
public void converterPageShouldComeUp() {
  ..
}

Verify that our refactoring and new test works by running:

steps/step5/functest> mvn integration-test

Note: Make sure you have the Selenium RC server and Jetty server running still.
Step 6: Starting and stopping Selenium RC server from the test
So far we have had Selenium RC server running in a command window. First step towards true automation start and stop the server from within the test. Firstly, stop the Selenium RC server running. We need to start the Selenium RC server before any tests are run, and we need to shutdown the Selenium RC server after all tests are run. TestNG’s @BeforeSuite and @AfterSuite annotations enable us to do so.

@BeforeSuite
public void startSeleniumServer() throws Exception {
  seleniumServer = new SeleniumServer(4444);
  seleniumServer.start();
}

@AfterSuite
public void stopSeleniumServer() throws Exception {
  seleniumServer.stop();
}

We need to add Selenium RC server as a test-scoped dependency to our functest project.

<project ..>
  ..
  <dependency>
    <groupId>org.openqa.selenium.server</groupId>
    <artifactId>selenium-server</artifactId>
    <version>0.9.0</version>
  </dependency>
  ..
</project>

Run:

steps/step6/functest> mvn integration-test

and verify that this works.
Note: Make sure that you have shutdown the Selenium RC server started earlier.
Step 7: Controlling a Tomcat server from the test
Our tests still need a Tomcat server running. It would be really useful if we could start the Tomcat server, deploy our application onto the server, run the tests and eventually shutdown server. We use Cargo to let us do this – Cargo has a maven plugin which can be configured in functest pom to achieve this. The plugin download Tomcat binary from the web, installs it, deploys our web application onto it. After the tests are run, the plugin stops the Tomcat instance too.

<project ..>
  ..
  <dependencies>

    <dependency>
      <groupId>${project.groupId}</groupId>
      <artifactId>mywebapp</artifactId>
      <version>${project.version}</version>
      <type>war</type>
      <scope>provided</scope>
    </dependency>

    <dependency>
      <groupId>org.openqa.selenium.server</groupId>
      <artifactId>selenium-server</artifactId>
      <version>0.9.0</version>
    </dependency>
    ..
  </dependencies>

  <build>
  ..
    <plugins>
    ..
      <plugin>
        <groupId>org.codehaus.cargo</groupId>
        <artifactId>cargo-maven2-plugin</artifactId>
        <version>0.3-SNAPSHOT</version>
        <configuration>
          <wait>false</wait>
          <container>
            <containerId>tomcat5x</containerId>
            <zipUrlInstaller>
              <url>

http://www.apache.org/dist/tomcat/tomcat-5/v5.5.16/bin/apache-tomcat-5.5.16.zip

              </url>
              <installDir>${installDir}</installDir>
            </zipUrlInstaller>
            <output> 
              ${project.build.directory}/tomcat5x.log
            </output>
            <log>${project.build.directory}/cargo.log</log>
          </container>
          <configuration>
            <home>
              ${project.build.directory}/tomcat5x/container
            </home>
            <properties>
              <cargo.logging>high</cargo.logging>
              <cargo.servlet.port>8080</cargo.servlet.port>
            </properties>
          </configuration>
        </configuration>
        <executions>
          <execution>
            <id>start-container</id> 
            <phase>pre-integration-test</phase>
            <goals>
              <goal>start</goal>
              <goal>deploy</goal>
            </goals>
            <configuration>
              <deployer>
                <deployables>
                  <deployable>
                    <groupId>${project.groupId}</groupId>
                    <artifactId>mywebapp</artifactId>
                    <type>war</type>
                    <pingURL>http://localhost:8080/mywebapp/index.html</pingURL>
                    <pingTimeout>300000</pingTimeout>
                    <properties>
                      <context>mywebapp</context>
                    </properties>
                  </deployable>
                </deployables>
              </deployer>
            </configuration>
          </execution>
          <execution>
            <id>stop-container</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>stop</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Notice that we added our web application as a dependency to functest project.
Close the Tomcat server that was started earlier, and run the tests once again:

steps/step7/functest> mvn integration-test

If we have done things alright, tests should pass sucessfully.
Step 8: Running tests on multiple browsers
Right now our tests only run in Firefox. If we want to run it another browser, we need to change the third parameter to constructor of DefaultSelenium. TestNG allows parameters to be passed to @BeforeTest method, so we make use of that.

@BeforeTest
@Parameters({ "browser" })
public void openBrowser(String browser) {
  selenium = new DefaultSelenium("localhost", 4444, "*"  + browser, "http://localhost:8080/");
  selenium.start();
}

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="seleniumTests">
  <test name="IEFunctionalTests">
    <parameter name="browser" value="iexplore"/>
    <packages>
      <package name="com.mycompany.app.functest" />
    </packages>
  </test>
  <test name="FireFoxFunctionalTests">
    <parameter name="browser" value="firefox"/>
    <packages> 
      <package name="com.mycompany.app.functest" />
    </packages>
  </test>
</suite>

We add one more test, which checks if the basic convertion works

@Test
public void zeroCelsiusShouldBeEqualTo32Fahrenheit() {
  assertEquals(convertToFahrenheit(0.0), 32.0);
}       

private double convertToFahrenheit(double f) {
  selenium.open("http://localhost:8080/mywebapp/index.html");
  selenium.waitForPageToLoad(TIMEOUT);
  selenium.click("id=converter");
  selenium.waitForPageToLoad(TIMEOUT);
  selenium.type("//input[1]", Double.toString(f));
  selenium.select("id=temperature_convertTo", "label=fahrenheit");
  selenium.click("id=temperature_0");
  selenium.waitForPageToLoad(TIMEOUT);

  String result = selenium.getText("//p[1]");
  String expectation = f + "C converts to ";
  assertTrue(result.startsWith(expectation));

  result = result.substring(expectation.length());
  result = result.substring(0, result.length() - 1);
  return Double.valueOf(result);
}

Notice that we abstracted the form submision into a private method – this enables us to use the code that performs the actual form submit in many tests.
Thats it! We can use this setup in a Continuous Integration server. It will download Tomcat from the web (only once!), install it, deploy our web application onto it, start Selenium RC server, open a browser window, test drive the application on the browser, close the browser window, stops Selenium RC server and eventually stops Tomcat server.