Submitted by MichaelJames on March 7, 2007 - 11:40pm.
Non-Java tools are often used for functional (black box) regression testing J2EE web applications. Such tools include Mercury WinRunner, SilkTest, (IBM) Rational Robot, and others. The lure of these tools is the "capture/playback" capabilities (making it easy to create tests without writing code), and scripting capabilities.
This seems like a positive step away from tedious, error-prone, and infrequently done manual regression testing. But you may be able to do better.
I label these tests semi-automated (or semi-manual if you're a "half-empty" type) because of the way I see them applied in practice: I see them initiated manually, followed by interpretation of the results by the team's "QA person."
These tools have some disadvantages:
I've yet to see them plugged them into a Continuous Integration server such as CruiseControl or Continuum. It is reckless and wasteful not to run regression tests continuously (think hourly) while developers are changing code, especially across multiple teams. The Continuous Integration server needs a way to create a known initial state (i.e. build, deploy, and reset the database), initiate the tests, roll up all their results (along with results of unit tests) into one binary pass/fail assessment, and identify which test has failed when that occurs. These things are relatively easy to do with JUnit-based tests. It should also be technically possible to do this with WinRunner/SilkTest/etc. but I haven't seen anyone do it yet. If you've gotten this to work, please write to me or post a response here.
License restrictions impede the reproducibility of the test results. In XP teams everyone has access to their own "push to test" button, and pushes it a lot.
License restrictions and proprietary scripting languages re-enforce the unagile idea that one person on the team is responsible for testing. It's hard enough to break this habit without artificial restrictions from the tools.
Overuse of capture/playback can make the tests break easily when user interfaces change. This can be mitigated by using the proprietary scripting languages.
JUnit tests are easy to plug into Continuous Integration, aren't bound by license restrictions, and are written in Java, one of the best known programming languages (especially on teams that are already developing Java applications!). Most traditional testers have no background in Java, thus would need to pair with Java programmers to create tests.
The JUnit framework was originally intended (and still primarily used) for "unit testing" -- that is, white box testing of individual components outside the context of the whole system. That's great and all, but creating deterministic (very thoroughly tested) control systems for aircraft and spacecraft has taught me that systems tend to fail more than individual components. We need end-to-end (system) tests even more than we need "unit" tests. This is one area I differ (slightly) with the mainstream XP crowd.
The good news is that a great selection of third-party libraries exist for system testing with JUnit. They're all free and/or open source.
JUnit alone is sufficient for EJB testing. As with the others, your build script will need to deploy and start the server. And you'll need some way to reset the database to a known state. Apache Cactus may be useful in some cases.
Watij will drive a real Internet Explorer browser in the JUnit framework (through Java code). This is pretty cool because so many bugs are related to browser behavior, especially JavaScript. Watij is still kind of new and lacked extensive support last time I used this. (We've also tried a Ruby-based tool if you're into that kind of thing.)
Selenium will drive a wide array of browsers for MS-Windows, Linux, and Mac OS X. This, combined with Selenium Remote Control is the current favorite at Danube.
actiWATE will emulate a web browser, including JavaScript (AJAX, etc.), in the JUnit framework. This might be useful when the tests need to run headless. actiWATE is free but not open source like the others. I found actiWATE easy to use.
Abbot is for testing Java Swing and AWT GUIs, such as desktop applications, Java WebStart applications, or Applets. The ScrumWorks desktop client is extensively tested with Abbot on MS-Windows and Mac OS X.
When someone at Danube checks in a code change, the screens of a Mac and a PC in the back room (each running CruiseControl) wake up and start GUI tests doing many of the same things you do as a user, except faster and much more abusively. And yeah, we still need to do a lot of manual testing.
Those are the third-party tools/libraries we can vouch for from personal experience. If you've had good experiences with others (or bad experiences with any of these), please let me know!
--mj
Michael James
Software Process Mentor
Danube Technologies, Inc.
i am a qa person and i am totally sold on junit and the benefits of continuous integration. however, you omit a key benefit of tools like silktest, winruuner, etc., which is that they support data-driven tests. this for qa is imperative. we need to test many permutations of an application event, not simply a few positive ones. for example, when working on an ecommerce site and testing a search engine, i want to script the invokation of the search function just once and make the input string a parameter, where that parameter is fed by an external data file that can contain a large number of test strings. we need to test way more than one or a few positive test cases, and data-driven testing allows us to do this. you have mentioned to me, michael, that we should look into FIT for this purpose, and i will, i just want to call out that for qa people this is a critical piece of the puzzle.