Canoo WebTest vs. Selenium
From a message I sent earlier to the AppFuse user mailing list:
After getting Dojo's DropdownDatePicker working with both Struts and Spring MVC, I've run into a snag:
You can no longer use Canoo WebTest to enter dates in a form.
It doesn't find the form element because HtmlUnit (the underlying browser implementation for Canoo) doesn't understand/render Dojo's components.
For users, workarounds include:
1. Don't write tests for entering data on forms that have required dates.
2. Don't make dates required.
Either way, it seems logical to move to Selenium for UI testing.
What do you think? Are Dojo-based components the future of Ajax support in Java web frameworks? If so, I wish Dojo allowed more "progressive enhancement" features instead of the invalid HTML feature it seems to rely on now. I'm more than willing to support Dojo if that's what AppFuse users want. However, in my experience
with Struts 2's datetimepicker, it seems pretty slow. Of course, if I could render my own input element and then add a datepicker next to it, I'd be much happier. AFAICT, the Struts' datetimepicker doesn't allow you to do this, but I could be wrong.
Of course, if Dojo 0.9 lives up to the hype and is much faster - maybe I should quit caring about Hijax and just start writing those Selenium tests?
For users of those "next-gen" Java frameworks (like Seam, Grails and Wicket), what do you use for UI testing? Does Canoo WebTest work for you when you have a lot of Ajax functionality? If not, do you use Selenium or leave UI testing up to the QA folks?
Update: I ended up using jscalendar for both Spring MVC and Struts 2's datepickers. It's simply the best tool for the job. Dojo is too bloated IMO (240K for a datepicker!) for such a small feature. This allows us to worry about getting AppFuse 2.0 out the door without migrating to Selenium first. We will eventually migrate to Selenium (it's a better tool for the job IMO), but probably not until 2.1. The only question is what's the best way to distribute Selenium tests?
And http://developer.yahoo.com/yui/examples/calendar/quickstart/1.html
Posted by Gilberto on May 31, 2007 at 01:00 PM MDT #
Gilberto
Posted by Gilberto on May 31, 2007 at 01:04 PM MDT #
jQuery, Yahoo's YUI and MooTools are all nice libraries. However, since Tapestry and Struts 2 bake in Dojo by default, it makes it a little difficult to not include it. Maybe we should just make it easier to include a different Ajax framework than Scriptaculous? Right now, we only use it for highlighting and setting form focus - maybe we should move those into global.js (which I've thought of renaming, suggestions welcome) and make them independent of any framework. Or we could continue our abstractions on top extractions and use Ext JS or jMaki - I believe both libraries allow you to use other libraries underneath.
The good news is we don't plan on Ajaxifying pages by default until AppFuse 2.2, so we have some time to make these decisions. For those of you who have experience with any of these libraries, now is the time to speak up. Personally, I still like Scriptaculous and Prototype, but I haven't worked as much with the others.
Posted by Matt Raible on May 31, 2007 at 06:23 PM MDT #
Posted by musachy on May 31, 2007 at 07:25 PM MDT #
Posted by Matt Raible on May 31, 2007 at 08:00 PM MDT #
Posted by Musachy Barroso on June 01, 2007 at 12:46 AM MDT #
For Nuxeo 5, which is based, on the front-end, on Seam, JSF and Ajax4Jsf, I had very limited success with Canoo webtest (I only could test 1 or 2 screens until I got untractable JavaScript errors).
We switched afterwards to Selenium, and it just works! Another benefit is that our tests don’t need to be written by the software engineers (using Selenium IDE), but still can be run using automated tools (TestNG and Hudson).
Stefane Fermigier, CEO, Nuxeo
Posted by Stefane Fermigier on June 01, 2007 at 07:06 AM MDT #
More generally concerning WebTest vs Selenium: Selenium is a nice tool (and it's weaknesses are not the ones that are often presented) but in test automation you have "automation" which means that it is a software engineering activity. If recorded tests are what you want, then Selenium is probably fine but you have to know that it's the less cost effective way to perform test automation (even if it makes sense for some very special cases). If you want to perform what I call "professional test automation", then using WebTest is more effective (ok, I should write a special post to develop that but you can have a look at WebTest new "Key features" doc).
[for info: as I'm HtmlUnit and WebTest committer, I'm surely a bit partial]
Posted by Marc Guillemot on June 01, 2007 at 10:00 AM MDT #
I think that this discussion is more about the testing tool than the problem with the date picker. It identifies something that is very reveling about the capability of the two tools. Selenium is the "real deal." You are writing web applications for the browser and Selenium exercises that client. In fact, it can exercise different implementations of that client. You cannot beat that, no matter how good your emulation layer may be, just as Wine will never be better than VMware.
That isn't to say that WebTest isn't useful, it just cannot be the final line of defense. The QA team could always use Selenium (or something comparable, including manual testing). Either way, it has to be done or else your client becomes part of your testing strategy. Selenium may not be perfect, but that project is going in the right direction.
Posted by Dan Allen on June 01, 2007 at 12:54 PM MDT #
Posted by Renat Zubairov on June 01, 2007 at 07:27 PM MDT #
Posted by Renat Zubairov on June 01, 2007 at 07:34 PM MDT #
Different tools for different tasks. As Dan says above, you can't get better than the real deal. Unless you don't have the real deal available. Or the Selenium APIs or reporting tools get in your way. Or you want to minimize resource consumption. Selenium has its place, as do WebTest and HtmlUnit.
I should mention that HtmlUnit (the engine behind WebTest) has integrated the unit tests from the Prototype and jQuery projects into its own unit tests, though they don't all pass right now.
Posted by Daniel Gredler on June 03, 2007 at 05:26 AM MDT #
Sounds silly, isn't it? In fact you would have exactly the same with Selenium!
Automated tests are here to be run as often as possible and to give the best feedback to find errors as early as possible (when they are cheap to fix) to improve the application's quality. Under this aspect, a simulated browser can catch more errors and therefore be a greater help.
I don't know what the "direction" of Selenium is. I can imagine that for instance Selenium IDE could catch js errors. I don't know for the hta but this can surely not be done with Selenium Core. Js errors are only one example, other kinds of errors should be detected as soon as possible, even if a "normal" browser hide them.
In fact the discussion here is a bit different and depends not only of the quality of the tools but of the kind of automated testers the AppFuse users are. For instance Selenium is more "visual" and some people better understand what they "see". On the other side, WebTest is for instance easier to setup and tests are cheaper to maintain.
Posted by Marc Guillemot on June 05, 2007 at 08:21 AM MDT #
Selenium and Canoo WebTest are representatives of two distinct classes of test tools. Test tools consist of a simulation and a verification part. We're currently only discussing the first aspect (the second would also be worth discussing).
Type A tools like WebTest use an internal abstraction of a browser and work against an expectation (if not even specification) of protocols. Type B tools like Selenium do the simulation part by remote-controlling a browser.
Some claim that only B is the "real deal" but be reminded that you depend on all the accidental behaviour that your chosen browser happens to show. You are doomed to fall for the same traps that manual testers do: assuming the webapp is correct because the errors are hidden. That's only as good as "programming by coincidence". Unfortunately, that is the "real deal" in many projects.
I for myself would never feel safe behind such a weak "defense line".
Posted by Dierk on June 05, 2007 at 12:20 PM MDT #
For testing, definitely go with Selenium.
>If not, do you use Selenium or leave UI testing up to the QA folks? QA is time consuming. Keep selenium tests for widgets in the repository also helps other developers can work on the code.
QA drawbacks include having invalid and duplicate bugs that just take time.
Selenium is useful for testing widgets (like a date picker) and pages. You might also want to check out Selenium-RC to write all your tests in Java.
The Dojo framework depends on eval, which can make debugging very hard. The date picker is slow and heavy and has more features than you need. Stick with the DynArch one.
Posted by Garrett on June 22, 2007 at 08:29 PM MDT #
2 words regarding WebTest vs Selenium: "Use Both!"
1) Selenium (and all browser-drivers) gives you "last mile" coverage of the actual customer/client usage in a real browser. But there's still work to be done. My current project at Google is to work on all those annoying things like unexpected modal dialog windows that currently stop Selenium in its tracks without browser-specific workarounds. The downside to browser-driving is that it's slow... unless you have a grid of servers from every platform (Mac, Linux, Windows) where you can run your tests in parallel on. :-) (Work is being done at that, too!) In the meantime, one could get that massive grid now if they start playing around with Amazon's Elastic Compute Cloud (EC2) (at least for Linux/Firefox testing).
2) WebTest (and all protocol-drivers) gives you speed and simplicity. You can do more testing of different kinds of scenarios faster with a protocol driver than you can with Selenium. Also, a protocol driver will make it easier to test more "failure modes" like simulating what happens if the network connection drops mid-request. Think of it like designing an airplane. Before a manufacture sells a new airplane on the market, they use simulators *AND* real-world flight tests. It's a BOTH/AND decision not EITHER/OR when it comes to product testing.
I'm eagely following the WebDriver project (http://code.google.com/p/webdriver/). It provides 1 common API that lets you toggle between using HtmlUnit for browser emulation and driving actual browsers (via COM for IE, a custom plug-in for Firefox).
While I'm waiting for WebDriver to stabilize... in the meantime... write many unit tests, fewer Canoo WebTests, and fewer still Selenium tests. But use all of them. The more ways you get coverage the better off you'll be.
Cheers,
Hugs
Posted by Jason Huggins on August 02, 2007 at 05:37 PM MDT #
Hello,
Using Canoo Web Tool it Only Test Java Coding 's Or All Domain
Thanks,
-Rajkumar
Posted by Rajkumar on January 31, 2008 at 09:57 AM MST #
Posted by Arin on July 04, 2009 at 06:05 AM MDT #
Posted by Matt Raible on July 09, 2009 at 03:00 PM MDT #