In the past year and a half, I have had ample time to work on my Selenium wrapper and now things are working pretty well. Here are some things I have learned.

Selenium takes a long time to get working, largely because you have to solve a few problems:

• How to make the same test run against multiple browsers
  
• How to make the same test run against a development, testing, stage, and production environment
  
• How to make a test run in a slightly different way on a different build of the test without having to do a bunch of hardcoding
  
• How to make a test know what should appear on a page based on user type, state, version (so on).
  

Everyone has to solve these issue but, as of today, there is no solution that helps a person do this. If your company has an existing Selenium harness up an running, they have already solved this issue, but for those companies who don't (and thousands of new Internet companies are springing up as we speak) they all have to start from scratch.

Web pages are basically a collection of components. Selenium 2 started off in the right direction by building the PageObject pattern into their application. For a while, though, they left us hanging. Now they have a LoadableComponent built into their application, which looks pretty promising. They idea is that components can travel from page to page, so there is no reason not to reuse them. What you really want to do is define a bunch of components, configure which pages those components appear on under which circumstances, and then run your test and have it all just work

Lastly, websites have a few things in common. Generally
1. You can log in to them
2. You can join
3. You can search
4. You can purchase something

The steps for doing these are a fairly standard. It seems that you should be able to genericize these actions.

I am currently developing a yet-to-be-named, open-source project that will take care of all of this It is based on my work that I spoke about in the previous post. However it uses Selenium 2, TestNG and, soon, Guiceberry. I hope to have it done soon. I'll keep you all informed.

Selenium my way

I was going to call this "Selenium, the right way" but decided that was a little arrogant, even for me.

I finally have the chance to start a fresh acceptance/regression automated test harness again and I chose Selenium RC. Why? The answers are obvious. It's free, version 1.0 is stable, it supports AJAX, can be extended to support Flash, plays nicely with all kinds of languages including the one I care about (Java), supports all the major and not so major browsers, and has matured into a first-class product since I first started noodling with it back in 2006.

I wanted our implementation to start with all the best practices. What's amazing to me is that, well, there's no one place to find examples of all those practices. So as a result of my adventures, I've compiled a list of all my digging.

Here's the environment I'm working on

1. OS: Windows Vista (snort)
   
2. Main Language: Java
   
3. IDE: Eclipse
   
4. Unit Test Framework: JUnit 4
   
5. Ant
   

Requirements:

1. Need to test multiple domains and subdomains
   
2. Need to support AJAX
   
3. Need to support Flash
   
4. Need to support various user roles (anonymous, regular, admin)
   

It seemed to me like this would be a pretty common setup. Well, I couldn't find examples of how to do this stuff. I can't find the quote (I think it's Annie Lamott) that likens the Internet to an overstuffed, disorganized garage ("I know it's in here somewhere") but I was definitely feeling that way. So I put together this list, not just for myself but for any other souls who have a similar setup.

PageObjects
Most of the basic examples tell you to simply record your steps and output them to your language of choice using the IDE and then paste that snippet into your code. Sure, that's cool, but it's not very scalable or extensible. The out of the box snippets your IDE gives to you are cool, and they're a great place to start, but they are brittle. If you want a robust set of methods that will be like building blocks for other tests, you'll have to do a little more than just paste in what the IDE spits out. Here is a great paper on how to make your test code less brittle.

To deal with this, the Selenium people talk about using the PageObject model. But if you aren't using Selenium 2 (which is in alpha as of this writing) then you're out of luck getting an example from them. Having just spent the last year on the bleeding edge, I wasn't excited about depending on alpha software. Furthermore, the PageObject model has it's share of detractors, including Gojko Adzic (one of my heros).

Based on my reading, I took the approach outlined here and here, which is somewhat of a hybird model of PageObjects and the action groupings Adzic speaks of. Basically, I have a bunch of PageObjects. But I also have what I call a Macro object, which includes all the PageObjects and contains methods that assemble them together in often used configurations (like go to the homepage, click the sign-in link, and once on the sign in page, enter user and pass and press continue).

CSS Selectors
Buried in the Selenium documentation is a helpful tip - CSS Selectors perform better than XPath for locators. It's true, they burn. I found a few a sites to help with CSS syntax. Unfortunately, CSS is not as fully featured as XPath and for those really tricky locators; you can still use XPath, but favor CSS Selectors.

Ultimately, the best way to control this is to use the id attribute of your element, but the test department doesn't always have control over that. In may case, the front-end developer always tries to make my life easier, but is not always able to offer me my ideal solution based on technological constraints.

Suites and Tests
I really don't get why people don't have any Suite examples. It seems to me you want your suite to do a bunch of setup (like initializing the Selenium object) that is going to exist for the test and then have your test up. So I found an example of how to do this here.

The disadvantage of this is that you never run individual tests (because you don't have a Selenium object to start if it's in your suite). Instead, you invoke the test from the Suite level, so you always need a suite. And, if you're debugging tests, you'll orphan a bunch of controllers. Not a huge deal in my opinion.

I extended both SeleneseTestCase and TestSuite because there are things I need to do in them. Specifically, I wanted to make the Selenium object available to all my PageObjects without having to pass it in specifically, so I simply created a static reference to it in my AbstractTestCase and created the getter and setter. My test suite sets up my AjaxSelenium object and sets it in the AbstractTestCase and also initializes my properties using my properties file, which it gets from the Java ClassLoader.

Extending the DefaultSelenium Object
The Selenium doc says you don't have to do this. However, Selenium RC doesn't implement the "waitFor..." methods from the IDE that you absolutely need if your site, like most of the sites on the Internet now, uses AJAX. I found that the best way to support AJAX is to extend the DefaultSelenium Object and found a great example right here.

Note, this code throws the ElementNotFoundException. It apparently comes from HTMLUnit, which I'm not using. So I just created my own.

JUnit
The Selenium community clearly prefers TestNG, probably because it has matured faster. JUnit is just about caught up. I decided to convince my team to upgrade to JUnit 4.7 because I read somewhere that it has better support for parallel test runners than 4.6, the version that the concept was introduced into JUnit. Since I eventually want to run Grid, I thought it best to bite the bullet now.

The Elusive UI Map
The Selenium documentation refers to a way to reuse locators through the use of a UI Map. Most searches on UI Map and Selenium lead to the UI-Element extension. Some people really like this extension, but I am not a big JavaScript/JSON fan, so I find it annoying and frustrating to use. I decided not to incorporate UI Map because most of what it offers I'm getting from my use of the PageObject model. The way I decided to deal with locators was to simply hard-code them into the methods for the page objects. I felt this was easier than creating namespaces for all of them in my properties file. I think this also makes it easier for me to find problems. Instead of digging though a properties file to find a variable, I simply find the method that failed from the stack trace in my custom ElementNotFoundException and fix the locator.

Firefox and self-signed Certificates
I don't claim to understand anything about security. All I know was that this was a pain in the butt. Here were the instructions I followed to get around the issue.

Incorporating Selenium into your Development IDE
There are complete instructions for how to integrate your code into IDE so you can run your tests via the TestNG or, in my case, the JUnit runner. However, I had a difficult time finding out how to make it so that I could start the server from within the IDE. Finally, I found some helpful instructions. Then I just turned it into an Ant script, which includes the server bit of my workaround for the Firefox self-signed certificate issue. Note, if you use the Ant Script, you'll still need to create the cert.db file by following the directions above if you have self-signed certs.

Properties file
Standard practice. There are some basic properties I need to run my tests (like user/pass stuff). I put all this in a properties file and have the AbstractSuite load it up, as described here. In addition, I create base properties and subdomain-based override properties because the data values vary from machine to machine.

Constants
I created a Constants interface for the things I wanted to share and simple had my AbstractTestCase extend it.

Handle Download Dialogs
Supposedly, Selenium 2 will do this because it has WebDriver. As I said, I didn't want to deal with Selenium 2 yet. So this got me around the problem

LoggingSelenium
If you want any sort of decent reporting, you'll need to download LoggingSelenium and set this up. There's a nice example here. Note that if you use the AjaxSelenium above, AjaxSelenium will need to extend LoggingDefaultSelenium

Other Good links

* Ajax, Selenium and Fitnesse
* FlashSelenium - This was the best link I could find on this. We are still implementing so this may merit further blogging.
* Data Driven Testing with Selenium - I have yet to try this one but it looks promising.

Solving Problems with the Business Quality Analyst Role - Part III

With automation divvied out to the developers, prioritization and requirements clarification with the Business Owners, and project management in the hands of the scrum master, what's left for the BQA to do?

The way we built the role, the BQA does the following:

• Write stories and acceptance criteria – Forget the old documents (specifications and test cases). All that is needed is a story that contains the requirements, acceptance criteria, and any necessary technical/implementation details. The BQA can gather this info from Business Owners, Designers, and Architects and
  assemble initial proposals of what should be included in sprints/milestones/releases – Since the BQA has the most information about the stories, it makes sense for the BQA to drive this effort in collaboration with Business Owners and Tech Leads.
  
• Drive the demo – The BQA walks the team through the features because they are the most familiar with it, having written the requirements and tested it.
  
• Advocate for testability – Testability features can be a huge time saver not just for testers but for the team at large. While not every feature needs accompanying testability features, an experienced tester will recognize which features will benefit from being more testable. As a BQA, one can advocate for testability early, keeping in mind that where testability is concerned, good is usually good enough.
  
• Write Documentation – While Agile favors conversation over documentation, there is nothing saying that documentation should be eliminated altogether. In fact, for some people, the very act of writing something about a subject increases their ability to learn about it. I feel that the BQA, being the hub of information, is the best candidate to document features, particularly when it comes to documentation on how to test a feature.
  
• Act as an alarm system – Traditionally, QA holds off on saying anything about a problem until a defect is reproducible. On an Agile project, this strategy may result in information that comes too late. The team is relying on everyone to raise the alarm as soon as you notice a problem. Time is of the essence. Since the BQA is central to the process, it makes sense for them to raise alarms as soon as possible when they notice problems. Note, I don't mean that BQAs should always file defect reports when they notice problems (see my earlier posts on this).
  

Business Analyst Anti-patterns

• Working in vacuum – You've spent a few hours by yourself without talking to someone and you're feeling stuck. You probably need to collaborate.
  
• Too much detail – You've built a table of acceptance criteria that took you 2 plus hours. It's long and complicated and no one understands it. You probably need to simplify
  
• “They can take it out later” – This is waste. You can also put it in later. Don't include it and confirm this with the user at the demo. Save the story for later so that any work you have already put in can be reused.
  
• Not spending enough time at the beginning – In part two of this article, I talked about letting things marinate and making sure you thoroughly understand how a feature is supposed to work. Some analysis and story writing takes much longer than we anticipate. Collaborate with your team if your exploration into a feature is producing new information.
  
  

Problems we encountered with making testers into BQAs

• BQA was the bottleneck – At the beginning of the project, you need to write the stories before you write code. We simply didn't know how many people we were going to need to write the stories and we underestimated. We had two BQAs for about 15 developers. Eventually we got to the point where there was BQA for every 5 or so developers. This was still a lot of work but was manageable.
  
• Testers tend to be less outgoing about their work – Most people in a software organization don't care about how testers do their job, only that they report the results and that those results are accurate. As a consequence, testers tend to be less reliant on other teams outside of development for information and tend not seek help in other areas of the business. Business Analysts need to talk to everyone so if a tester who is becoming a BA is hesitant to seek information outside of the development group, they are not going to be effective.
  
• Testers naturally do things wrong – Given a minimal set of instructions, testers tend to get the instructions wrong. In part, this is what makes a good tester. But many testers also need to learn that there is a time to follow directions and a time to break the rules. Also, many testers could stand to ask for more help and clarification when the directions don't make sense to them. When testers are doing Business Analysis, they will need to follow directions and get clarification. It doesn't serve them to find bugs during the analysis phases of the project.
  

Solving Problems with the Business Quality Analyst Role - Part II

If I'm a tester, how can I become an Agile Business Analyst?

Some of the skills you'll need are those you may already have. Here's some of the concepts I learned as I was improving my skills as a Business Analyst and, at the same time, increasing the quality of my project.

• The “Funnel” – When writing stories for a feature, start big and get smaller. Don't start with details. In fact, don't even bring in details until you have a set of story titles that cover the feature.
  
• Be concise – If you can say it three sentences, see if you can say it in two. Think of your stories in terms of journalism. Be sparing with your words. Your audience, the business owners and developers, don't want to spend more than a few minutes with your story. Interestingly, you can also start applying this principle to defect reports.
  
• Know your feature – Particularly if you are rebuilding a feature that already exists, don't rely on other people's description. Prove to yourself that the requirements you have heard are accurate. Better still, if the feature exists, play with it before you talk to anyone and know what it does so that instead of asking how it works, you are confirming how it works. This makes you look good. If you don't have the technical ability to determine how something works, create a task to have someone do that for you. The situation you don't want to be in is to find out that something doesn't work the way someone told you it did and you didn't bother to confirm. That's a BA foul.
  
• Define the boundary in your story titles – Story titles separate distinct pieces of functionality from one another. It is important that your story titles clearly summarize the functionality and that they distinguish that functionality set from other functionality sets in the feature.
  
• Be clear – This is particularly important when dealing with offshore teams. Avoid vague language. Use bulleted lists to help you convey your point. You don't want someone offshore to get stuck because you used the wrong preposition, and believe me, it can happen.
  
• Collaborate – You are not alone, other people will help you. Find someone you can check in with often. Find people to check your stories for clarity, particularly if you are working with an offshore team. Have them check to make sure your story titles describe distinct functionality. Ask them to check for accuracy. It takes a village.
  
• Prepare – Don't play a story until it's ready because the team needs it. Wait until you feel it is ready. Sometimes, a story needs to marinate. This is not always easy because sometimes there is pressure to “feed the team” particularly if the coders have devoured the sprint backlog and are ready for new stories. I'd urge you to resist the temptation. After a while, your gut will tell if something is ready.
  
• Get visual – I like to use charts and diagrams to explain a piece of functionality. Visualization is also a great tool for learning about a feature and can help when you need to describe something. A picture is sometimes worth 1000 words.
  
• Be on the lookout for patterns – As you are building your story titles or writing stories, you will likely notice that some pieces of functionality follow the same rules as others. When you see this, it can help to take an Object-Oriented approach. Rather than transcribe the same rule into each story, have a set of Global Business Rules that you can refer to from within your story. That way, you can reference them when possible. And if those rules change, you will only have to change the rule and not the individual stories.
  
• Analyze, Verify, Recommend, Prioritize – This is the pattern for getting a feature ready for development.
  
  
  
  • Analyze – Find out how you think the feature works.
    
  • Verify – Talk to other people to confirm that feature works the way you think it does.
    
  • Recommend – Write the story and make recommendations about how to resolve problems and/or retain scope.
    
  • Prioritize – Work with the team to determine the order of the stories.
    
  
  

In the next part, I'll talk about the duties of the BQA and some BQA anti-patterns.

Solving Problems with the Business Quality Analyst Role - Part I

Several Agile Thought leaders, most notably David Anderson, have proposed combining the Business Analyst and Quality Assurance role. The idea harkens back even to Goldratt, who suggested moving Quality Assurance further up in the production line, to improve throughput. Gerry Weinberg has suggested that quality starts at the requirements gathering phase, where it is important to apply the scrutiny and attention to detail that a good tester brings to his/her work.

Combining the roles of Business Analyst and Quality Assurance makes theoretical sense and solves the ever lingering problem of what to do with testers in Agile projects who aren't coders. However, the implementation of such a role comes with its own special problems. At my last company, Tacit Knowledge, we found this out when we introduced the role on one of our projects.

Why would you do this?
Our reasoning was simple. We were having problems finding resources who were both experienced QA Leads who also knew how to code. We decided to stop looking. What we had on staff was QA Leads and Business Analysts. So we paired them.

We all know that the ideal Agile Product owner is indeed a rare bird. Another motivation behind this role was to be realistic about Agile Product Owners. Part of the BQA's role is to write the agile stories, a task typically left to the Product Owner. This relieves the Product Owner of that task and allows them to focus on requirements clarification and prioritization, which, in most organizations, is plenty of work for one person.

Who fits the role?
We found that the role required a combination of testing skills, business analyst skills and project management skills. People who are accustomed to wearing multiple hats and who were flexible in terms of what skills they needed to bring to the table at any given time were generally successful.

An accurate job description of our implementation of the role would entail the following:

• Analyzes information gathered from business sources and/or existing code base and collaborates with team to clarify ambiguous customer requests or answer questions
  
• Harvest requirements and translate into development-ready stories with testable acceptance criteria with the right amount of detail just in time
  
• Facilitates meetings with business, specifically requirements meetings and demos of working
  software
  
• Facilitates meetings with developers to introduce features and clarify requirements
  
• Advocates for testability
  
• Identify problems early in the process and collaborates effectively with team to determine resolution
  
• Takes large sets of functionality and breaks them out into smaller, logical and iterative pieces
  
• Prioritize requirements for sprints, releases and milestones in collaborative effort with business owners
  
• Provide requirements clarification for development
  
• Creates concise documentation (cheat sheets) to aid in validation
  
• Validates acceptance criteria
  
• Coordinate business acceptance of completed development
  
• Remind the project community to focus on the project goal
  
• Select and insert appropriate tools to assist in defining project goals, requirements, or communicating project progress.
  

What are the advantages of combining the two roles?
One of the major advantages to the role is that it reduces some of the waste that is inherent in the system. Testers tend to have to do a certain amount of Business Analysis for themselves, either because the acceptance criteria weren't clear enough to be tested or because the requirements changed by the time the feature got to the testing phase. By combining the roles, you invariably end up with testable acceptance criteria when the feature enters design or development. In addition, a tester can identify testability features that will pay off in the long term. This reduces the amount of waste produced in the development cycle. Finally, because the tester is already familiar with the feature, the testing takes less time.

What are the problems?
Unfortunately, not every tester or Business Analyst makes an effective BQA. The BQA role requires the right mix of flexibility and diligence. Generally, testers tend to be a little too diligent for the role and Business Analysts tend to be lax when it comes to the testing piece. Testers are reluctant to give up their diligence because the trait is highly prized in traditional testing departments. Most testers have seen other testers fail due to a lack of diligence. But what works in testing can prove to be a hindrance in Business Analysis where people often have to trade-off attention to detail to get a feature moving through the system.

In the next part, I'll talk about how can you make a business analyst out of a tester.

Towards an Adaptive Software Development Task Workflow Based on the Theory of Constraints – Epilogue

I had lunch with a former-colleague, Brian O'Rourke. He said he had read the blog and had missed how the workflow I described was tied to Kanban. I explained that, in part two, I briefly discussed how each phase had it's own Kanban system by virtue of having a bunch of tickets in the “Ready” state.

What Brian pointed out to me is that there is a simple way to adapt the proposed workflow so that it clearly represents a Kanban system, and you can do it on a whiteboard. It would look like this:

	Ready	In Progress
Analysis
Requirements
Design
Development
Testing
Acceptance

Once you do this, you simply move the story cards around on the whiteboard freely. This is an adaptation, of course, of the traditional "Agile wall" concept. It simply brings a different level of organization to the idea. Most agile teams organize their Agile Wall by feature. I tend to like to organize my feature work differently.But that's for a different post.

There are a couple advantages to having this system on a whiteboard.

1. These days, tools are either story oriented or defect oriented. There's very few which do both. I'm sure that will change soon, but right now, the business-side of the operation tends to track their work in a separate tool from development. The proposed table is a no-tech way of getting everyone to see everyone else's back log
   
2. There is no limit to the transitions – Since it's not in a tool, stories can simply by moved from backlog to backlog. Any rule enforcement would happen at the daily standup meeting (assuming you are having one and many people are, regardless of whether they are Agile or waterfall)
   

The disadvantage comes when you have a remote team, since they have no access to the whiteboard. I'm not going to go into whiteboard solutions here. You'll need to experiment with this. Suffice it to say that I don't believe tools should be a limitation, but rather they should enhance the work of the team. No one should spend a lot of time administering tools or keeping their data up to date.

Another disadvantage of this is that it probably would not work so well in a waterfall methodology, in that you would probably tend to get clumps of stories that would quickly become unmanageable (although, that would be a great illustration of the problems with the waterfall approach).

Assuming you don't have an offshore team and aren't using waterfall, I see no reason why Brian's nifty proposal wouldn't work like a charm.

Towards an Adaptive Software Development Task Workflow Based on the Theory of Constraints – Part III

In my last entry, I outlined the happy path of the workflow. Now it's time to talk about "retrograde" transitions.

A “retrograde” transition occurs when a task has to go backward in the workflow. In my implementation there are 7 such transitions but 3 are essentially the same (Clarify Requirements), having the same target with different sources.

Any retrograde transition whose target comes before the development phase in the happy path could generally be called a “Clarify” transition.

The transitions I've outline here are:

• Needs More Information – Occurs when the person writing the specification, typically the BA, requires more information from the product owner to complete the specification. This could be called “Clarify Analysis” but that seems unclear to me.
  
• Clarify Requirements – Occurs at several points in the workflow when anyone doesn't understand the specification. Requirements gatherers tend to be the hub for project information, hence the number of transitions back to the requirements gatherers is large.
  
• Clarify Design – Occurs when the technical or user interface design is unclear. I'll talk about disambiguating between technical and UI design later.
  
• Not Working to Specification – Simply put, this is a defect. I'm not going to go into detail about how to handle this situation and what to do when; That's for another entry.
  
• Needs Product Owner Analysis – Occurs when testers determine that a feature is so messed up that whoever wrote the specification couldn't possible have the right information.
  

I'm not sure this last transition is necessary but I've included it as an example. You could, in theory, link all "In Progress" states to all "Ready" states, just so that the team is empowered to make those transitions. However, that implies that every member of your team has all the information to make the correct determination for each transition. I don't like to force people down one path or through one group since that creates a bottleneck. On the other hand, specialization -- when used wisely -- can create efficiencies so it may make sense to limit the number of retrograde transitions. Whatever works.

There are two possible drawbacks to using this workflow:

1. When going backwards in the workflow, we tend to want to assign a task back to the person who worked on it originally because that person has the most context and therefore is the most efficient choice for getting the work done. For this reason, you may want to allow tasks that got into the "Ready" state from a retrograde transition to be "Assigned". Yes, I know; This contradicts the rule I stated in the previous post. And a rule like this can get confusing because people might just start directly assigning tasks in the happy path if they see Assigned tasks in the "Ready State". You probably wouldn't want to enforce the rule through automation because it would likely take lot of effort to code and doesn't give you a huge return. So I suggest creating a filter to monitor "Assigned" tasks in "Ready" states to make sure that direct assignment happens only for retrograde transitions.
   
2. “Ready for Design” could mean that a task is ready for technical design, UI design, or both. Typically, design work is done by separate teams and can happen concurrently without dependencies, though each team can also be dependent on the other. I don't know of any task tracking databases that allow you to represent concurrency of work in this way. Most teams handle this by assigning whoever is writing the specification to coordinate with designers and technical architects and manage the status of the tasks in the Design phase. I don't see a way around this. However, I still think that it is important to have separate statuses for the Design phase. These statuses allow the team to distinguish between the designers backlog and the backlog for whoever is managing the designers work. It also allows the person who is both managing designers and also contributing in another phase to indicate when their contribution is ready for the next phase.
   

Regarding the rule about having no more than one in-flight task at a time, I use filters to track this. I'm a big fan of filters. On my last project, I created a filter that allowed me to see if team-members had multiple tasks assigned to them. It's usually impossible to get this down to one task per person, so I am generous and allowed 3. I regularly sent out a note to team-members who had more than 3 tasks assigned to them at once to ask them to update their tasks. This typically worked very well and most people complied by updating their tasks or talking to me about why they had such a large number of in-flight tasks assigned to them.

I must admit that I have not yet had the opportunity to try this workflow out. If any you know of any teams working with such a workflow, or if you are inspired by this one and get the chance to work with it, please let me know.