Finding Bugs is the Cool Thing to Do

I was presenting in a meeting and a handful of times, I said something like “I was able to find this bug and it was really cool” or “it was cool when I found this”. In a laughing manner, someone attending the meeting asked one of my developers if it was cool to them that I found those bugs. Part of me felt bad while the other laughed along with the crowd.

Spider Squashing

If you’re like me, then you believe that spiders are literally hell on earth. There are few bugs (in southeastern Wisconsin) that compare to the spider, especially those big black ones with thick legs, strutting around your house waiting to attack you while you sleep. After screaming like a girl, there is nothing more satisfying than taking a giant boot and obliterating that spider and all its worth. It feels so good and reminds me of the man that I am, that I protected my family from the evil Dr. Spider.

bugs

Software Bug Squashing

Finding a bug may never be a cool thing to a developer. It may point out their failure, or an area that they didn’t think of. Some bugs are so obvious that they should have never gotten past the gatekeeper, and over to us in QA. But it happens. As a QA analyst, finding the bug is very cool. Not that we like pointing out failures in others or anything, but rather as a team, we are working together to make the best product available. Finding and fixing any bug is actually an accomplishment because it means our users will not find the bug, or be bothered by it’s existence. They will not see the bug crawling around in production and scream like a girl when they see it. How is this not a cool thing?

Additionally, I am merely giving my developers an opportunity to be the big dawg on campus by obliterating the bug and becoming my rescuer, although they don’t seem to agree how cool it is that I found the bug.

So maybe it’s not COOL, but it is beneficial, and for that, there is no sorrow in finding and reporting a bug.

When Testing Requires Throwing the Computer Out The Window

OK, so maybe not exactly throwing your computer out the window, but I recently saw a post on LinkedIn (here) that got me thinking about how I sometimes think and feel about testing software. The image is posted below, and we’ll do two things. First I’ll describe how I relate testing to this image. Secondly, I wanna hear you caption this photo in the comments.

3b9ea7c0-90d7-45ba-9257-3735d6c9eab2-original (1)

Throwing Your Computer Out the Window

There are a few times a week that I want to throw my computer out the window. This can be for a handful of reasons.

Literally

Sometimes it gets late in the day and a bug is still crawling around your fingertips that you just cannot catch. You know you saw it, you know that it’s a bug, and that by catching it, you might get some immediate or long term recognition for it, but you just can’t seem to reproduce it. Every time you try, there is nothing wrong, everything works, but you know what the reproducing steps are, it just won’t happen this time.

I get the same feeling anytime I have a bug that I am able repro time and again, and then I call the developer over to show them and I can’t do it. The ole “works on my machine” or “nothing ever goes wrong when you’re here, can you just stand here”. It drives me nuts when I cannot repro a bug in front of the developer.

Lastly, anytime I log in to test, and it’s just a bad day for bugs. It happens, there are days that are inevitably buggy, and I can’t get through the workflow without a bug blocking me. Then when fixed, it causes more and more bugs. This repeats itself all day until the end, when you realize how little you accomplished today, because every time you went in to test, there was a bug blocking you.

Metaphorically

So maybe throwing the computer out the window isn’t the greatest test step in my test plan, but I say it to describe an example of testing and then losing power, or connection to the internet, or your computer just decides to blue screen. These aren’t really scenarios we test, maybe ever, but it’s always something to keep in mind when testing. What if I’m right in the middle of something and the site crashes or an unexpected error occurs. Do I have to start all over from the beginning? Did the site save any of my progress? Maybe a good alternative is to break up the workflow into segments that save on each step. This might prevent a large amount of data to be lost.

This could happen more on mobile devices when transferring between networks as you walk in and out of buildings and connect to WiFi. I wrote about this testing scenario on Mobile Testing Limitations on a Desktop Computer.

Caption Time

If you don’t have a LinkedIn account, there would be no way for you see the comments on the photo. But some of the good ones were “Troubleshooting Windows” or “Windows just crashed again”, and even “Jim put his computer to sleep”. I want to hear if you have anything better? Comment below!

Mobile Testing Emulators

It’s time to write about the research I’ve been doing to find a solution to the testing limitations of mobile testing on a desktop computer. I know that all of the issues could not be solved, but I wanted to see what we could accomplish with online software.

I think it’s important to first define the difference between Simulators and Emulators. While these two are nearly synonymous, there are slight differences that could assist in your determination to look into one product or another. Simulators imitate the appearance, design, or the basic features of a device, whereas an emulator will reproduce the features and actions of the device.

Lastly, my research was to find something that could resolve URLs locally, because we want to test our websites in lower testing environments before we release to production, so it’s important to be able to test our sites this way. That being said, let’s get started.

BrowserStack

I’ve got to start with my favorite product that I tested with, and say the most about it. BrowserStack has it together. They offer a laundry list of devices to choose from, with different browsers to test on with each device. They have both emulators and simulators, so you can test on actual real devices, and also by simulation if the device is a bit older. You are also able to resolve URLs locally so we were able to access our sites in house. It allows automation testing using Selenium and Appium, using actual mobile devices. I didn’t look into this feature too much, I focused more on functional testing for my efforts. Lastly, it has a built in code debugger, and the ability to inspect and read the console, which is really nice for testing iOS.

A few notable items is that the trial is 30 minutes of functional testing, but this can be overcome by signing up with different email addresses once that expires. The trial only allows you to tap into a couple iOS devices, and a couple older Android devices. The trial is a little laggy and doesn’t always display the design without minor glitches, but it worked pretty well for my needs. On the plus side, they show the entire phone so it really feels like you’re testing on a device, and you get the full experience.

BrowserStack Home

The cost of the full version can add up quickly, but they do have a variety of offerings to allow you to customize to your needs. The other downside of this product (and most the others) was that the OS versions are not the most up to date, but rather uses the OS version that was released on the phone you are testing (for example, iPhone 5 uses iOS 6.0). The last downside is their screenshot tool cannot resolve URLs behind a login screen, but that’s only an issue for our type of testing.

Overall BrowserStack gets my vote.

CrossBrowserTesting

A product of SmartBear, CrossBrowserTesting has a lot of really great features to test with. Their trial is 100 minutes of manual testing, which allows you to get a lot done to see if you want to pursue the product further, however it only allows 5 minutes at a time, so you have to be fast. Like others, their trial only allows a few Apple and Android devices, but upon payment, unlocks a ton more options. They also have a lot of devices to choose from, and many of their devices are available for testing both as real emulators and simulators. They even offer real devices as far back as iPhone 3GS and Galaxy S3, if that is a need for you.

Their trial is extremely laggy, and pretty slow to load up (which cuts into your 5 minutes), but unsure if that is just the trial or if that would be the paid version as well. Another minor downside is that their viewable area is just the screen, not the full device, but that part didn’t bother me too much, it just doesn’t give you the entire feel of using the mobile device.

They offer a great screenshot tool that allows you to quickly compare up to 25 different browsers, and they’ll even tell you design differences that they find. This tool even allows you to go behind a log in screen by passing in a username and password (though I wasn’t able to get this to work on their mobile devices, only the desktop versions).

Overall this product was great, other than it being very slow on connection, and a 5 minute trial window, this obviously has a lot of great features.

GenyMotion

GenyMotion is a pretty cool product, and is locally installed on your computer, which in theory provides some speed increases and doesn’t fully rely on a solid web connection. This product also allows you to test things like interruptions, battery usage, network connectivity changes, and more. It actually seems to do a lot of really cool things that were not available in the above products.

However, a few things to note is that this is only for testing Android devices, there is no iOS or other available devices. Also, because it’s locally installed, you are required to have the space available and the memory requirements. It also requires some additional software, and to run against local URLs, it requires a bit of server configurations.

SauceLabs

This was a fun product to try, because SauceLabs offers every device and every OS version to all those devices. You simply select a device, and then an available OS and get started. However, it takes a very long time to spin up the testing simulator. It appears that you can test local sites, but it doesn’t come without a lot of server/proxy work, of which I was not going to test in my short research period.

The coolest thing about this was all the available devices and OS versions, allowing you to test different scenarios and configurations.

The Others

There was another handful of products that I tried that didn’t give me what I needed or were old and outdated. These products may help in some areas, but for testing locally, or needing to test on the latest device, they didn’t step up to the plate. Or, some required money up front or further work to even try it out, of which I also wasn’t interested in my research period.

  • iPadian
    • Requires $20 to download.
  • Air iPhone
    • Old and outdated. Not even worth the time.
  • Xamarin Testflight
    • Requires in depth knowledge of Visual Studio and runs locally.
  • MobileTest.me
    • Old and outdated, does not work like I needed it to work.
  • Safari browser
    • Doesn’t work great on Windows and doesn’t do what I needed.
  • MobiOne
    • More of an App builder and tester, not for testing our sites.
  • Smartface
    • I just could not get it to install, and then was contacted by their sales team a bunch.
  • Sigos AppExperience
    • Free trial requires meeting with a sales rep. I didn’t do this, but the app looks cool.

The Winner

If you find yourself needing this type of testing software, I recommend BrowserStack. It costs some money, but will give you a lot of really great features. I was able to find a handful of bugs during my trial period as well, which was extra fun to realize the value before looking further into this product.

Mobile Testing Limitations on a Desktop Computer

When developing and testing mobile websites, there can be a handful of things to keep in mind, things that won’t be testable from a desktop computer. Although responsive design and basic functionality are testable, there is a greater list of things that need to be tested otherwise.

I have been doing some research lately for some mobile simulators and emulators so that mobile testing can be accomplished, but would not require purchasing all devices to keep them in house. That being said, let’s get into some of the testing limitations for mobile websites.

Design Limitations

There are a handful of things that cannot be tested by resizing your screen, even if using responsive design or a mobile website. Some of these might be intended, but most just need to be considered when developing or testing.

Hover affect

The hover affect is often used when designing desktop applications or websites, yet completely (almost) unavailable for use when designing a mobile app or website. Keep this in mind and do not allow crucial information to be hidden behind a hover affect.

A couple examples of a hover affect would be a button or link, as well as popups or tooltips. Images also contain hover affects sometimes, such as zoom features or even informational text about the image. Be sure to test alternative options for these things if they contain information that the user needs.

Hidden pages

Sometimes there is a need for pages to be hidden when working in a mobile view. This can be because the functionality is too difficult or time consuming to develop on mobile, or because it’s not an on-the-go page that will be visited. Always be aware of these pages however, if there is a difference between hiding from the user’s view and the page should be inaccessible for the user. If the latter, be sure to test entering the URL directly, and ensure the page doesn’t load or that some type of message displays.

I’ve been asked why we should worry about users entering URLs directly, as this requires them to know extensions beyond the basic .com. I usually give these questioners the basic answers like, an email contains the link, or, people may have favorites set up and shared between devices (more prevalent with Chrome and Android devices), but really, the reason is more around business requirements and expectations, or simply going above and beyond our responsibility as a tester.

Business decided features

Similar to the above approach, sometimes individual features should be hidden from the mobile user, due to restrictions on functionality, security reasons, a product manager’s decision, and others. This functionality is usually pretty easy to test responsively by resizing your screen, and the only details to note are the break points. At what pixel resolution should these features be hidden.

However, some companies like to design their mobile websites with a device detection and then rerouting to a mobile site. If this is the case, you might be able to test this on a desktop computer by navigating to the URL, but be sure to test the mobile rerouting on a device, as that cannot be tested otherwise.

Just be sure that if there are features that are not supposed to show on a mobile device, that they don’t.

Pinch zooming

Mobile pinch zooming is something that can be slightly tested using a desktop computer, simply by scrolling (zooming) in and out of the page. However, testing this way may not be sufficient, as this is not how mobile devices render their screens on pinch zoom.

Imagine the pinch zoom like the Microsoft Magnifier tool. It simply zooms in on the screen instead of restructuring the pixel density. So when testing zoom, be sure your site doesn’t realign objects, or if it does, that it’s purposeful.

Fingertip density

An important thing to test is how objects appear on a mobile device, and if it is too small for use. This is especially important for buttons, or anything that requires a click, as the standard fingertip is a quarter-inch or more. No need for frustration due to fat-fingering, just be sure the buttons are big enough to click on.

Device Limitations

There are also some things that are much more difficult to test without using an actual physical device. Though some might be somewhat testable with an emulator, there is nothing like testing on the device itself.

Slide outs or dragging

A lot of sites have side navs or other features that are accessible by sliding out from the side, or by dragging objects on the screen. Trello, for example, does a great job of utilizing their drag and drop feature on the cards, with the ability to drag to the side causing the page to scroll. Be sure to test this with respect to the device’s ownership over certain dragging motions such as the drag down from top or up from the bottom showing the phone’s menu. Or on an iPhone, if you drag from the left, it pulls up the app switcher tool.

Excessive battery use

Something that is nearly impossible to test on a desktop computer is how much battery your app or website will use. Depending on the user of your site, this may not matter, but don’t discount the battery hog that a site could be, especially when doing things like reporting or building documents.

Interruptions

Interruptions happen all the time when working, and this is no different when using your mobile device. Some interruptions aren’t as bad, such as a text that can be ignored. But there are other notifications that take over the whole screen, like a phone call, that if accepted or ignored, should return you to your site. However, based on the security of the site you’re testing, you may consider any interruption a reason to log the user out. Always consider interruptions when testing, such as locking the device, answering a call or text, or merely switching applications to check the date on your calendar.

Network traffic changes

It’s happened to all of us, we are working on something on WiFi, then we walk outside with our phone and it switches from WiFi to LTE. How does the device handle this switch, and how do we even test this? If this is a concern for your site or app, then be sure to walk around while testing. Go to the elevator, the concrete stairs, or even for a walk outside. Be sure your site handles this switch as expected and no errors are thrown on network changes.

There are a handful of other limitations to testing mobile sites on a desktop computer, so be sure to think through these when testing, and go above and beyond the testing effort next time. It’s sure to pay off for your end users.

The One With All The Names

A lot of testing is simple creativity. Everyday we get opportunities to be creative, whether testing an in-depth scenario we thought of earlier, doing some exploratory testing, or perhaps sitting in a meeting and discussing different options for a particular workflow. Where I always enjoy being creative, is when I have to set up my test data. However, there are important things to remember while testing and using test data, so that you can be more efficient to yourself and your developers. Let’s talk about Name fields.

What’s in a Name?

Most software products have some form of data entry involving personal information, such as a name, email, phone number, etc. These fields are often required in order to continue the workflow, so it’s important to use something that is quick to enter for testing efficiency, but also memorable for later use. These fields also need to be tested in other ways, such as the allowable characters, overall length, and duplicate entry.

It is useful while testing, to have a few different names prepared in order to meet different scenarios. It’s also important when demoing your products to different users, to use names that make sense and are not distracting, depending on who you’re talking to.

The Good

When testing my products and using data that only I or my development team will see, I like to use character names. I use Homer Simpson and his wife, Marge, with three kids: Bart, Lisa, and Maggie. The all American family that fits into most scenarios without problems. This way, when I log a bug about children not returning in a personal profile, a developer can quickly triage the bug and know the expected results: that I should have three kids, their names, genders, etc.

I will also stick with a theme or category when testing names, because when I need to pull up data in the database or the UI, I can quickly see if someone doesn’t belong. For these scenarios, I sometimes use superheros, like Clark Kent and his wife, Lois Lane. Then I’ll add a bunch more users to the product like Batman, Robin, Ironman, Spiderman, and many others (crossing universes is not a problem for me). Using names of interest makes testing repeat activities fun and exciting.

The Bad

When demoing a product to another team, especially upper management, it might not be in good taste to use a distracting name such as the Simpson family, but rather something more generic like the Smith family. This is especially important when doing usability testing with our end users, because a simple misspelling or clever use of a family name, could result in a poor usability session, or distract the user from providing all the necessary feedback that we hope for.

We find it works best during training sessions, to use basic names like John Smith and his wife Jane. This way, people are able to read the names, quickly identify who the users are and their marital status or genders, and then move on to important information.

Sometimes random name generators are fine as well, but if you aren’t careful, you can get names that aren’t really names, but more just words. Or you can end up accidentally creating someone famous. This can also turn an important conversation into a distracting one, when the name turns out to be Quail Bird, John Hamm, or Donald Trump. If we need names for a demonstration, it’s important to review the list of randomly returned ones, and edit any that might take focus away.

Additionally when using random names, it doesn’t represent anything or mean anything to the tester or developer, so when needing particular information, it can get lost when thinking, which user was that, or I can’t remember the person’s name I was using. That doesn’t mean it’s not helpful to have randomly generated names sometimes, but it’s important to remember who you’re using in what tests.

The Ugly

I have made the mistake of using very poor names in my testing before. Sometimes I will need to get through the data entry form or create a quick name, so that I can test something important down the line. I will enter a name like a a, b test, or lfdjsio asdf. Most times these are not a big deal to use and can allow you to move on through the form, but sometimes I need to get information back from the UI or the database, and I cannot remember who I was testing with. These type of names mean nothing, and our relate-able brains do nothing to store the name.

These are also much more difficult for a developer to recognize the scenario, and triage a bug against it. They don’t provide context to the problem, and don’t mean anything to a name, so this can actually turn into more questions like, is this a name or just some random data?

The biggest problem I’ve seen when people demo their product with names like this, is it doesn’t provide context into the UI. Let’s pretend for a moment that we are showing a user’s profile and personal information, and at the top left of the page, it says:

dsaflk safdjlk
9999 dfslakj fds
fdsafddfsdf, WI 12345

A user might be able to gather from that, that it’s a first and last name, and their address, but it doesn’t provide any value towards the conversation, and takes peoples thoughts away from the goal. Now let’s use a good example:

John Smith
9999 S. Theodore Drive
Franksville, WI 28876

Right away, a user will see this, understand the context, and be able to move to important things. Also, when they are looking at the UI, they will know what information they are expecting, and not be caught off guard by random letters and numbers.

Testing the Name Fields

All of the above is important information when using names during testing or demonstrations. However, it’s extremely important to thoroughly test the name fields on a form, during the testing cycle. Entering basic names like Homer Simpson should pretty much always work. If there was a bug for not accepting this value, I would be shocked. But there are a lot of names out there in the world, so here are some examples of things to test:

Very long names

Most people’s names are fairly short, 10 or less characters. But there are a lot of examples of people’s names that are 15-30 characters, and depending on the region of your product, you could get even longer. It’s important to test length to ensure the data doesn’t truncate, or if it does, that it’s intentional.

Hyphenated names

More common with last names, but occasionally there is a Mary-Jo, John-Paul, or Anne-Marie. Don’t forget this when testing to ensure the data can be stored and returned appropriately.

Names with punctuation

Similar to the hyphen, the last name is a more common scenario. Be sure to test names with apostrophes like O’Brien and O’Connor, or umlauts, tildes,and many other (but not all) foreign characters. Examples like Chloë and Beyoncé should be tested.

Duplicate names

We all know someone with their first and last name the same. Be sure to test this, as well as two users with the same exact names. Maybe they’re strangers, or a father-son combo.

This is definitely not an all encompassing list, but should kick start your testing efforts to ensure your system is fully supportive of it’s users. It’s important to have bug free software, and creativity while testing is often the first place to start. Just remember who’s going to see your data and when to use appropriate testing data.

What is your go-to name or theme when entering test data?

The License Plate Conundrum

The other day I was driving home from work, and at a stoplight, I noticed that the license plate in front of me looked a little different than I remembered others looking. I stared at it for a bit, analyzing it like I do all things around me, and realized that it was not in the normal format that I recalled others. Then I glanced at every other car around me and found that this car’s plate was in fact, different than all the other plates.

Before you continue, here’s a challenge:

WI recently released a new license plate format with their letters and numbers. Can you think of what it is? There are thousands of them already out there now, so you have definitely seen one.

Analyzing the World

All around us there are inconsistencies between similar objects and being in QA teaches us to find those differences. For some, this is a learned ability. It requires time and practice, and the desire to find the problems. For myself, this ability has always come naturally. I’ve always enjoyed the Spot the Difference books, or Hidden Objects in Highlights, or even What’s Wrong with this Image found in similar magazines.

The idea of consistency is actually a simple concept. Either the two pictures, sentences, or layouts are the same or they are not. It’s very black and white. But sometimes it takes an extra moment or a trained eye to find those things. The next time you are driving down the street or viewing the world, try to find an inconsistency between two objects that appear the same, but actually have a difference.

Game Time

I was recently working with some friends on building a fence for a community center, and their sign read like this:

Hours of Operation:
Mon – Fri: 8:00am – 5:00pm
Sat: 9:00am – 3:00 pm
Sun. 8:00am-3:00pm

How many differences or inconsistencies do you see in this sign? Believe it or not, there are three things that I would report as an inconsistency bug. If you found more than three, then either you’re wrong, or I will have to look harder!

For fun, let’s spot the difference. Find the single difference between these two photos.

SpotDifference1

Last one.

SpotDifference2

Would you believe me if I told you I found both of these in under 30 seconds? There is actually a secret to this, and has nothing to do with being in QA.

If you cross your eyes, so that each eye is looking at the opposite photo, the photos become blurry, but it’ll make one clear image in between them, that you can actually then look at and move around on. The difference will appear green and flashing.

Inconsistency Bugs

Sometimes these types of bugs seem trivial to log, or maybe even fix. So what if one page says “Log in” and the other page says “Login”? Or, should we care that the buttons are sentence cased in most of the site, but there is one page way over here that has a title cased button? What about when there is a pixel difference from the item next to it?

All these things can deter users from desiring to use your product. Think about the last time you were on a website, and noticed something slightly off. Perhaps you couldn’t quite place what it was, but it may have bothered you, or you spent time thinking of it, or if you’re like me, you went back to the previous pages to see if you are crazy, or to confirm that you saw the difference. Whatever it is, it wastes time for the user, and causes them to think twice about their workflow.

Even the tiniest of bugs can cause harm to the system. They might not be critical or severe bugs or cause data issues, but they could cause a hiccup in the user’s workflow, and therefore should at least be mentioned.

If you read my thoughts on Collaborative Testing, you’d find that a lot of these things can be caught during that process, and fixed very quickly, so that no bug even needs to be logged. This is the ideal scenario. However, if it’s missed then, but caught during testing, it is always something that should be reported and allowed a decision to be made, because you never know if your user will catch it, and what the result of that could be.

The License Plate

By now you’ve had plenty of time to think about what format the new WI license plates are in. But in case you can’t figure it out, here is one more clue:

AAA-123
AAA-1234
123-AAA
1234-AAA

Think about your license plate and if there are any plates from your past you can remember.

Just so you have the answer before you go, the new plates are in AAA-1234 format. This was because they hit the max on letters/numbers, and it was time to start over again.

What are your thoughts on those minor inconsistency bugs? Log them or don’t? Or, do you even see them most of the time?

Bonus points if you can identify the TV show that this post’s title is based around.

Collaborative Testing

There are many different styles of QAing a project, and I’ve found that much of it depends on who you are working with, what project you’re working on, and what is expected of the final outcome. I have worked hard at developing the Collaborative Testing environment at Zywave, but that doesn’t mean it’s the best method for everyone, it’s just another method that might work, if you’re feeling stuck in the ways you’re currently testing.

So What is Collaborative Testing?

At Zywave, we define Collaborative Testing as the QA team (often more specific to the functional testers) working side-by-side with the dev team, to ensure accuracy of the acceptance criteria, prior to their work getting into the QA environment, where it officially becomes ready to test. This isn’t the all inclusive definition, nor is it one that has been officially decided on, but rather just a statement about what it looks like.

So what does it look like?

When our project managers/directors/BAs give the team a story with acceptance criteria (and everything has been planned and decided on), the dev team begins their work building out the functionality. During this time, I often ask for them to show me what they’re doing, walk me through their code (even though I don’t really understand it all), show off the UI a little, so we can have an open discussion of their work. We have been able to avoid a lot of problematic situations, because we were able to clear up questions prior to them completing there work.

I have found that people interpret stories slightly differently when simply reading the story. For example, we’ve all seen the picture of the swing in the tree, what the BA planned, the designer created, the dev developed, and the qa tested. That image shows that each team member thought about the swing a little differently, and therefore possibly incorrect from what the project manager was hoping for.

These are things we attempt to avoid with Collaborative Testing. By working along side of the developer while they are developing the code, it ensures that we are on the same page with each other. Then, when questions arise, we can approach the BA and get further clarity on expectations. It saves a lot of time and effort, that is often seen with over-the-wall or waterfall approaches.

What about bugs?

Bugs are always going to happen. It’s inevitable (a post to come perhaps). However, I have found that by working closely with devs during the development process, we are able to prevent a large handful of bugs from making their way to the QA environment. We do this a few different ways.

When I Drive

Sometimes the developer will call me over and ask me to sit at their computer and test their code locally, while they go get a cup of coffee. Assuming I’ve already got a good understanding of the story requirements and possibly written my test plan, I am able to get into their work and pull things apart. Just like I might have done (or still will do) when the code is in QA, I can test it hard on their computer, and jot down a list of items for them to fix that I was able to catch. This, however, does not get the developer thinking like I do, but it does allow me to get into an in-depth test scenario and potentially reveal a big issue that I was expecting to see.

When Developers Drive

A similar style would be for the developer to drive, and I will look over their shoulder and ask them to perform miscellaneous clicks and entries, ensuring accuracy of the requirements. Sometimes we are able to get into deeper scenarios, but in this method, it’s often a bit more surface level of the functionality. This method gets the developer into the mindset of the QA analyst, beginning to think about different workflows to test on their own, before calling me over to collaborate with them next time.

I have found many developers use either or both methods with me at different times, and the results of these conversations are great. I’ve seen devs testing their code in ways they normally wouldn’t, or when they call me over, and I throw a wild scenario at them, they tell me they’ve already tested and verified it works, they will prove it, and then inform me that they knew I was going to ask that. We have now begun to save everyone time, and the developer is beginning to produce less bugs in their code up front.

What Are the Dilemmas?

There are a handful of things that can hold this process up, cause it to not work so great, or prevent Collaborative Testing from happening all together. Some of these things can be worked on or fixed, and some things will just never change.

The Dev

We all know about the stereotypical developer, with their quiet personality, perhaps a bit shy, maybe staring at the floor when they talk to you, etc. Sometimes we gain this interpretation from experience, and other times from TV and movies like IT Crowd, The Social Network, or Silicon Valley. In my experience, I have found a wide variety of developer styles:

  • The stereotypical devs
  • The very outgoing devs
  • The confident or possibly cocky devs
  • The “my code is perfect” devs
  • The “love the feedback” devs (my favorite)
  • and many more

Each dev works differently. It’s up to you as the tester to figure out your team member’s personality, and challenge them to try something new. At Zywave, our Agile approach allows us to try something for a short time, and if it works, we keep it. If it fails miserably, we evaluate why, and then either try again or drop it completely.

The QA

Collaborative Testing isn’t for everyone, no method is. If you are the type of tester who prefers working entirely alone, or are not bold enough to ask a developer to show you what they have, then you might not be interested in Collaborative Testing. Perhaps you’re stuck in your old ways and what you’ve been doing just works. Well I challenge you to step out of your comfort zone, have a discussion with your development team, and attempt some of these methods described above. The efforts will pay off when you realize they fix almost all of your bugs before their code even gets into your hands.

Outsourcing

This has been an interesting experience for me and a challenge to overcome. I’ve had the opportunity to work with outsourced teams, where the developers where in another location, and I was in the office. We had to communicate entirely over chat programs and the occasional phone call, while still attempting to get collaboration done together.

We have made this work by first, explaining the concepts above, and what I wanted to do with them. Second, repeatedly ask them where they’re at in their work throughout the day, because it’s so easy to go a whole day and not talk when there is no in-person interaction. And lastly, joining them in a screen-sharing program and while they walk me through their work, design, and development of the UI. Then I get the opportunity to have them go through a few scenarios before promoting this work to QA. This falls under the developer driving methods above.

Company Policy or Structure

One of the hardest barriers to overcome, and perhaps one of the most common I’ve heard from talking with the local QA community, is that the office structure is setup in a way to promote department collaboration instead of the product or workflow. Meaning, the developers all sit together in one room, the QA in another, and so on. This is a difficult scenario, and one that may not work for this type of testing efforts. However, if the company allows, you could go for a walk and visit their desk, and take that time to have them show you some things.

Another structure I’ve heard is the over-the-wall approach that some companies are so used to. With over-the wall testing, code gets out of the developers hands entirely, and into QA for testing. Then when there is a bug, they hand it back to the dev to fix. QA moves on, and developer eventually gets to the problem, fixes it, and tosses it back over. This cycle repeats, and essentially wastes a lot of time for both parties.

Follow Up

What are you thoughts on this? Have you tried something like this before? Is this just crazy talk? Let me know what you’re thinking or if you have any other advice on this subject. I love learning additional ways to test with different people and products.