Flaky tests are annoying. Flaky tests are infuriating when they are run in continuous integration and you can’t see why it’s falling without bugging the SCM guys. There are a few ideas to help alleviate this problem.

1. First and foremost, have a test infrastructure in place that allows you to write stable tests. Whether that means using the page-object pattern or ensuring you and your developers are on the same page with regards to stable html tags for you to key off of.
2. Have some good logging in place that you can dig into to figure it why it failed. If a test fails, hopefully you have a log of what happened before the test failed, when the test failed, and some sort of stack-trace.
3. On test failure, take a screenshot of the browser. After implementing this into my test suite and CI build, my anxiety over flaky tests has been greatly reduced. Being able to see the state of the browser at the time of failure has allowed me to quickly diagnose and treat a flaky test.

Implementing the screenshot capability into SpecFlow was surprisingly easy. First off, I added a little static method to a WebDriver extensions class that looks like this:

1
2
3
4
5

   public static void TakeScreenshot(this IWebDriver driver, string filename)
          {
              Screenshot ss = ((ITakesScreenshot)driver).GetScreenshot();
              ss.SaveAsFile(filename, System.Drawing.Imaging.ImageFormat.Png);
          }

It takes in a webdriver driver instance and a string for the screenshot filename. In my SpecFlow hooks, I added an AfterScenario hook that looks like this:

1
2
3
4
5
6
7
8
9

    [AfterScenario]
    public static void AfterScenario()
    {
      //Check to see if there were any errors in the scenario and if there were, take a screenshot of the browser
        if (ScenarioContext.Current.TestError != null)
        {
            driver.TakeScreenshot(ScenarioContext.Current.ScenarioInfo.Title.ToString() + ".png");
        }
    }

It’s just that simple. I set the file name as the scenario title so I can easily identify them. By default, the screenshots are saved to either the bin/debug or bin/release folder. In TeamCity, I had our SCM team save any .png files in the bin/release folder and save those as artifacts.

I found Trish Koo’s blog on how to automate the input of test data while doing exploratory testing enlightening. A lot of the blog/twitter chatter recently has been about test automation and specifically doing too much automating, at least on the GUI level. As a new tester, I sometimes have to remind myself that some things don’t need to be automated. That being said, anything that can help relieve the pain of manual testing is always helpful. I don’t know about how you do quick exploratory testing, but any time I come across a text field, odds are I’m going to fill it with some variation of “test”. While I found her solution helpful, having to use a keyboard shortcut to copy data to my clipboard, then use another to paste the data, is too much work for me. For the past couple of months, I have been playing with TextExpander and I thought I could add a snippet to input test data. I started by finding some test data and putting it in a .txt file, one item per line. From there, riffing on Trish’s code, I added the following snippet:

1
2
3
4
5
6
7

     #!/usr/bin/env ruby -wKU

      dict = File.open('/<PathToYourFile>/File.txt')
      lines = dict.readlines
      word = lines[rand(lines.size)]
      puts word
  

I’ve found this solution helps speed up my manual testing in addition to adding more realistic data to the database.