SpecFlow project uses incorrect step definition or throws an ambiguous step error
One of my co-workers was having difficulty with their SpecFlow project this last week. When the test cases were executed either an incorrect step definition was being used or an error message appeared indicating that the step definition is ambiguous.
I was able to root cause the problem and want to document what I found.
Notes
- SpecFlow [specflow.org]
- MS Unit Test runner [msdn.microsoft.com]
- C# Selenium Driver [via nuget, by openqa.org]
- Derek Slager's .NET Regex Tester [derekslager.com]
- SpecFlow is dependent on good regexes. If you have broad regexes things could fail
Background
There were several scenarios in the SpecFlow project that were failing. Here is an example of a failing scenario:
Scenario: Main menu must have a white background color
Given I am using Firefox
And I am not logged into the page
When I log in as administrator
Then I should see ".dashboard" have a background color of "FFFFFF" displayed as "true" in "Firefox"
The above-scenario fails on the Then I should see ... test step with this error:
InvalidSelectorException was unhandled by user code
The given selector .dashboard" have a background color of "FFFFFF is either invalid or does not result in a WebElement. The following error occurred:
[Exception... "An invalid or illegal string was specified" code: "12" nsresult: "0x8053000c (NS_ERROR_DOM_SYNTAX_ERR)" location: "file:///C:/Users/Administrator/AppData/Local/Temp/2/anonymous1323563363.webdriver-profile/extensions/fxdriver@googlecode.com/components/driver_component.js Line: 6878"]
Here's a screenshot:
Edit (10-June-2012): You may see an error that look like this as well:
Assert.Inconclusive failed. No matching step definition found for one or more steps
[Binding]
Troubleshooting
I noticed that a Two-parameter step definition was being called instead of the expected Four-parameter step:
[Then(@"I should see ""(.+)"" displayed as ""(.+)""")]
public void IShouldSeeParamDisplayedAsValue(string elementIdentifier, string flag) {
// Method code in here
}
This is the step definition we were expecting to see called:
//[Then(@"I should see ""(.+)"" have a background color of ""(.+)"" displayed as ""(.+)"" in ""(.+)""")]
[When(@"I should see ""(.+)"" have a background color of ""(.+)"" displayed as ""(.+)"" in ""(.+)""")]
public void ThenIShouldSee_DashboardHaveABackgroundColorOfFFFFFFDisplayedAsTrueInFirefox(string elementIdentifier, string backgroundColor, string flag, string browserType) {
// Method code here
}
In looking at the parameters that were being passed to this step definition (via Visual Studio Debugger) it was clear that something was wrong. My Co-worker had organized their project such that parameters are denoted by wrapping double-quotes around them. In this instance SpecFlow seemed to be combining 4 parameters down to 2 and calling an unexpected or incorrect method.
Here are the parameters that were passed to method:
- .dashboard\" have a background color of \"FFFFFF
- "true\" in \"Firefox"
I noticed that the expected step definition was not available as a Then statement in the scenario since that part was commented out. I re-ran the scenario after un-commenting the [Then(@... to make the 4-parameter method available and received a different error message:
BindingException was unhandled by user code
Ambiguous step definitions found for step 'Then I should see ".dashboard" have a background color of "FFFFFF" displayed as "true" in "Firefox"': Elements.IShouldSeeParamDisplayedAsValue(String, String), Elements.ThenIShouldSee_DashboardHaveABackgroundColorOfFFFFFFDisplayedAsTrueInFirefox(String, String, String, String)
Here's a screenshot of what I saw in Visual Studio:
This was a helpful error message. It implied that SpecFlow was now confusing 2 step definitions instead just using the one that my co-worker considered to be incorrect. This error message contains a list of the step definitions that it considers to be ambiguous (listed by their method name rather than their SpecFlow regex name), which is helpful for tracking down the problem if you don't already have some idea of what is supposed to be happening.
I took a step back and thought through what I had found:
- When the 'correct' method was unavailable, SpecFlow used an unexpected step definition and failed.
- While debugging I can see that the 4 parameters were combined down to 2 to make the data fit this method
- When the correct method was available (after uncommenting it's [Then... attribute Specflow complained about an ambiguous message
- SpecFlow uses regex matching groups to find the correct method, so either there is a bug in SpecFlow or a problem with the Regex statements being used in the project
From what I could tell it appeared that SpecFlow was reading the double-quotes as part of the parameter, rather than a parameter delimiter. I took a closer look at the regexes used in the Step Definitions and found that the overly-broad .+ (That's a period with a plus sign immediately after it) was being used. Since this regex can match 'anything', I replaced it with a tighter regex that more closely matches the expected input and excludes the double-quote:
[a-zA-Z0-9#._-]+
This regex will only match letters, numbers, an actual period, underscore and hyphen. Perhaps not the most 'perfect' regex, but significantly better than what was there before
Update (10-June-2012): If you enter an incorrect regular expression you may see an error like this:
Class Initialization method xxx threw exception. System.Argumentexception: System.ArgumentException: parsing "yyy" [x-y] range in reverse order..
Error Stack Trace
System.Text.RegularExpressions.RegexParser.ScanCharClass(Boolean caseInsensitive, Boolean scanOnly)
System.Text.RegularExpressions.RegexParser.ScanRegex()
System.Text.RegularExpressions.RegexParser.Parse(String re, RegexOptions op)
System.Text.RegularExpressions.Regex..ctor(String pattern, RegexOptions options, Boolean useCache)
TechTalk.SpecFlow.Bindings.StepDefinitionBinding..ctor(...)
Conclusion
The change in regular expression solved the problem. Now we don't see any error messages when executing the test project. The take-away from all this is to not use overly-broad regular expressions in your specflow project. You will probably regret it if you do.
Also, be sure to use a valid regular expression. Use an online regex checker if you are unsure. Jusr as a note, if you want to include the open and closing brackets ( [ and ] ) in a character class in c# you have to place the closing bracket first in the character class like this:
[][/a-zA-Z0-9#._-]+
The above regex will match any number of characters (upper or lower case), numbers, periods, underscores, hyphens, Pound / hash symbols along with open and closing brackets