Sam Bedekar, Microsoft
We will introduce the Simple Language for Scenarios (SLS). In SLS, we constrained users into writing test cases in a particular text-based format (not XML based). This reduced coding-style flexibility, but gave us a single paradigm of writing test cases and a point of injection via the interpreter. We used this point to achieve several goals – we were able to automatically marshal calls between different thread models, adapt existing cases for API parameter testing and enable security fuzzing without requiring each tester to have to write code to do so. For string injection we wrote an API string testing class where the script engine could automatically call each method in a loop and pass in all possible parameters within the context of an existing case simply by annotating the “start” and “end” points in the test case where the engine needed to loop. Additionally, with API testing, we were able to run a case to a particular point, and then bring up a UI to allow ad hoc testing. The steps executed by the tester in the UI would then be written back as script language that could then be incorporated into the test case library.
Ultimately, we landed with a hybrid model that managed to leverage the strengths of both approaches. Along the journey, we faced several facets to consider including development time for a test case, reducing cost of failure analysis, adaptability/extensibility of a test case to new environments and more. This paper will go into the technical details of our findings and how other software projects could benefit from our lessons-learned to more effectively deploy automated test cases.