Omar Alshathry is a research student at the Software Technology Laboratory at De Monfot University, UK. His research interests are software quality assurance, project management and process improvement models.

Quality initiatives could make a significant difference, but suffer from tight budgets. Since testing is never finished, it is desired to drive testing efforts to parts of the software product that would benefit the most.

The root cause of the software crisis is that we cannot see or touch the software like we can see or touch the work products of other engineering disciplines, such as construction or mechanics. You can instantly see whether a car is missing the bumper, a door or the windshield. The quality of a software end product is not as obviously apparent.

What if we could make the quality of software visible? This would govern testing initiatives and could help to decide if we are really ready for go life. The talk will provide in-depth insights on how to visualize different aspects of software quality. Reference to valuable literature is given (e.g. University of Maryland, Professor Edward Tufte, etc.).

The talk will feature a demonstration of the software quality visualization (qa-visualizer) using the Mozilla Firefox browser code base as an example. The qa-visualizer’s implementation is based on several open source projects (CouchDb, Python plus some extensions, JavaScript, YUI3). Mark Fink is the maintainer of the open source qa-visualizer.

Mark Fink earned a diploma degree in Computer Science at the Georg-Simon-Ohm UAS in Nuremberg, Germany. Programming is pure fun for him, so after graduation he started as … a programmer! Already in his first project assignment he realized that testing of the product was not given enough attention, which resulted in insufficient quality of the product. That experience was bugging him constantly. Consequently he moved into testing and test automation.

Mark has years of professional experience in the areas of continuous integration, performance testing and functional testing. He started programming in Python in 2005 when he needed tools for bulk file analysis. Python is his first choice for programming testing tools and utilities.

Mark Fink currently works as a performance test engineer at a Swiss bank.

The gauge I try to use is – if the user feels the product is slow, your product is slow – no matter what process or tools you have used for performance; i.e. the customer is always right. If you ignore your customer, they will ignore your product

There is no silver bullet; you can build a product that meets the performance expectations of users by planning it upfront, managing with the right set of Metrics and leveraging Six Sigma techniques. It’s about understanding your audience/users/customers – your product is used by experts, novices and executives – its not one or other – your product should take care of performance for all of them.

In this presentation, Mukesh will talk about Six Sigma techniques that can be used to improve performance of your web & mobile application and how you can plan the right thing, do the right thing and get the right things for the right user at the right time – every time. He will provide demo, tools and techniques along with metrics for managing performance of your applications.

He will also share his experience on how he used Six Sigma techniques in Microsoft to improve performance of Bing, MSN, Messenger, Mobile, Hotmail and Outlook.

Mukesh is a Sr. Engineering Manager in Test at Microsoft, managing and leading multiple teams in Redmond & India to build a next generation Online Service Advertising platform for Web & Mobile. Prior to this, he was driving quality of service, strategy and continuous improvement in online services. He has worked on many teams in Microsoft and has experience shipping releases of major products. His passion is to work with marketing teams to understand user intent, build tools to capture user experience and drive product improvements. For the past 15 years, he has held key positions including developer, business analyst, solutions architect and process/quality manager. He has directed & coached engineering teams to deliver projects on time with high quality and within budget.

Prior to Microsoft, he was associated with several multinational corporations leading project, quality and program management. Along with a bachelor’s degree in computer engineering and science, he has achieved various certifications. In 2006, he was honored as “Best Six Sigma Black Belt” by iSixSigma magazine. He is author of the bookDelivering Successful Projects using TSP and Six Sigma and is currently writing a book on Web Performance Improvements.

This game illustrates how working with larger groups of people introduces new challenges. Communication issues in this instance. Likewise larger software development projects can introduce new challenges for agile teams to overcome.

In this paper we’ll look at different techniques for addressing three challenges commonly associated with larger projects:

• Scaling the development team size – It’s easy to know what everyone is doing without taking up most of your day when there are three or four of you. This is much harder to do, without spending more time, as the development team grows to 10, 20 or more.
• Big projects have big features – You’re much more likely to run into features which will require multiple iterations to complete on larger projects. How do you handle these iteration-spanning features in an agile manner?
• Reducing the post development end game – External factors can often increase the time between completion of feature development and the software available for sale.

These approaches may help address similar issues on your projects.

Don Hanson founded his first company and began writing commercial software in the early 90′s developing an evolving line of animation plug-in products. He has since worked on projects ranging from the mobile client for a wireless navigation startup to enterprise-level commercial software products. Don is the development manager for McAfee’s flagship enterprise security management product.

This paper explores the challenges of testing software on a robot in the physical world. Through the case study of running a LEGO robot with motors and sensors on a FIRST LEGO League competition table, it characterizes the environment factors, some of which are not controllable. Through the sensors that are available on a robot, it characterizes the uncertainties from the sensor readings and the location of the robot. It then describes how it employs a software testing process to test if the robot can perform missions reliably given the less than perfect environment and sensors.

The software testing approach in the physical world has only scratched the surface of the complexity of the quality assurance of a robot. Early results from the case study here demonstrates that taking lots of sensor readings, establishing a relationship between the sensor data and the results of the experiments would simulate testing on the equivalence of many different environments and reduce the number of runs to catch failures.

The contributions of this paper are:

1. Characterization of environment factors in a complex system such as the physical world.
2. Establishing a method to develop the relationship with imprecise sensor readings and the functionality of the software on the robot.
3. Applying the relationship to improve the software testing process to assure the quality and reliability of the robot.

Kingsum Chow is a principal engineer from the Intel Software and Services Group (SSG). He has been working for Intel since receiving his Ph.D. in Computer Science and Engineering from the University of Washington in 1996. He has published more than 40 technical papers and patents. Outside of work, he has coached a FIRST LEGO League team since 2006.

This paper follows the experimental ride of our team of functional testers and the new approach towards Application Security Testing. As a team, we faced many challenges from the beginning in terms of understanding the Threat Model approach, issues of implementing the same in a time-constrained manner, as well as dealing with the failure to finally implement and yield value. We subsequently created a self-adjusting model, which we have coined as “Adaptive Application Security Testing Model”.

This model follows a systematic ladder approach where your expertise, skill set and result oriented strategy are enhanced. It tries to break free from the traditional approach and instigates an elementary and adaptable methodology. It tries to bridge the gap by identifying security defects early in the test cycle without compromising the rigorous Threat Modeling approach. Overall, it covers security efforts, starting from developing the abuse cases, creating attack model, carrying out attacks on the product and subsequent follow-up.

From our expertise, we have broken down Adaptive model into two sequential testing objectives – Peripheral and Adversarial.

As applicable to functional defects, initial security issues can be found with minimal effort. Peripheral Security Testing tries to expose this behavior and takes a lead in finding early security defects. This specially designed black-box approach, with the help of attack models, exposes the surface but crucial vulnerabilities.

On the other hand, Adversarial Security Testing requires the tester to dwell inside the product with a set of certain prerequisites. These include research on historical vulnerabilities, code base knowledge, architectural understanding and application security testing expertise. This hard-to-break approach, once followed in righteous manner, gives you extra mileage in highlighting product vulnerabilities.

The snapshot below explains the process followed by the two approaches. It is important to adapt these approaches and follow them in a sequential way.

The Adaptive model aims to maximize the efforts of an Application Security tester. With hard-hitting deadlines and resource optimization, this paper gives you a strategy to achieve application security without dealing with compromises.

To conclude, here are some of the strategic points, which an application security tester can leverage:

1. Position themselves with respect to Adaptive Ladder
2. Understanding the Application Security from a rudimentary but Adaptive perspective
3. How to create Attack Models
4. Real-time case studies

Ashish Khandelwal has more than 5.5 years of Software Testing experience. He holds a B.Tech degree from IIT Kanpur and works as a Senior QA Engineer with the McAfee Host DLP product solution group. Being a CEH, he is interested in the latest security testing trends, and continuously seeks to improve software security. Ashish works towards becoming a technology solution consultant/architect by providing technical insight to different verticals in testing solutions.

Webster’s definition: “Something complex” or “The quality or state of being complex.” Look up “complex” and you get “hard to separate, analyze or solve”.

This defines a lot of what software testing sometimes feels like to me, and after some research with colleagues, I find I am not alone.

To study this further, I enlisted the Weekend Tester (India) crew last week (http://weekendtesting.com/archives/1031) to have them test what I thought was a complex application: Mifos, a microfinancing application by the Grameen Foundation.

I gave them this charter: Read their new user manual – http://en.flossmanuals.net/bin/view/Mifos/WebHome. Using this manual, read the welcome (http://en.flossmanuals.net/bin/view/Mifos/Welcome) to learn a bit about the purpose of Mifos and pick one of the following sections, then test the application to discover new bugs, usability improvements and user manual improvements

I wanted to discuss the notion of the product’s complexity in the debrief. But what I didn’t account for was that the conversation was heightened by the fact that we used a very SIMPLE supporting application to collaborate our test notes in real time (www.typewith.me). This application was SO simple that it provided a nice contrast to Mifos and spurred some great conversation, leading me to conclude the following:

1. Complexity means too many factors to handle at once. One tester said: “I stopped reading the manual because there was too much to read (too complex). I felt I had to try the product to get context then re-read the manual. It was only at that stage that I could read the manual in detail to find issues with the wording, etc. I found the manual was non-technical, and I needed some technical content to understand how to enter data in the fields – e.g. the limits of the fields, or field types, or basic date rules.”
2. Complexity means being overwhelmed — having too little time to process information. “The manual was too complex, but not the wording. It was written in a very basic way. It was just too much too soon. My brain was not prepared to read all that information. I had to try to learn it quickly in increments. That meant doing about 3 passes over the manual, as I was learning the data entry form.”
3. Complexity is the inability (or desire) to distill or simplify information: “If we model at each level or state, then we improve our level of understanding, and the depth of confidence with complexity. It’s just like a baseball game: you move from base to base, eventually scoring a run. Modeling helps us find a safe place when looking at complex pieces.”
4. Complexity depends on the emotional relationship with the object of study: Whether it be confidence, safety, or familiarity, said a tester, “You have to acknowledge the level of detail you are at, and move between your ‘safe zone’ and the complex zone. There is always a level you are confident with. If you freak out, then go back to your ‘safe place’ and start again.

With this paper, I intend to share how these 4 ideas can be explored to make something that may seem complex be more simple.

Jon has over 15 years of experience, including Fortune 500 and start-up companies, serving in the Quality Assistance and Software Testing domain. He is a co-author of a Microsoft Patterns and Practices book on Acceptance Testing (2010) and is an award-winning keynote speaker for major testing conferences (STAR, QAI, BCS SIGIST). He has served as the vice president of conferences for the Association for Software Testing; invented a method for managing and measuring exploratory testing; a method for coaching testers (Open-Book Testing) and categorizing risk (Color-Aided Test Design) and has published over 50 articles, whitepapers, and presentations about the notions of value of exploration and rapid, risk-based testing. He has been with Quardev for the past 6 years and serves as the speaker chairman for the acclaimed Seattle non-profit QASIG testing forum (www.qasig.org).

An overall approach to garnering leadership support, identifying key improvement opportunities and driving progress is shared. Specific success metrics are shared as well as data on what constitutes world-class, what is typical for commercial software organizations and how several large software companies are delivering for their customers in a world-class manner. Specific suggestions are discussed on how to help organizations become more customer focused to ensure they are working on what matters most to external and internal customers.

This discussion includes the use of the Net Promoter concept to discover what is delighting customers and causing them to recommend the product or service to others, as well as how to identify detractors and what needs to be most improved with the software.

The benefits of a number of key engineering best practices including personal reviews, peer reviews, unit testing and the Team Software Process and Personal Software Process are discussed as well as key barriers to their adoption. A roadmap is shared for how to approach quality improvement initiatives with a multi-year view, starting first with eager (innovative) adopters and ensuring they are rewarded and recognized for their initiative. Next, you work with early adopters, who will be attracted by the innovators, who will be promoters of the changes as well as supplying proof point that these changes pay off for employees, customers and ultimately the business.

Self-directed teams adopting engineering best practices including TSP and PSP were able to ensure that quality, schedule, scope and cost were not strict trade-offs. Through a combination of better planning and estimation, improved project execution and effective use of quality best practices, such as unit testing and peer reviews, these teams delivered high quality software on schedule and budget, with good work-life balance for the team. Knowledge gained from retrospectives and improved metrics helped drive continuous and significant improvement in product and process. The biggest change was in the mindset that a team owns its work processes and has primary responsibility for defining and improving it.

The goals and metrics used to define, track and drive improvement will be discussed. Often establishing what’s important and mechanisms, including relevant metrics with the right altitude and precision, are key to progress. These metrics assess progress from a customer, employee and business stakeholder perspective and demonstrate the continuous improvement required to ensure continued individual and organizational commitment. Lastly, the long-term results from one large organization are shared to serve as a proof point for the value of adopting these best practices from the viewpoint of all key stakeholders, especially employees and customers.

Jim Sartain is a Senior Director responsible for Software Quality. He leads a team responsible for inspiring, driving and enabling continuous quality improvement across Adobe. Prior to Adobe Systems, Mr. Sartain held quality/process improvement and product development positions at Intuit and Hewlett-Packard for 25 years. While at Intuit, Jim drove significant improvement in quality and software engineering practices across the company. His last role at HP was as the CTO/CIO for an Airline Reservation Systems business that serviced low-cost airlines, including JetBlue and RyanAir. Mr. Sartain earned his bachelor’s degree in computer science and psychology from the University of Oregon, and his M.S. in management of technology from Walden University.

Test automation so far has been perceived as something that can cater to our regression or smoke tests and, if time permits, to functional tests. How often have you been challenged with such an approach? How often have you thought of complementing this with white box driven automation approach? In this paper, we propose to bring innovation to this age-old approach of test automation. This paper substantiates its ideas with real case implementations from the projects where the concepts are applied. This experience helps us conclude that a test automation strategy that includes both white box and black box automation techniques can streamline the achievement of more effective and efficient testing.

This paper talks about various white box automation ideas that can be implemented across the different stages of the software development life cycle that complement the black box automation efforts. Some novel ways of white box testing and how it can be automated are also discussed. It describes when white box automation can be more effective and essential than black box automation.

In the current complex software environment, maintaining black box automation scripts can become a very cumbersome job with the frequency of changes to the application under test. Since there is no direct API call involved, UI changes can make black box automated test scripts fragile. This paper does not insist on replacing black box automation completely, rather it proposes to complement them with white box test automation.

Another important aspect of test automation is functional validation and verification. Employing the right approach for automating the validation and verification step as part of test automation is critical and will also be discussed in this paper.

The concepts shared are exemplified using our observations and the practices and processes we adopted to achieve more out of our white box automation. Finally, we share our learning while implementing white box automation.

Key Takeaways:

1. Shortcomings of black box automation
2. How to take test automation upstream in SDLC
3. How white box automation can fill in gaps in your overall automation strategy
4. White box automation strategy
5. Case studies and experience sharing
6. Challenges while implementing white box automation and how to overcome them

Sushil Karwa is a QA Technical Lead at McAfee. He has a Master’s degree in Quality Management from BITS Pilani University, India. His testing and QA experience include a focus on white box test automation for a web-based product. He has been associated with McAfee for the last 7 years. His role involves identification of security risks, threat model preparation, security code audits and test planning for security management framework. At McAfee, he has been involved in initiating and implementing the white box test automation process for various product teams. Sushil is a Certified Ethical Hacker and Security Analyst from EC-Council, USA.

Sasmita Panda is a Senior QA Engineer at McAfee and works with the Host Intrusion Prevention System product team. She has a Master’s Degree in Computer Application from Madras University, India. Her testing and QA experience include a focus on white box testing automation, security testing and identification of different techniques for measuring code coverage.

Many projects lean on API documents that have been automatically extracted from source code comments. Other teams construct elaborate diagrams that explain every detail of the architecture. While these can both be helpful as a reference, they do little to answer the basic questions about the code: what’s this for? What file should I look in first? What’s this project’s equivalent of a “Hello world!” program?

In this presentation, we’ll use the metaphor of a magazine article to think about ways to help someone learn an unfamiliar code base. That person may be a new hire today, or it may be you, five years from now.

Getting the emphasis and level of detail right are crucial. Everything else, especially the specific writing tool you choose, is secondary. Even so, we’ll spend some time looking at a couple of open-source software packages that may be of help on your quest to produce great documentation.

Ian Dees saw his first Timex Sinclair 1000 over 20 years ago, and was instantly hooked. Since then, he’s debugged embedded assembly code using an oscilloscope, written desktop apps in C++, and joyfully employed scripting languages to make testing less painful. Ian currently writes GUI code for handheld instruments as a Software Engineer at Tektronix. Ian is the author of the book Scripted GUI Testing With Ruby.

This scenario is more common than we as technology companies want to admit. It is an especially challenging experience in the consumer and small business marketplace where our PC and network environments are not managed by IT department policies and processes. At HP, we have categorized call-center calls to help us monetize the cost of install failures in the field. We knew the data – at one point 70-75% of calls were categorized as “install” calls. We developed tools to help us mine what the failures were, but we could not seem to effectively change the install call-center rates in the consumer marketplaces. That has been changing!

This paper will describe the design and development of a test lab that has been effecting change in the R&D requirements and development processes to reduce the install call rates while supporting the adoption of wireless technology advances in the customer segment. The paper will:

• Describe the process of aligning on a definition for install success
• Identifying factors that contribute to install success and failures
• How unactionable, field-found issues influenced test lab design
• How test lab design objectives drove executive sponsorship, development, and implementation of a whole new testing service
• How the whole process has driven changes into R&D requirements and deliverables
• Examination of recent call-center data to validate affects of these changes

Kathleen Naughton is a SW Test Architect with Hewlett-Packard. Kathleen has over 20 years experience in the technology industry implementing a parts order system on one of the first nation-wide WANs for Tektronix, verifying graphical software development kits for CA, designing testing for streaming media cell phone software with Packet Video and, most recently, printer and scanner test design and lead for Hewlett-Packard’s Imaging and Printing Group. She has been the printer installer test lead for over 5 years, co-chaired the Install Success Taskforce, and directed multiple programs and studies performed to reduce the install call rates. She is an active member of the Anita Borg Institute’s organizing committee for the Grace Hopper Celebration of Women in Computing conference as well as a regular presenter at that conference.

In the past, the flow of code was often hampered by component teams not efficiently and effectively qualifying their code contributions or reverse integrations (RIs). As a result, dependent component teams started stalling, as breaking changes proliferated with respect to product and test code – they paid a heavy price in terms of failure analysis. Those dependent teams also became wary of merging those unstable code contributions into their own code bases as part of their forward integrations (FIs). In short, the flow of code became unpredictable, resulting in a code base that was often in an unstable state.

In this paper, we describe some of the challenges that we faced in reinvigorating the code flow, while ensuring that code quality was not compromised. We more closely examine some of the issues that resulted in stagnant code flow, discuss some of lessons gleaned from a test perspective and focus on the new strategies, processes and tools we are developing to make integration testing more efficient and effective. The paper uses examples and data from this “fearless FI” initiative to illustrate and emphasize key points.

Jean Hartmann is currently, a Principal Test Architect in Microsoft’s Developer Division with previous experience as Test Architect for Internet Explorer where his main responsibility is driving the concept of software quality throughout the product development lifecycle. Prior to Microsoft, Jean spent twelve years at Siemens Corporate Research as Manager for Software Quality. He earned a Ph.D. in Computer Science in 1993 while researching the topic of selective regression test strategies.

The aspiration of any software company or organization is the delivery of software products within defined goals of scope and constraints of schedule and resources. The competitive nature of business requires that organizations live up to these goals, and work within these constraints more effectively and efficiently over product lifecycle generations.

What constitutes improving over time for a software QA organization varies depending on perspective:

• For a tester it may mean more quickly differentiating between improper and expected behavior.
• For a customer support analyst it may mean less issues found in the field.
• For a test engineer it may mean optimizing test coverage based on risk assessment.
• For a quality engineer it may mean identifying issues in earlier lifecycle phases.
• For a QA manager it may mean making sure a quality/test team has the proper knowledge and tools in time to validate new technologies.
• For a QA director it may mean doing more with less.

The ability to consolidate so many perspectives into a comprehensive evaluation of improvement at the organizational level is a complex challenge. A QA organization needs to regularly ask itself these questions:

1. How do we know we are doing the right thing?
2. How do we know we are improving over time?

Realizing quantitatively whether your software QA organization lives up to the dynamic needs of your business is not as simple as looking at defect trends over time. An appropriate evaluation involves various aspects of the software QA function, including operational behaviors, talent, customers and budgets.

This paper chronicles an approach taken by one software QA organization to evaluate its own health through establishing goals, benchmarking, defining organization-level metrics, and on-going self-assessment.

Michael L. Hoffman is a 20-year veteran of the software development/QA industry. He has filled a variety of engineering, management, and strategist roles at companies such as Jeppesen, Hewlett-Packard, Insight Distribution Software, Sharp, and GTE. Michael’s experience includes the areas of communications systems, factory control/integration, embedded systems, PC applications, and web-based solutions. Additionally, Michael teaches various software courses for the Oregon Institute of Technology and is Chair of Oregon Tech’s software engineering industry advisory board.

The overall goal of this practicum project was to provide advice and guidance to distributed software teams, and to offer suggestions for further work in this field. The primary objective was to specify and evaluate selected software engineering processes adapted to support collaborative software teams. A secondary goal was to evaluate the processes using open source software or free-ware tools. The project focus was explicitly aimed at processes over tools – the software tools being the means rather than the ends for learning about such team collaboration. Although this experience report focuses on how the project was conducted, it also summarizes the principle results.

The practicum team consisted of eight students split into two sub-teams with slightly different distribution characteristics, as well as distinct, but related, responsibilities. Similar team-of-teams structures, of varying sizes, are not uncommonly found in practice. This thereby offered a unique opportunity to realistically emulate day-to-day activities, such as project management meetings, specification document reviews and code inspections, while performing the various stages of the software engineering lifecycle.

The project started with a clean slate, without limitations as to which processes would be explored and defined or which tools would be used. The project team was tasked to develop a project specification and development plan to meet the practicum objectives and to utilize tools to shape and evaluate the fruit of their work.

The emergent and evolutionary nature of the project informed the targeted results. More specifically, the students experimented with different collaboration processes and tools during the specification and planning phase of their work, which fed the implementation phase. This focused their attention on some of the distributed team collaboration building blocks – both base processes and representative foundation tools. Some of these were rejected, while others were kept and incorporated into their emergent processes. In other words, the students “ate their own dog food” in an evolutionary fashion as they progressively evolved towards their final work products.

Raleigh Ledet is a Software Engineer with Apple, Inc. (ledet@apple.com). Raleigh was previously with Wacom Technology Corporation (Vancouver WA). Raleigh operates a small shareware company called Mage Software and was formally a board member and web developer of the non-profit organization DOGPAW (Vancouver, WA).

Kal Toth is Executive Director of the Oregon Master of Software Engineering Program (OMSE) and Associate Professor in the Maseeh College of Engineering and Computer Science, PSU. Kal was formerly with Hughes Aircraft of Canada, CGI Group, Intellitech, National Defence Canada, and Datalink Systems Corp.

We are going to present a case study on performance testing of the Microsoft Office Communicator Mobile application for Windows Mobile 6.x phones. We will discuss our approach to performance testing, the complexity of automating performance tests and how architectural improvements have contributed to improved application performance and better resource usage. Our automated tests measure 6 unique performance metrics for 20 different scenarios. We will also provide details on the performance metrics and tools used to gather and present this data – and how we’ve used this data to make important decisions throughout the product cycle.

Rama Krishna has been working at Microsoft as a Software Development Engineer in Test for five years. He has worked in the Windows Networking division and recently in the Unified Communications group.

Prior to Microsoft, Rama Krishna worked as a Software Developer at Cisco Systems for four years. He earned a Master’s degree in Computer Science in 2001.

This experience report relates parables of mistakes I made and what I learned that you should apply to avoid them. These lessons learned are also cross-indexed as examples for parts of the book: Lessons Learned in Software Testing (Kaner, Bach, and Pettichord).

Verify Stress tests out of band.
Test Oracle size <= System Under Test size.
Test in the presence of failures.
You must understand the goals of your release.
Under testing sequence and state.
Skipping coverage of the hard stuff
Over and under testing combinations.

The many times I’ve taught various aspects of software testing, I find many students learn best by hearing stories that demonstrate a principle in action. I’ve been testing for so many years, I’ve gathered a number of scars that I attempt to help newer converts to the discipline avoid. Many of my lessons learned occurred before a book called Lessons Learned in Software Testing (Kaner 2001) was ever published, but in reading the book I could see how my experiences related to those in the book and correlate them as lesson #. The majority of the lessons are from my mistakes around over testing!

You can test too much in one test case causing unnecessary blocking due to bugs. You can test too much for a release by exceeding its requirements. You can verify too much during stress testing resulting in less stress testing or overly expensive stress testing.

Testing is a business activity with a cost. Using your resources wisely to most effectively cover the risks requires:

• Understanding the product and release goals,
• Dealing with common issues like known errors,
• Anticipating likely errors
• Choosing the right evaluation methods

Keith Stobie is a Principal Software Development Engineer in Test working on Bing at Microsoft. He plans, designs, and reviews software architecture and tests. Previously, he was working on the Protocol Engineering team model based testing (MBT), including test framework, harnessing and model patterns, and the XML Web Services group. With twenty-five years of distributed systems testing experience, Keith’s interests are in testing methodology, tools technology and quality process. Keith has a BS in computer science from Cornell University; is an ASQ Certified Software Quality Engineer, ASTQB Foundation Level; and a member of ACM, IEEE and ASQ.

This paper presentation will describe:

• How my robot squads help with testing
• Why I use robots rather than automated scripts
• What are my robots’ characteristics and attributes
• How I define my robots’ job functions
• How I make the robotic team productive
• How to share robots with others

Jerry Yang is a Senior Software QA Engineer with McAfee, where his main focus is on test design for complex enterprise product compatibility testing and performance testing. He has over 15 years of experience in the Software industry, including clients such as Intel, ADC, Kentrox, and McAfee.

In this paper, we will describe how a solid stress framework, complete with integrated structured randomization and methodic meddling of temporal properties makes practical software quality assurance possible. Specifically, we discuss the methods and practices applied to provide solid assurance to a critical commercial component – the native Concurrency Runtime stack from Microsoft. First, by applying random distributions in individual tests and integrating such individual tests via a statistically fair scheduler, we describe how to cope with traversing the seemingly infinite interaction patterns. Second, we will expose how such testing helps identifying hangs stemming from deadlocks and livelocks. Thirdly, we will talk about methodically injecting randomization into the temporal properties of the software system, and how that can be used to assure us to find bugs with reasonable probabilistic expectation. We will conclude with a brief survey of the effectiveness of our stochastic stress framework over other tools.

Atilla Gunal is a Software Development Engineer in Test in the Concurrency Runtime team at Microsoft. He has a BS degree in Mathematics Engineering and MS in System Analysis.

Rahul Patil is a senior QA lead for Microsoft’s new native concurrency runtime technologies. As a QA lead in charge of a brand new concurrency platform, Rahul has had to address reporting quality on new risks introduced by non-determinism inherent in concurrent software. He also holds a Master’s degree in Software Engineering and has been working at Microsoft for the past 6 years testing various SDKs and platforms.

Liu Hong is a Lead SDET in Microsoft Platform and Tools division at Microsoft Cooperation. She has worked at Microsoft since May 2006. She holds a Ph.D. degree in Civil Engineering from Clemson University. Before her career at Microsoft, she first worked at Aon Cooperation as a research engineer, and afterwards as software developer in various companies such as Infomove and Applied Inference in the Puget Sound area and, immediately before she joined Microsoft, as an independent contractor for Gray Hills Solutions.

Maxim Markin is a Software Design Engineer in Test at Microsoft Cooperation.

In this paper, we share our experiences in shipping a set of web-services that use large volumes of data and Artificial-Intelligence algorithms to translate text between human languages. These web-services are part of Microsoft’s Machine Translation products, shipped by an incubation team under Microsoft Research.

The areas of performance and stress testing yield bugs that present unique challenges in their findings, investigations, fixing and regressions. In the early days of testing services, a lot of the necessary infrastructure and tools were written by the individual teams geared towards perceived unique characteristics of each service. Over time, these have evolved into now publically available automation libraries that handle a lot of the commonly performed functions, such as results summarization, performance monitoring, etc. While these tools are beneficial and add value to a test team’s quality coverage, it is essential to use them in a targeted manner, with the priority being measurement of quality metrics and the tools being a means to obtain them. Using the MSR-Machine Translation system as a case-study, we attempt to present a test approach based on real-world scenarios and practical solutions that have helped ship a set of web-services that receives traffic of millions of users a day, with continuously improving quality and performance.

A big lesson from our experiences has been the value of ‘Keeping It Simple’, regardless of the complexity of the systems under test. A proper understanding of user scenarios to help correctly prioritize test cases, combined with focused automation, helps achieve the goals of software quality and agile development schedules, leading to successful products and passionate users.

Anand Chakravarty has been working as a Software Design Engineer in Test at Microsoft for close to 10 years, a vast majority of that time in testing server applications. For the past 4 years, he has been part of an incubation team under Microsoft Research, which has shipped products such as the one available at this link: http://www.microsofttranslator.com/BV.aspx?ref=BVNav&from=&to=zh-CHS&a=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FPortland%2C_Oregon

We believe it is essential that companies proactively establish a baseline pedigree for their software via a forensic code audit. The successful completion of a forensic code audit represents a moment in time where all known third-party materials are appropriately catalogued, and risks associated with those materials are fully understood by the company’s relevant business and legal stakeholders. But a one-time pedigree analysis alone is not sufficient to prevent downstream problems. A forensic code audit should be part of a comprehensive quality pedigree program that includes a set of well-defined prophylactic policies and procedures surrounding the use of third-party materials. These policies and procedures take into account the entire software lifecycle, including any customer support obligations that may remain once a program is deprecated. A company that proactively implements a quality pedigree program is better positioned to respond to customer requests, react to lawsuits or potential licensing problems, or to justify a particular valuation of their intellectual property in the context of a merger or acquisition.

Building upon the authors’ 2009 presentation, we explore the practical mechanics of a forensic code audit, and discuss the other policies and procedures that can be used to manage a quality pedigree program as a part of your overall software quality plan.

Susan Courtney is a forensic software analyst & has worked as a consultant for Johnson-Laird, Inc. She has done forensic software analysis for patent and copyright cases, performed electronic evidence analysis and assisted with data preservation and discovery. She is also the owner of SLC Software LLC, a software consulting firm that provides a variety of services including business systems analysis, quality assurance & project management.

Barbara Frederiksen-Cross is the Senior Managing Consultant for Johnson-Laird, Inc. in Portland, OR. She is a forensic software analyst specializing in the analysis of computer-based evidence for copyright, patent and trade secret litigation. She began her career as a computer programmer in 1974 and began working as a forensic software analyst in 1987. She was appointed as Court Data System Advisor to the Honorable Marvin J. Garbis (US District Court, District of Maryland) in Dec. 2000, and has provided forensic analysis services and assisted with data preservation and discovery services on many cases.

Marc Visnick is a forensic software analyst and attorney based in Portland, OR, and a senior consultant with Johnson-Laird, Inc. He specializes in forensic software analysis for patent, copyright and trade secret litigation, as well as software due diligence, independent development project design and supervision, and electronic evidence preservation, recovery and analysis. Over the past 6 years he has participated in hundreds of forensic audits for software-related mergers, acquisitions and source licensing transactions. He is Chair-elect of the Oregon State Bar Computer and Internet Law Section.

Pivot will change the way people perceive and explore the information that surrounds them by visualizing information in new ways, exposing hidden relationships and making it easier to act on these newly discovered insights. Pivot makes it easier to interact with massive amounts of data in ways that are powerful, informative and fun. Please view the following 2010 TED talk which allows you to see Pivot running and a description from Gary Flake, the founder of Live Labs: www.ted.com/talks/gary_flake_is_pivot_a_turning_point_for_web_exploration.
To learn more about Pivot and download the released rich client, visit www.getpivot.com

Pivot is a tool you can use in your business immediately.

• Pivot shipped as a Customer Technology Preview in November of 2009 at the PDC (Professional Developers Conference). The rich client application (WPF) was our unveiling to the world of our faceted search experience. We have announced a Silverlight control that will be available in the summer of 2010. This has spurred a quite a bit of interest internal and external to Microsoft.
• No extra work is required to change your business data.
  • Pivot uses a common format, XML, to describe collections of data and Deep Zoom image format to allow for smooth zooming/panning of the information. The format extension is CXML. In order to view your data in Pivot today, you can select a subset of information and use our Excel spreadsheet plug-in which creates the CXML and Deep Zoom image format from your data and imagery. Translating to this format is simple, which makes the Pivot experience easy to obtain.
• Pivot brings two capabilities to (cached) data:
  • Search: Pivot allows for an indexed Search across whatever content you are viewing.
  • Faceted browsing: Using the data as the user interface, Pivot allows you to easily swim throughout your information. One of the most powerful aspects of Pivot is the exposure of common properties (or facets) within a set of data, which then allow you to slice and dice the information leading to insights and answer tough business questions.
• Test data is part of business data, so
  • Pivot can improve test decision making through:
    • Visualizing trends: Pivot allows the tester or manager to gain insight into the data that you collect. Testers collect and interpret a lot of data!
    • [Unexpected] events: Pivot will highlight events that don’t match the expected distribution curve. In this way it’s especially useful to help identify and then act on information that may be otherwise hidden
  • Increasing the probability of divining insight from data:
    • Insights into data provide one of the most valuable resources to testers – pointing us where to focus our time and energy.
  • Interactive visualization can be a team sport which:
    • Raises the bar for individual testers, test managers, and product owners. Testers can use Pivot to analyze test data. Managers can use Pivot to analyze the release. Product owners can use collections created by Test to truly understand the information that the test team is using to gauge quality.
• Examples – ways we are incorporating Pivot in test
  • Product Studio collection. We can view our entire bug database used to develop the Pivot CTP within Pivot!
  • Code Coverage. Using a code complexity calculation and Magellan code coverage data, we can view Seadragon code coverage statistics and learn where to add more coverage!
  • Test run results. We have a collection created from multiple test runs which allow you to see at a glance what areas are passing/failing and what common properties the failures have in order to investigate the root cause!
• Involvement
  • Contribute your Pivot innovations to http://getsatisfaction.com/live_labs_pivot

Max Slade is a Principal Test Manager at Microsoft in Live Labs. He has worked in the field of software testing, calibration and measurement science for the past twenty years in a variety of teams from start up to established businesses. The challenges that he is most drawn to are incubations where understanding the business needs for quality, building teams, processes and infrastructure are paramount.

These obstacles need not be a hindrance to the testing process. Learn what the common stereotypes are and how to overcome pre-conceived notions, devise efficient work practices and create a more cohesive testing team across the globe.

Katherine Alexander has been working as a Test Engineer since 2006. She is currently at Vertex Business Services. Her focus at Vertex is on custom projects; collaborating with the client to discover solutions that meet their business needs.

This paper describes how the SharePoint 2010 team solved this problem by creating a tool capable of sampling real world usage data and building a model capable of describing and building the load test necessary to simulate real-world traffic patterns. The tool was designed to build load tests that run using a shipped web-testing product, Microsoft Visual Studio Test System 2008, thus making the tests and data portable to any other environment where this product is available. The solution allowed us to incorporate new load patterns relatively quickly, both from in-house deployments and customer deployments of our server product. Adding such workloads to our test suites had previously been so expensive and difficult as to be non-feasible. With the new solution in place bug discovery rates increased as did fix ratios, as we knew that the load tests were based on expected traffic patterns. We were able to add pre-production validation testing to our internal deployment processes, providing costing and stability predictions, something that was not possible before. Finally, the test tools were released to market to allow customers to do their own capacity planning and management via a load test kit able to adapt to their own workload needs.

Wayne Roseberry has 20 years experience working in the software industry. He is currently employed as a software test lead at Microsoft. Projects at Microsoft include MSN 1.0 – 2.0, Site Server and all releases of SharePoint from version 1 through 2010.

At the end of the test cycle we had a focused initiative to increase the block code coverage from 91% to 100% in order to measure the return on investment (ROI) of such an effort.

We calculated the ROI based on the number and importance of the issues found, the time invested, the increase in code coverage and bug yield in comparison with other test activities. Our results showed that it’s neither prohibitively expensive to achieve effective-100% code coverage nor did we find an exponential increase in the number of bugs with the higher code coverage, as suggested in some papers in the field. Although we did find some systemic holes in our test bed, they did not uncover any major issues in the product. In this paper we discuss what parts of our development process (such as the propensity of testers to develop more complex scenarios first) may have contributed to this lack of bugs.

Cristina Manu got her PhD in Mathematics in 2002 at the University of Bucharest. Currently she is a Software Development Engineer in Test at Microsoft working in the Parallel Computing group

Pooja Nagpal is a Software Development Engineer in Test for the Parallel Computing group at Microsoft.

Donny Amalo is a Software Development Engineer in Test for the Parallel Computing group at Microsoft.

Roy Patrick Tan is a Software Development Engineer in Test at Microsoft. He got his PhD in Computer Science from Virginia Tech in 2007.

This paper will present how a test team at Microsoft used code reviews as a method to improve code quality and more importantly as a learning process for the entire test team. The paper will also discuss how smart and consistent application of lightweight root cause analysis and the creation of code review checklists led the path to success – and how any team can use these principles to reach these same levels of success.

Alan Page began his career as a tester in 1993. He joined Microsoft in 1995, and is currently a Principal SDET on the Office Communicator team. In his career at Microsoft, Alan has worked on various versions of Windows, Internet Explorer and Windows CE, and has functioned as Microsoft’s Director of Test Excellence. Alan writes about testing on his blog (http://angryweasel.com/blog), is the lead author on How We Test Software at Microsoft (Microsoft Press, 2008) and contributed a chapter to Beautiful Testing (O’Reilly, 2009).

We will introduce the Simple Language for Scenarios (SLS). In SLS, we constrained users into writing test cases in a particular text-based format (not XML based). This reduced coding-style flexibility, but gave us a single paradigm of writing test cases and a point of injection via the interpreter. We used this point to achieve several goals – we were able to automatically marshal calls between different thread models, adapt existing cases for API parameter testing and enable security fuzzing without requiring each tester to have to write code to do so. For string injection we wrote an API string testing class where the script engine could automatically call each method in a loop and pass in all possible parameters within the context of an existing case simply by annotating the “start” and “end” points in the test case where the engine needed to loop. Additionally, with API testing, we were able to run a case to a particular point, and then bring up a UI to allow ad hoc testing. The steps executed by the tester in the UI would then be written back as script language that could then be incorporated into the test case library.

Ultimately, we landed with a hybrid model that managed to leverage the strengths of both approaches. Along the journey, we faced several facets to consider including development time for a test case, reducing cost of failure analysis, adaptability/extensibility of a test case to new environments and more. This paper will go into the technical details of our findings and how other software projects could benefit from our lessons-learned to more effectively deploy automated test cases.

Sam Bedekar is a Test Manager at Microsoft. He has worked in the Unified Communications space for nine years. Sam is very passionate about test and the impact it can have on product quality and the industry. Sam believes that Test Engineering has vast potential to grow and is excited about leading efforts to bring the state of the art forward.

This paper describes how the data collected by a test group at Hewlett-Packard provides decision support to the program management team, insight into test productivity and improvement opportunities for the software development teams. The data mining goes beyond the normal collection of defect patterns during product development and includes test execution productivity, defects found per hundred test cases, defect aging patterns and code volatility. The data has provided insight into performance of the test teams, a comparison of effectiveness between manual and automation testing and supported the natural rhythmic observations of product development. An organization may choose to limit the use of test data to only provide release decisions but in reality, the data has rich information to guide improvements to the overall software development lifecycle.

Bhushan Gupta has twenty five years of experience in software engineering, fifteen of which have been in the software industry. Bhushan recently joined Global Store Solutions group at NIKE Inc. as a Software Quality Lead after a thirteen year tenure at Hewlett-Packard. He joined HP as a software quality engineer in 1997 where he was responsible for identifying key process areas to reduce rework. Based upon Hewlett-Packard quality parameters he developed quality goals, quality plans, and software validation strategies for several products. As a software process architect at HP, he customized the agile lifecycle for different groups and developed productivity metrics. His latest title at HP was “New Production Introduction Quality Lead” which included monitoring the quality of the new product under development.

Bhushan Gupta has a MS degree in Computer Science from New Mexico Institute of Mining and Technology, Socorro, New Mexico, 1985.

This paper summarizes my experience working with several outsourced testing teams. Following a brief overview of the differences between onshore and offshore testing, the paper presents the strength of traditional offshore test teams and how to utilize them to improve test efficiency and increase development capacity; as well as the challenges and risks that test managers will confront when using outsourcing testing. Finally, based on real world experiences, providing several specific suggestions, strategies and examples of how to manage test team composed by both onshore and offshore engineers.

This presentation offers test managers an ability to use outsourced testing resources to increase development capacity efficiently while keep development cost at a management level.

Hao Zhao has over 20 years experience in the software field. He is currently a QA Manager at Expedia.

Many times as software test engineers we are brought on to the project after the requirements have been gathered. This gives us very little visibility into the early discussions on what the user needs to accomplish with an application solution.

Therefore we test and certify the application based on the functional documentation provided to us without really understanding how the business is intending to use the application in production.

Methods/procedure/approach:
As part of our work to improve the software testing process for our group (LCAIT), we have been looking into ways to move upstream in the process and ensure that we add business value by testing our applications based on user scenarios that reflect the usage of the application.

This required us to change our approach in writing test cases and change our focus from the traditional functional testing to user scenario focused testing. To achieve that objective we used visual representations of possible user actions and application responses that align with the user scenarios. Using visual representations enables us to easily identify gaps in the functional requirements and allows other disciplines to more easily review the test cases.

This paper will describe how we break out our functionality into individual test cases and use these as building blocks to test the end-to-end user scenarios. It will also describe how we extract information from the flowcharts to be used during manual and automated testing.

April Ritscher has been a senior test engineer at Microsoft for the last 10 years. As test lead, she has worked on a variety of different projects including a project for Bill Gates. She has received several awards including 5 Ship-it awards and 3 IT Pro awards. She has worked on several global teams with both India and China. She is very passionate about quality and is always looking for ways to improve the software testing process.

TeKila is an aggregation of several Open Source powers – Google Web Driver, HTML Unit, Fitnesse, TestNG and Selenium. It offers a tool kit to test Web Application at different levels and in modes.

Often, the search for a silver bullet tool for automating ends in a compromise. In demanding times when everything is changing rapidly, speed and flexibility cannot be compromised.

The ever-rising burden of releasing features fast, with high levels of quality, punctuality and performance, demands automation. The task of writing a test automation framework, which does more than UI testing for rapidly growing web-based applications, is challenging. Many find it tormenting, some attempt it and only a few succeed.

A bridal suit has to be stitched; ones in the market would just not fit. Similarly, along with creative ideas and innovative approaches, engines of automation frameworks for projects need to be powered with multiple cylinders. There are numerous tools available for use; this introduces a new paradox – the paradox of choice – what to choose, from where to choose, how to choose, etc are questions everyone needs answers for.

A few of the many challenges are:

• Several Web Technologies being used (HTML, Java Script, AJAX, Flex)
• Multiple Browsers
• Multiple Operating Systems
• Multiple stages of testing

In our attempt of combating automation we came up with TeKila. TeKila is an aggregation of the best of various Open Source Powers enabling us to do:

• UI testing on multiple browsers and Operating Systems
• Business Logic layer testing using APIs
• Flex based UI testing
• Profiling data (like page load time, first/last request, http timing, status, etc)
• Integration with continuous integration build system with different Test Suits classes

What did we achieve through TeKila:

• Structured testing effort
• Multiple team members to contribute in automation
• Leverage work done by others

Separation of concerns:

• Test code
• Web Page Objects
• Suite Formations
• Loose Test-Tool Coupling

Nikhil Bhandari holds an Engineering degree along with 9.5+ years of experience as a Lead Software Engineer – QA. He is currently working with Intuit, and previously worked with companies like Oracle, McAfee & Satyam Computers in Bangalore, India. He has utilized various testing tools and scripting languages for developing test automation frameworks for both desktop and online applications on various platforms. He has been a speaker at STARWEST Conference 2008 (USA), Free-Test Conference 2009, 2010 (Norway), Step-In Forum Evening Talk (India).

Kapil Bhalla has a Master’s Degree in Computer Applications from NIT Karnataka, India and has been working with Intuit India for the last 2+ years. Over the past year he played a key role in developing Open Source Automation Testing Framework.

So, what does this mean for the quality professional of today?

Managing complexity means recognizing differing and critical needs, how to experience the upsides of each need, avoid the downsides, and do it all intentionally, with awareness of the choices being made. The skills to accomplish this, as covered in the paper, include the ability to:

• Recognize system dynamics at play, called polarities, that may be getting in the way of your quality goals
• Manage these polarities through a deceptively simple process; and
• Apply this process as a communication and conflict resolution tool to create dialogue about what is important and why.

This paper introduces the process of polarity management as a way of seeing, thinking and communicating around opposing dynamics to help you become (or remain) a proactive player in support of your software quality efforts.

Denise Holmes, Principal of Edge-Leadership Consulting in Portland has a background in designing and delivering experiential leadership development, coaching individuals and teams through change and applying system dynamics to her consulting. Denise built her expert facilitation and coaching skills in the worlds of health care and high tech. Denise holds an M.A. in applied behavioral science from the Leadership Institute of Seattle. She also holds an M.B.A. from Marylhurst University and a B.A. in international studies from the University of Oregon.

The Acceptance Test Driven Development methodology elevates TDD concepts one level higher and provides us with a quick solution for testing integration with SCRUM. Automated acceptance test cases are designed and developed in parallel, or even before the features. ATDD improves both development and testing by enhancing focus on users’ requirements. It also provides a better use of testing resource time, including optimization of tester ramp up. Last but not least, it’s really fun!

I would like to share with you the experience of our SCRUM team, which consists of developers and testers, working together on a big software project for a year and a half. In this complex environment, after fifteen sprints, we realized that some change is needed and added the ATDD methodology.

I will show you (by graphs) the increase in tests and bugs by using automated acceptance tests, and the improvement in the team’s satisfaction after the change. I will also show the importance of combining manual tests with automatic acceptance tests, taking into account the automation limitations. The ATDD methodology solved some of our key problems and helped us improve our project quality and our SCRUM process. It may help you too.

Sari Kroizer has a formal education in software engineering. Sari’s experience includes 1 year of programming, web development as outsourcing;1 year of FW validation for REDLAN; and 3 years of FW/SW validation for Intel, in part 1.5 years as a SW validation team lead.

In the HP printer business, lean integration is:

• Reducing cost by driving synchronization of delivery across technology components.
• Improving the customer’s experience by making customer workflows functional and visible early and often throughout the lifecycle.
• Making complex programs easier to manage by providing visibility into what the product can and cannot do.

This paper will include an introduction to Lean software and systems development and a reference list. Lean systems development is a superset of agile software development. Lean is able to handle situations that the most commonly known agile methods do not address, including large, complex and partial waterfall systems, by applying methods deriving from queuing theory and statistics.

The paper will then demonstrate the methods with progress reports and/or results from actual projects in the HP Inkjet and LaserJet businesses. There is enough data as of this writing to demonstrate what we are doing, why and how well it is (or isn’t) working. The paper will show the management methods utilizing cumulative flow diagrams and the power of this method. Higher quality and productivity will be demonstrated with examples and anecdotes.

I’m very excited about the Lean methods – in my opinion, Lean is the next big thing in software engineering – and I hope to be able to share it with the Pacific Northwest Software Quality Conference audience.

Kathy Iberle is a senior system quality engineer at Hewlett-Packard, currently working at the HP site in Boise, Idaho. Over the past 25 years, she has been involved in software and system development and testing for products ranging from medical test result management systems to inkjets and laserjets to Internet applications. Kathy has worked extensively on training new test engineers, researching appropriate development and test methodologies for different situations and developing processes for effective and efficient software testing.

Kathy has an M.S. in Computer Science from the University of Washington and an excessive collection of degrees in Chemistry from the University of Washington and the University of Michigan.

However, story-driven development brings unique challenges for software testing. As stories are implemented, relying on each other, a complex software system emerges capable of doing more than just what the stories told. With the addition of each story, testing surface and capabilities (and defects) of the software grow exponentially.

Usually, different sub-teams develop different parts of the software, making it very difficult to manage and understand the complete set of capabilities of the software. As a result, at the end of each sprint the software product consists of: (1) features told with developed stories and (2) features/defects emerging from untold stories, coming from interactions of the implemented stories. Sometimes these features are as designed, but often hiding significant integration bugs.

This paper describes various techniques developed and used by the Visual Studio Architecture Tools team over a few releases, through experiments in testing within agile development. It also presents some case studies to illustrate and emphasize key points, ranging from using architecture diagrams as part of test planning to particular test coverage along with test spectrum range. The guidance presented focuses on a set of best practices that teams can adopt to accomplish optimal test coverage over the test spectrum.

Erbil Yilmaz has over 7 years of software testing experience with Microsoft working on Architecture Tools within Visual Studio Team System, owner for security and performance exit criteria. He is working as part of the Test Architecture Group within VS Architecture Tools driving integration testing with agile development, effective test planning and performance testing strategies. He holds an M.S. in Computer Science specializing in Computer Security. He also has B.S. degrees in Computer Engineering and Mathematics.

In this complex world, how do we define quality? We define quality goals with an ever-increasing number of “ilities”. We perform all kinds of testing: acceptance, performance, functional, concurrency, stress, exploratory, stability, build verification and, finally, regression testing.

Given all of this complexity, it can be difficult as a quality team to make decisions:

• How should we define quality?
• Where should we invest our time and money to best assure quality?
• How do we know when we are done?
• Where should we improve our processes?

The answers to these questions lie with our ultimate quality consultants – our customers.

One of our values at Intuit is “Customers Define Quality”, and this paper will share how we interact with our customers at every stage of the lifecycle to understand their definition of quality.

The paper will describe how we include the customer in our requirements definition phase to understand the most important customer problems, generate ideas to solve those problems and to test these hypotheses instead of just relying on the opinions of HiPPOs (Highly Paid Persons with Opinions). Since errors and defects injected during the requirements phase can be the most expensive errors, investing in quality at this stage is especially productive.

We use several customer driven processes during the design and coding phases. Design for Delight goes hand-in-hand with the requirements formation & testing process, with the goal of ensuring that we build the product that solves important problems for our customers. We also introduce the customer advocacy role, usually part of the Quality Assurance team. The paper will also describe how we build customer empathy in every developer.

During the testing phase, our customers help us prioritize our testing, both manual and automated. Our customers also help us understand how they use our products, which helps guide our testing process. Finally, we involve our customers in the testing phase.

After release, the support and feedback processes allow us to react very quickly if there are customer issues, sometimes when only one customer affected. The customer feedback also helps calibrate our development and test processes and provide an even deeper understanding of our customers.

Customer Driven Quality helps focus our quality assurance activities on our most important stakeholder, our customers.

John has been developing software in a variety of roles for 24 years. He has held positions ranging from development and test engineer to project, development and quality manager. He has experience in Aerospace, Telecommunications and Consumer software industries. Currently, he is the Quality Leader for QuickBooks Online, a SAAS offering from Intuit, Inc. He has earned BS in Computer and Electrical Engineering, MS in Computer Science and MBA.

Testing geo-spatial data and route planning algorithms is a good example for this; the inherent complexity of geo-spatial data, the representation of the data and also novel algorithms that utilize this data make testing more of a daunting (or fun?) task at both levels: understanding and evaluating. This paper presents a collective testing approach that we have developed at Bing Maps to attack the complexity of our route planning product focusing on the following areas:

• Heuristic-based tests to identify defects that are otherwise hard to find,
• Efficient test infrastructure to promote agility,
• Intuitive visualization solutions for complex geo-spatial entities and scenarios and, finally,
• Integration of complex statistical analysis techniques with the testing process.

This approach has provided improved coverage in our testing and enabled us to identify more quality issues during development. We also benefited from a more efficient evaluation mechanism for complex issues found during testing. As we head towards providing far more geo-spatial coverage and enhanced features to our customers, our testing approach will have to adapt and evolve accordingly; this paper presents some pointers in this area as well.

Engin Uzuncaova is a software test engineer with the Bing Maps Directions team at Microsoft, Redmond, Washington since October 2008. He received his Ph.D. degree from the University of Texas at Austin in December, 2008. His research centers around developing efficient techniques for specification-based test generation.

While most agree the certification is important, what does it really take to pass the exam? Does one have to have a photographic memory, an affinity for interpreting a book that reads like a dictionary or be a mathematics wizard? Join Balbinder K. Banga as she aims to dispel the myths surrounding what it takes to pass the exam. The presentation will focus on the top three “must haves” for passing the PMP exam.

Balbinder has coached hundreds of students to successfully pass the exam and will share three common findings that she has discovered to be consistent among successful test takers.

Balbinder K. Banga, PMP is the founder of Glacier Group Training, a company whose
mission is to prepare students to successfully pass the Project Management Institute’s PMP certification exam. Through the years, Balbinder has developed an extensive knowledge base on the PMBOK and the PMP exam. She has written a PMP Exam Prep Booklet and taught her PMP exam prep class at Boise State University, Portland State University, University of Northern BC, Kansas University and Simon Fraser University. Balbinder also teaches the class onsite at corporations in the Portland area.

Balbinder is a PMP and a Microsoft Certified Professional. She has a Master’s degree in Engineering Management with a focus on Project Management. She volunteers with PMI and is the Director of Academic Outreach for the Portland PMI Chapter.

This webinar will answer these questions and show you how to strategize your testing for the Cloud. It will also discuss:

• What are the fundamentals of testing in the Cloud?
• What are the characteristics that need to be tested?
• How would you conduct application functional testing, performance testing, and security testing, in the Cloud?

Paul Trompeter has a broad range of experience in the computer industry at a variety of companies including Litton, Motorola, Hewlett-Packard (Tandem Computers), Network Appliance, SDT, WellPoint, Technicolor, GDI InfoTech, and Headstrong. Prior to moving into the testing arena, he held positions as a development programmer and manager, a project manager, and a program manager. For the last twenty years, however, Paul has been directing enterprise-wide and global SQA and test efforts spanning many industries, including aerospace, manufacturing, entertainment, financial, and healthcare. Paul holds degrees in math, technology management, and organizational change. He has published journal articles, and participated in pre-publication reviews for several books. Paul has also successfully conducted business in many counties across the globe including those in Asia, South America, Central America, and Europe.