Built to be Fair? Bias, Verification and Validation, and the Future of AI

As artificial intelligence (AI) and machine learning (ML) technologies are increasingly integrated into healthcare - from diagnostic support tools and patient risk stratification to hospital resource management - the consequences of bias in these systems carry profound implications for patient safety, equity, and trust. This paper examines the critical role of verification and validation (V&V) in ensuring that AI/ML systems used in medical and hospital contexts are not only clinically effective but also fair and accountable. We begin by unpacking the multifaceted nature of bias in healthcare AI, including disparities in training data, algorithmic assumptions, and deployment environments that can disproportionately affect underserved or marginalized patient populations. We then explore how V&V practices - such as dataset audits, bias-aware validation protocols, fairness testing frameworks, and post-deployment monitoring - can be leveraged to detect and mitigate these risks. Through case studies and recent research in medical AI, we demonstrate how rigorous V&V processes are essential for aligning AI systems with ethical principles, regulatory standards, and clinical expectations.

Paper | Presentation

Nancy (Wei) McCormack

I'm Nancy McCormack, a Principal AI/ML Verification and Validation Engineer at Mayo Clinic, where I've been working for nearly four years to help shape the future of healthcare technology. In this role, I collaborate closely with clinicians and cross-functional teams to develop AI/ML verification and validation processes, standards, templates, and guidelines. Additionally, I work with testing leads from other organizations to calibrate practices, build common learning paths, and establish unified processes.

With over 16 years of testing experience across diverse industries-including Medical/Healthcare, Semiconductor, Networking, and IT-I bring a broad perspective to my work. I also have 8 years of engineering management experience, during which I've led global testing and automation teams across multiple time zones. My management expertise includes driving project timelines, coordinating cross-functional teams, and ensuring alignment across stakeholders, all while fostering a collaborative environment.

Mentoring is one of my greatest passions, whether I'm guiding other engineers, supporting early-career professionals, or working with summer interns to help them grow in their careers.

I hold a bachelor's degree in electrical engineering and a master's degree in computer science, providing me with a strong foundation in both hardware systems and software development.