If your company has developed software before, then you know it’s never as simple as writing some code and put it out in production. Regardless of who the software is for (clients, employees, third parties), doing proper Quality Assurance (QA) is a must. Otherwise, you would be blind to the software’s limitations and may even deliver broken or totally unusable products.
Quality Assurance and QA outsourcing stand today as a core process of any software development project. The design, build, test, and deploy process needs to be done right and in that order to achieve success. As such, QA engineers work all throughout the software development life cycle using agile methodologies and testing all progress in small and iterative increments, making sure that the product always responds to the appropriate goals.
One would expect Artificial Intelligence projects to implement QA just like this. However, that’s rarely the case. While the standard iterative 4-stage process is maintained for the most part, AI-driven operations can simply be put into production. Why? Because of the inherent nature of AI: it’s constantly learning, it’s constantly evolving and, therefore, it needs continuous management.
That means that you don’t do QA for AI projects the same way you would do QA for any other project. Here’s why.
The Role of QA & Testing In AI Projects
By definition, AI needs to be tested over and over again. If you want to develop AI that actually works, you can’t just throw some training data at an algorithm and call it a day. The role of QA & Testing is to verify the “usefulness” of the training data, and whether or not it does the job we are asking from it.
How is this done? Via simple validation techniques. Basically, QA engineers working with AI need to select a portion of the training data to use in the validation stage. Then, they put it through a crafted scenario and measure how the algorithm performs, how the data behaves, and if the AI is returning predictive results accurately and consistently.
If the QA team detects significant errors during the validation process, then the AI goes back into development, just like you would do with any other software development project. After some tweaks here and there, the AI comes back into QA until it delivers the expected results.
But, unlike standard software, this isn’t the end for the QA team. Using some different testing data, QA engineers need to do all of this again for an arbitrary amount of time, which depends on how thorough you want to be or how much time and resources you have at your disposal. And all of this happens before the AI model is set out to production.
This is what most people know as the “training phase” of AI, where the dev team tests the algorithm multiple times for different things. QA, however, is never focused on the actual code or the AI algorithm itself—they need to assume that everything has been implemented as it is supposed to be, and focus on testing if the AI actually does what it is supposed to do.
This approach leaves two main things for QA engineers to work with: the hyperparameter configuration data and the training data. The former is mostly tested through the validation methods we discussed before but it can also include other methods like cross-validation. In fact, any type of AI development project must include validation techniques to determine if the hyperparameter settings are correct. That’s just a given.
After that, all that’s left is testing the training data itself. How do QA engineers do that, though? They can’t simply test the data quality, they also need to test the completeness of the data and ask a bunch of questions to help them measure the results. These are always a good starting point:
- Is the training model designed to accurately represent the reality the algorithm is trying to predict?
- Is there any chance that data-based or human-based biases are influencing the training data in some way?
- Are there any blindspots that explain why some aspects of the algorithm work in training but fail to perform as expected in real-world contexts?
Testing the quality of the training data can generate many more questions like these as the project progresses. Keep in mind that, to answer them accurately, your QA team will need access to representative samples of real-world data and a comprehensive understanding of what AI bias is and how it relates to the ethics of AI.
Artificial Intelligence Needs to Be Tested In Production
All in all, your QA team must know when your AI software is properly validated, when the training data is up to standard, and when the algorithm is proven to deliver the expected results consistently.
However, every AI project will always have a unique way to manage and process data—and, as we all know, data is growing and changing at all times. This is why the QA approach for AI development extends to the production stage.
Once all of the above gets a green light, Quality Assurance begins a new cycle, testing the performance and behavior of the AI as it receives new real-world data. Regardless of the size or complexity of your AI project, what you want is to always keep close tabs on the evolution of your AI. And the best way to do it is through a proper QA process.
Today, this is known as “Machine Learning Operations” or, more succinctly, ML Ops. That involves version control, software management, cybersecurity, iteration processes, and discovery stages in which the QA engineers take care of everything that can happen once the AI is in production. I hope this article helped you expand your perspective on QA and Artificial Intelligence. Best of luck!
If you enjoyed this, be sure to check out our other AI articles.
