Biased AI Is Everywhere. Here’s How to Build a Process That Cuts It Off

An artificial intelligence system asked to judge a beauty pageant picked white contestants over black ones. Another, designed to improve natural language processing, resulted in a system that created analogies like, “man is to woman as computer scientist is to homemaker”. Yet another, meant to recognize images, kept equating kitchens with women.

How can we avoid building biases like these into future machine-learning algorithms? The honest truth is that it’s not going to be easy—at least, not until we can eliminate bias from humans (good luck!). After all, machines are learning about the world we’ve built, which means their biases are really our biases being reflected back at us. But you can help put your product on the right path by proactively and systematically designing your process and your team to root out bias every step of the way.

Here are four practical steps to getting started.

Diversify, diversify, diversify

Let’s be blunt. Tech companies have diversity problems — both in staffing at big players and in funding new startups. Loading up a team with the same type of person has always led to problems with products. (Did you know that cars were historically more dangerous for women, because until 2011, carmakers only used crash-test dummies sized for the average male?) But in tech, that lack of diversity can lead to software biases that extend far beyond a single product. That’s because AI isn’t just getting used once, but as the backbone for all kinds of other products to sit on top of.

Bringing in diverse perspectives and experiences increases the likelihood that you’ll notice a problem. And since research indicates that diverse teams simply perform better, you’ll be more careful with decisions and less likely to blindly follow a wrong path.

Look past desired outcomes

When designing, it’s common to focus solely on the experience you want to enable. But when you laser-focus on positive outcomes — imagining only delighted users, and then working backwards — it’s easy to miss flaws in your plan.

Designers and technologists should pair up positive thinking with worst-case-scenario planning. By asking, “what’s the worst that could happen?” or “How could this go wrong?” you can head off potential problems far before they become product failures or PR nightmares.

Don’t take datasets at face value

Machine learning is just that: learning. That means the machine has to learn from something. We call that training data: the information the machine is fed to learn how to make decisions. But datasets aren’t neutral — they come from our already biased world.

Take the example of the technology that created those gender analogies, which was a system trained by crunching through Google News articles. Google News results aren’t neutral language devoid of bias — they reflect the stories media companies decided were important, and the people journalists chose to interview. Given current stats about men dominating the tech industry, is it any surprise the system connected computer scientists with men?

Get real in your testing

Even with the best intentions, bias can creep in. But it doesn’t have to make its way to market. When testing design and product decisions, we need to ensure we get them in front of diverse users early in the process. This means going beyond the common startup approach of testing with friends and family, and instead developing a robust, realistic set of users to vet your product.

Think about diversity in as many ways as possible: race, age, ethnicity, gender, sexual orientation, education level, physical ability, native language, country of origin, and more. What are the most important facets of diversity for your product? Who have you inadvertently left out? Pausing for just a moment to improve your testing process can save you huge headaches in the long run.


Sara Wachter-Boettcher is a web consultant and author of the book “Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech.”