Definition
When software makes a decision about a person, the business that runs it stays answerable for the result.
At a glance
- The software is a tool; the operator owns the outcome of any loan denial, hiring screen, or price it sets[1].
- Four tests: can you explain it, trace it, justify it, and fix harm if it goes wrong?
- Regulation is making it mandatory in the EU and proposed in the US.
- Audited, explainable systems cut legal, discrimination, and reputation risk.
Why it matters
Real tools have gone wrong: risk scores in lending and criminal justice showed bias, and one ride-hailing service’s wait times tracked neighborhood ethnicity and income[1]. The danger is the “black box,” where even operators can’t say why a decision was made[1].
The law
The EU AI Act classifies systems by risk and requires high-risk ones like credit scoring and hiring to be assessed before launch and monitored after[3]. The proposed US Algorithmic Accountability Act would have the FTC mandate impact assessments for systems making critical decisions in employment, housing, healthcare, and finance[4][5].
What to do
Treat it like a financial audit, but of the system’s data, design, and decisions[2]. Document how each system works, run bias checks before and after launch (models drift), give people a way to appeal, and get third-party audits — increasingly expected, not optional[2].
Bottom line
Automated systems pass blame straight to the business, so document how they decide, test for bias, and be ready to show your work.