Abstract
1
Executive summary
2
Introduction
3
ML audit catalogue
3.1
Project management & governance
3.1.1
Misalignment/ diversion from project objectives
3.1.2
Lack of business readiness/ inability to support the project
3.1.3
Legal and ethical issues
3.1.4
Inappropriate use of ML
3.1.5
Transparency, explainability and fairness
3.1.6
Privacy
3.1.7
Autonomy and accountability
3.1.8
Risk assessment: Project management and governance
3.2
Data
3.2.1
Personal data and GDPR in the context of ML and AI
3.2.2
Risk assessment: Data
3.2.3
Possible audit tests: Data
3.3
Model development
3.3.1
Development process and performance
3.3.2
Cost-benefit analysis
3.3.3
Reliability
3.3.4
Quality assurance.
3.3.5
Risk assessment: Model development
3.3.6
Possible audit tests: Model development
3.4
Model in production
3.4.1
Risk assessment: Model In production
3.4.2
Possible audit tests: Model in production
3.5
Evaluation
3.5.1
Transparency and explainability
3.5.2
Equal treatment and fairness
3.5.3
Security
3.5.4
Risk assessment: Evaluation
3.5.5
Possible audit tests: Evaluation
4
Summary and conclusions
Appendix One
Classic IT audit components in ML/AI context
Personal data and GDPR in the context of ML and AI
Equality and fairness measures in classification models
Auditability checklist
Appendix Two
Abbreviations
Technical terminology
Model evaluation terms
Roles
Audit helper tool
Bibliography
Auditing machine learning algorithms
Appendix One