• Abstract
  • 1 Executive summary
  • 2 Introduction
  • 3 ML audit catalogue
    • 3.1 Project management & governance
      • 3.1.1 Misalignment/ diversion from project objectives
      • 3.1.2 Lack of business readiness/ inability to support the project
      • 3.1.3 Legal and ethical issues
      • 3.1.4 Inappropriate use of ML
      • 3.1.5 Transparency, explainability and fairness
      • 3.1.6 Privacy
      • 3.1.7 Autonomy and accountability
      • 3.1.8 Risk assessment: Project management and governance
    • 3.2 Data
      • 3.2.1 Personal data and GDPR in the context of ML and AI
      • 3.2.2 Risk assessment: Data
      • 3.2.3 Possible audit tests: Data
    • 3.3 Model development
      • 3.3.1 Development process and performance
      • 3.3.2 Cost-benefit analysis
      • 3.3.3 Reliability
      • 3.3.4 Quality assurance.
      • 3.3.5 Risk assessment: Model development
      • 3.3.6 Possible audit tests: Model development
    • 3.4 Model in production
      • 3.4.1 Risk assessment: Model In production
      • 3.4.2 Possible audit tests: Model in production
    • 3.5 Evaluation
      • 3.5.1 Transparency and explainability
      • 3.5.2 Equal treatment and fairness
      • 3.5.3 Security
      • 3.5.4 Risk assessment: Evaluation
      • 3.5.5 Possible audit tests: Evaluation
  • 4 Summary and conclusions
  • Appendix One
    • Classic IT audit components in ML/AI context
    • Personal data and GDPR in the context of ML and AI
    • Equality and fairness measures in classification models
    • Auditability checklist
  • Appendix Two
    • Abbreviations
    • Technical terminology
    • Model evaluation terms
    • Roles
  • Audit helper tool
  • Bibliography

Auditing machine learning algorithms

Audit helper tool

Download the audit helper tool (Excel)