CWA 18398 · assessment rubric

C.3 Service Delivery

Level e-2

From role 3.2 AI Service Support Specialist · EQF EQF5

Ensure the reliable operation and support of AI-enabled services by applying established procedures, tools, and monitoring solutions, while recording incidents and maintaining documentation in accordance with operational guidelines.

3.1 Identify key AI service components, including models, data pipelines, inference services, and supporting ICT infrastructure in typical operational environments.
Assessed by Written AI service inventory and practical identification exercise assessing correct identification of models, data pipelines, inference services, and supporting ICT infrastructure in a typical operational environment.
3.2 Explain AI service level indicators (e.g., availability, latency, accuracy) and their relevance to operational performance using real-world AI services.
Assessed by Short report or presentation of AI service metrics explaining availability, latency, and accuracy and their relevance to operational performance in a real-world AI service.
3.3 Apply standard procedures and tools to monitor AI service performance and availability in routine operational contexts.
Assessed by Lab submission of AI performance monitoring logs demonstrating correct use of standard procedures and tools to monitor AI service performance and availability in a routine operational context.
3.4 Record and report AI-related incidents or anomalies by following established operational guidelines.
Assessed by Completed AI incident report form assessed for accurate recording and reporting of AI- related incidents or anomalies according to established operational guidelines.
3.5 Maintain operational documentation and monitoring tools for AI services using standard templates and instructions.
Assessed by Updated AI operational documentation review assessing completeness, accuracy, template use, and maintenance of monitoring-tool documentation according to standard instructions.

Level e-3

From role 2.10 AI Observability & Monitoring Specialist · EQF EQF6

Enhance the reliability, security, and effectiveness of AI-enabled services by analysing performance, availability, data quality, and incidents, and applying monitoring tools and collaborative practices to implement improvements in dynamic operational contexts.

2.1 Explain AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
Assessed by Written AI principles report, assessed for explanation of AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
2.2 Apply AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
Assessed by Submission of AI monitoring dashboard outputs or screenshots, assessed for use of AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
2.3 Analyse trends in AI service performance, incident patterns, and operational metrics to detect risks or deviations by evaluating historical service records.
Assessed by Analysis report of AI trends and incident patterns, assessed for analysis of AI service performance, incident patterns, and operational metrics to detect risks or deviations using historical service records.
2.4 Evaluate AI service effectiveness, security and continuity against defined criteria, suggesting improvements in practical operational scenarios.
Assessed by AI service evaluation report, assessed for evaluation of AI service effectiveness, security, and continuity against defined criteria and for practical improvement suggestions.
2.5 Collaborate with multidisciplinary teams to implement corrective measures that enhance AI service reliability and compliance in organisational or cross-functional contexts.
Assessed by Documented AI corrective action plan or team report, assessed for multidisciplinary collaboration and implementation of corrective measures that enhance AI service reliability and compliance.

From role 2.11 AI Incident Response & Reporting Lead · EQF EQF7

2.1 Explain AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
Assessed by Written AI principles report, assessed for explanation of AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
2.2 Apply AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
Assessed by Submission of AI monitoring dashboard outputs or screenshots, assessed for use of AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
2.3 Analyse trends in AI service performance, incident patterns, and operational metrics to detect risks or deviations by evaluating historical service records.
Assessed by Analysis report of AI trends and incident patterns, assessed for analysis of AI service performance, incident patterns, and operational metrics to detect risks or deviations using historical service records.
2.4 Evaluate AI service effectiveness, security and continuity against de�ined criteria, suggesting improvements in practical operational scenarios.
Assessed by AI service evaluation report, assessed for evaluation of AI service effectiveness, security, and continuity against defined criteria and for practical improvement suggestions.
2.5 Collaborate with multidisciplinary teams to implement corrective measures that enhance AI service reliability and compliance in organisational or cross-functional contexts.
Assessed by Documented AI corrective action plan or team report, assessed for multidisciplinary collaboration and implementation of corrective measures that enhance AI service reliability and compliance.

From role 2.12 AI Platform Engineer · EQF EQF6

4.1 Explain AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
Assessed by Written AI principles report, assessed for explanation of AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
4.2 Apply AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
Assessed by Submission of AI monitoring dashboard outputs or screenshots, assessed for use of AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
4.3 Analyse trends in AI service performance, incident patterns, and operational metrics to detect risks or deviations by evaluating historical service records.
Assessed by Analysis report of AI trends and incident patterns, assessed for analysis of AI service performance, incident patterns, and operational metrics to detect risks or deviations using historical service records.
4.4 Evaluate AI service effectiveness, security and continuity against defined criteria, suggesting improvements in practical operational scenarios.
Assessed by AI service evaluation report, assessed for evaluation of AI service effectiveness, security, and continuity against defined criteria and for practical improvement suggestions.
4.5 Collaborate with multidisciplinary teams to implement corrective measures that enhance AI service reliability and compliance in organisational or cross-functional contexts.
Assessed by Documented AI corrective action plan or team report, assessed for multidisciplinary collaboration and implementation of corrective measures that enhance AI service reliability and compliance.

From role 2.7 AI Deployment Engineer · EQF EQF6

6.1 Explain AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
Assessed by Written AI principles report, assessed for explanation of AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
6.2 Apply AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
Assessed by Submission of AI monitoring dashboard outputs or screenshots, assessed for use of monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
6.3 Analyse trends in AI service performance, incident patterns, and operational metrics to detect risks or deviations by evaluating historical service records.
Assessed by Analysis report of AI trends and incident patterns, assessed for analysis of service performance, incident patterns, and operational metrics to detect risks or deviations using historical service records.
6.4 Evaluate AI service effectiveness, security and continuity against defined criteria, suggesting improvements in practical operational scenarios.
Assessed by AI service evaluation report, assessed for evaluation of service effectiveness, security, and continuity against defined criteria and for practical improvement suggestions.
6.5 Collaborate with multidisciplinary teams to implement corrective measures that enhance AI service reliability and compliance in organisational or cross-functional contexts.
Assessed by Documented AI corrective action plan or team report, assessed for multidisciplinary collaboration and implementation of corrective measures that enhance AI service reliability and compliance. DEVELOPMENT & OPERATIONS [2]

From role 2.8 MLOps Engineer · EQF EQF6

6.1 Explain AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
6.2 Apply AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
6.3 Analyse trends in AI service performance, incident patterns, and operational metrics to detect risks or deviations by evaluating historical service records.
6.4 Evaluate AI service effectiveness, security and continuity against defined criteria, suggesting improvements in practical operational scenarios.
6.5 Collaborate with multidisciplinary teams to implement corrective measures that enhance AI service reliability and compliance in organisational or cross-functional contexts.

From role 4.6 AI Operations Manager · EQF EQF6

2.1 Explain AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
Assessed by Written AI principles report assessing explanation of AI service delivery principles and their relationship to performance, security, continuity, and compliance in operational or business contexts.
2.2 Apply AI monitoring tools, dashboards, and logs to assess service availability, model performance, and data quality using real-time operational data.
Assessed by Submission of AI monitoring dashboard outputs or screenshots assessed for use of monitoring tools, dashboards, and logs to evaluate service availability, model performance, and data quality.
2.3 Analyse trends in AI service performance, incident patterns, and operational metrics to detect risks or deviations by evaluating historical service records.
Assessed by Analysis report of AI trends and incident patterns assessed for use of historical service records, operational metrics, and detection of risks or deviations.
2.4 Evaluate AI service effectiveness, security and continuity against defined criteria, suggesting improvements in practical operational scenarios.
Assessed by AI service evaluation report assessed for evaluation of effectiveness, security, and continuity against defined criteria, with suggested improvements for practical operational scenarios.
2.5 Collaborate with multidisciplinary teams to implement corrective measures that enhance AI service reliability and compliance in organisational or cross-functional contexts.
Assessed by Documented AI corrective action plan or team report assessed for multidisciplinary collaboration, implemented corrective measures, and contribution to AI service reliability and compliance.