KROG
🇳🇴

CWA 18398 · assessment rubric

C.5 Systems Management

← All competences

Level e-3

From role 2.10 AI Observability & Monitoring Specialist · EQF EQF6

Ensure the reliable, secure, and optimised operation of AI systems in hybrid and cloud environments by managing deployments, monitoring performance, mitigating risks, and continuously improving processes through operational, analytical, and compliance-oriented practices.

  1. 4.1 Explain and relate AI system architectures, including model serving, data pipelines, and integration with IT services, in hybrid and cloud computing environments.

    Assessed by AI system architecture diagram review with oral or written explanation of AI architecture, assessed for correct relation of model serving, data pipelines, and IT-service integration in hybrid and cloud computing environments.

  2. 4.2 Apply operational methods to deploy, configure, and scale AI platforms and inference services, using automation, containerization, and orchestration tools.

    Assessed by Evaluation of AI deployment/configuration files and automated deployment simulation check, assessed for application of operational methods to deploy, configure, and scale AI platforms and inference services using automation, containerization, and orchestration tools.

  3. 4.3 Analyse AI system behaviour by examining performance metrics, logs, and operational indicators, to identify bottlenecks, inefficiencies, and anomalies in AI workflows.

    Assessed by AI performance report assessment, data analysis exercises, and case study analysis, assessed for analysis of performance metrics, logs, and operational indicators to identify bottlenecks, inefficiencies, and anomalies in AI workflows.

  4. 4.4 Resolve AI system operational issues by applying incident management and troubleshooting practices, using structured problem-solving approaches and diagnostic tools.

    Assessed by AI incident log evaluation, problem-solving case study, and practical troubleshooting demonstration, assessed for application of incident management, structured problem- solving, and diagnostic tools to resolve AI system operational issues.

  5. 4.5 Evaluate AI system performance, reliability, and operational risks, including degradation and dependency issues, by assessing service-level metrics and potential failure scenarios.

    Assessed by AI risk & reliability assessment report review and scenario-based evaluation, assessed for evaluation of AI system performance, reliability, operational risks, degradation, dependency issues, service-level metrics, and potential failure scenarios.

  6. 4.6 Apply security, access control, and compliance requirements to ensure secure and compliant AI operations, in accordance with organisational policies and regulatory frameworks.

    Assessed by AI security & compliance checklist audit with written or practical assessment of compliance procedures, assessed for application of security, access control, and compliance requirements in accordance with organisational policies and regulatory frameworks.

  7. 4.7 Contribute to improvements of AI operational practices by proposing adjustments based on observed system behaviour, through iterative monitoring and feedback in evolving organisational contexts.

    Assessed by AI operational improvement proposal review and presentation of recommended improvements, assessed for proposed adjustments based on observed system behaviour, iterative monitoring, and feedback in evolving organisational contexts.

From role 2.12 AI Platform Engineer · EQF EQF6

Ensure the reliable, secure, and optimised operation of AI systems in hybrid and cloud environments by managing deployments, monitoring performance, mitigating risks, and continuously improving processes through operational, analytical, and compliance-oriented practices.

  1. 5.1 Explain and relate AI system architectures, including model serving, data pipelines, and integration with IT services, in hybrid and cloud computing environments.

    Assessed by AI system architecture diagram review, with oral or written explanation of AI architecture; assessed for correct explanation and relation of model serving, data pipelines, IT-service integration, and hybrid or cloud computing environment characteristics.

  2. 5.2 Apply operational methods to deploy, configure, and scale AI platforms and inference services, using automation, containerization, and orchestration tools.

    Assessed by Evaluation of AI deployment/configuration files and automated deployment simulation check; assessed for correct application of operational methods to deploy, configure, and scale AI platforms and inference services using automation, containerization, and orchestration tools.

  3. 5.3 Analyse AI system behaviour by examining performance metrics, logs, and operational indicators, to identify bottlenecks, inefficiencies, and anomalies in AI workflows.

    Assessed by AI performance report assessment, data analysis exercises, and case study analysis; assessed for analysis of performance metrics, logs, and operational indicators to identify bottlenecks, inefficiencies, and anomalies in AI workflows.

  4. 5.4 Resolve AI system operational issues by applying incident management and troubleshooting practices, using structured problem-solving approaches and diagnostic tools.

    Assessed by AI incident log evaluation, problem-solving case study, and practical troubleshooting demonstration; assessed for application of incident management, structured problem- solving, and diagnostic tools to resolve AI system operational issues.

  5. 5.5 Evaluate AI system performance, reliability, and operational risks, including degradation and dependency issues, by assessing service-level metrics and potential failure scenarios.

    Assessed by AI risk & reliability assessment report review and scenario-based evaluation; assessed for evaluation of AI system performance, reliability, operational risks, degradation, dependency issues, service-level metrics, and potential failure scenarios.

  6. 5.6 Apply security, access control, and compliance requirements to ensure secure and compliant AI operations, in accordance with organisational policies and regulatory frameworks.

    Assessed by AI security & compliance checklist audit, with written or practical assessment of compliance procedures; assessed for application of security, access control, and compliance requirements in accordance with organisational policies and regulatory frameworks.

  7. 5.7 Contribute to improvements of AI operational practices by proposing adjustments based on observed system behaviour, through iterative monitoring and feedback in evolving organisational contexts.

    Assessed by AI operational improvement proposal review and presentation of recommended improvements; assessed for proposed adjustments based on observed AI system behaviour, iterative monitoring evidence, feedback, and feasibility in evolving organisational contexts.

From role 2.8 MLOps Engineer · EQF EQF6

Ensure the reliable, secure, and optimised operation of AI systems in hybrid and cloud environments by managing deployments, monitoring performance, mitigating risks, and continuously improving processes through operational, analytical, and compliance-oriented practices.

  1. 7.1 Explain and relate AI system architectures, including model serving, data pipelines, and integration with IT services, in hybrid and cloud computing environments.

    Assessed by AI system architecture diagram review with oral or written explanation of AI architecture, assessed for correct relation of model serving, data pipelines, and IT-service integration in hybrid and cloud computing environments.

  2. 7.2 Apply operational methods to deploy, configure, and scale AI platforms and inference services, using automation, containerization, and orchestration tools.

    Assessed by Evaluation of AI deployment/configuration files and automated deployment simulation check, assessed for application of operational methods to deploy, configure, and scale AI platforms and inference services using automation, containerization, and orchestration tools.

  3. 7.3 Analyse AI system behaviour by examining performance metrics, logs, and operational indicators, to identify bottlenecks, inefficiencies, and anomalies in AI workflows.

    Assessed by AI performance report assessment, data analysis exercises, and case study analysis, assessed for analysis of performance metrics, logs, and operational indicators to identify bottlenecks, inefficiencies, and anomalies in AI workflows.

  4. 7.4 Resolve AI system operational issues by applying incident management and troubleshooting practices, using structured problem-solving approaches and diagnostic tools.

    Assessed by AI incident log evaluation, problem-solving case study, and practical troubleshooting demonstration, assessed for application of incident management, structured problem- solving, and diagnostic tools to resolve AI system operational issues.

  5. 7.5 Evaluate AI system performance, reliability, and operational risks, including degradation and dependency issues, by assessing service-level metrics and potential failure scenarios.

    Assessed by AI risk & reliability assessment report review and scenario-based evaluation, assessed for evaluation of AI system performance, reliability, operational risks, degradation, dependency issues, service-level metrics, and potential failure scenarios.

  6. 7.6 Apply security, access control, and compliance requirements to ensure secure and compliant AI operations, in accordance with organisational policies and regulatory frameworks.

    Assessed by AI security & compliance checklist audit with written or practical assessment of compliance procedures, assessed for application of security, access control, and compliance requirements in accordance with organisational policies and regulatory frameworks.

  7. 7.7 Contribute to improvements of AI operational practices by proposing adjustments based on observed system behaviour, through iterative monitoring and feedback in evolving organisational contexts.

From role 2.9 AI Reliability Engineer · EQF EQF6

Ensure the reliable, secure, and optimised operation of AI systems in hybrid and cloud environments by managing deployments, monitoring performance, mitigating risks, and continuously improving processes through operational, analytical, and compliance-oriented practices.

  1. 3.1 Explain and relate AI system architectures, including model serving, data pipelines, and integration with IT services, in hybrid and cloud computing environments.

    Assessed by Evaluation of AI documentation repository for organisation, version control, and collaborative workflow evidence; peer-review of contributions.

  2. 3.2 Apply operational methods to deploy, configure, and scale AI platforms and inference services, using automation, containerization, and orchestration tools.

    Assessed by Assessment of developed AI technical specifications, compliance records, and transparency reports for multiple system components; grading rubric based on completeness and accuracy.

  3. 3.3 Analyse AI system behaviour by examining performance metrics, logs, and operational indicators, to identify bottlenecks, inefficiencies, and anomalies in AI workflows.

    Assessed by Review of consolidated AI documentation integrating datasets, model outputs, and system logs; assessment of clarity, cohesion, and accessibility.

  4. 3.4 Resolve AI system operational issues by applying incident management and troubleshooting practices, using structured problem-solving approaches and diagnostic tools.

    Assessed by Verification of AI documentation against regulatory, ethical, and stakeholder requirements; submission of compliance verification report.

  5. 3.5 Evaluate AI system performance, reliability, and operational risks, including degradation and dependency issues, by assessing service-level metrics and potential failure scenarios.

    Assessed by Evaluation of customized AI documentation templates and modified project documentation; assessment based on alignment with project-specific requirements and adaptability.

  6. 3.6 Apply security, access control, and compliance requirements to ensure secure and compliant AI operations, in accordance with organisational policies and regulatory frameworks.

  7. 3.7 Contribute to improvements of AI operational practices by proposing adjustments based on observed system behaviour, through iterative monitoring and feedback in evolving organisational contexts.

From role 4.6 AI Operations Manager · EQF EQF6

Ensure the reliable, secure, and optimised operation of AI systems in hybrid and cloud environments by managing deployments, monitoring performance, mitigating risks, and continuously improving processes through operational, analytical, and compliance-oriented practices.

  1. 4.1 Explain and relate AI system architectures, including model serving, data pipelines, and integration with IT services, in hybrid and cloud computing environments.

    Assessed by AI system architecture diagram review with oral or written explanation, assessed for model serving, data pipelines, IT service integration, and hybrid or cloud computing context.

  2. 4.2 Apply operational methods to deploy, configure, and scale AI platforms and inference services, using automation, containerization, and orchestration tools.

    Assessed by Evaluation of AI deployment/configuration files and automated deployment simulation check, assessed for deployment, configuration, scaling, automation, containerization, and orchestration of AI platforms or inference services.

  3. 4.3 Analyse AI system behaviour by examining performance metrics, logs, and operational indicators, to identify bottlenecks, inefficiencies, and anomalies in AI workflows.

    Assessed by AI performance report assessment, data analysis exercises, and case study analysis assessing use of metrics, logs, and operational indicators to identify bottlenecks, inefficiencies, and anomalies.

  4. 4.4 Resolve AI system operational issues by applying incident management and troubleshooting practices, using structured problem-solving approaches and diagnostic tools.

    Assessed by AI incident log evaluation, problem-solving case study, and practical troubleshooting demonstration assessing incident management, structured problem-solving, and use of diagnostic tools to resolve AI operational issues.

  5. 4.5 Evaluate AI system performance, reliability, and operational risks, including degradation and dependency issues, by assessing service-level metrics and potential failure scenarios.

    Assessed by AI risk & reliability assessment report review and scenario-based evaluation assessing performance, reliability, degradation, dependency issues, service-level metrics, and failure scenarios.

  6. 4.6 Apply security, access control, and compliance requirements to ensure secure and compliant AI operations, in accordance with organisational policies and regulatory frameworks.

    Assessed by AI security & compliance checklist audit and written or practical assessment of compliance procedures, assessed for security, access control, organisational policy alignment, and regulatory compliance.

  7. 4.7 Contribute to improvements of AI operational practices by proposing adjustments based on observed system behaviour, through iterative monitoring and feedback in evolving organisational contexts.

    Assessed by AI operational improvement proposal review and presentation of recommended improvements, assessed for use of observed system behaviour, iterative monitoring, feedback, and relevance to evolving organisational contexts.