Automating federated learning across multiple organizational sites is a significant challenge in this project. The implementation will feature components for the three main phases of federated learning and components for the meta-learning phase. An orchestration component utilizing open-source projects like Kubernetes (federated as KubeFedv2 or edge orchestration variants such as KubeEdge) will be employed to automate the deployment and monitoring of the federated learning process. These components are designed to adapt to the devices they will operate on, considering hardware capabilities such as the availability of GPU acceleration. Furthermore, they will be highly configurable to meet the unique requirements of each use case, ensuring their versatility for use in various federated learning scenarios.
Our work embarks on a journey of innovation with the following specific objectives:
- Define and develop federated lifecycle management and a federated learning orchestrator to manage the process lifecycle.
- Define and implement service management to operate AI-based services that can be configured and deployed for federated learning.
- Define and develop components for deploying and monitoring the learning phase.
- Define and develop components for deploying and monitoring the learning phase, fostering a collaborative environment with mechanisms for defining and enforcing sharing policies between participating organizations. Define and implement components for deploying and monitoring the inference phase.
- Define and implement meta-learning components to continue learning new operational data in parallel with the inference phase and decide when to deploy updated global models.
| Key performance indicator (KPI) | Leader | Contributor | Chronology |
|---|---|---|---|
| ●Open-source implementation of the AIDE methodology covering all phases of the federated learning process ● To realize the federation layer for decentralized orchestration, we propose the design and implementation of AI-based workload management modules. These modules will provide a consistent view of the system through cognitive monitoring (knowledge plane) and maintain a consistent set of actions for resource management (control plane). We will use multi-agent reinforcement learning strategies to achieve possibly opposing goals. The benefits of these modules include: Improved global topology management. Enhanced local cluster resource management. More accurate service model description for high-level application requirements specification. ● Increase in autoscaling responsiveness compared with a multi-cloud scenario without a federation layer (target: 20%); Reduction in deployment time compared with a multi-cloud scenario without a federation layer (target: 50%). | IMEC | UCLOUVAIN CETIC | ● March 2023 : Initial open-source implementation of generic federated learning components on top of the selected orchestrator. CI/CD pipelines for rapid integration of changes. ● End 2023 : initial release of AI-based workload management components In case of extension: ● End 2024 (if extended): second iteration of open source implementations of AIDE building blocks, taking into account feedback from use cases and initial performance and scaling evaluations. ● End 2024 (if extended) : second iteration of AI-based workload management components. |