Abstract: With the introduction of user’s data security and privacy protection policies, the enterprises are faced with the problem of data security and user privacy when they providing better services to users. Federated learning can help companies continue better output of innovative products and improvement of service quality under the premise of satisfying the data security and regulations. As the world’s first industrial-grade open-source framework, FATE supports a federated learning architecturesystem and provides the great performance federated learning mechanism for machine learning, deep learning, and transfer learning. FATE itself also supports multi-party security computation protocols, such as homomorphic encryption, secret sharing, hash algorithm, with a friendly cross-domain interactive information management solution.
Cooperated with VMware CloudNative lab team from VMware China R&D open innovation center, the world’s first industrial-scale open-source framework, FATE can be used and deployed in the Private Cloud and Public Cloud. Click here to experience the CloudNative deploy solution packaged in the container.
On October 31st, FATE v1.1 has been released. In this version, FATE has released the KubeFATE project with the team from CloudNative lab, VMware China R&D open innovation center. FATE now can be deployed using Docker Compose or Kubernetes (Helm Charts) through encapsulating all components of FATE in containers. Applications are developed in a DevOps fashion, and the advantages of deploying applications based on container are quite obvious. Applications can not only run on the platforms that supports containers without distinction but also be flexibly scaled at multi-instance level on demand. Now the project has been released on GitHub: https://github.com/FederatedAI/KubeFATE
At present, the mainstream cloud platforms, such as AWS, Azure in foreign countries and Alibaba Cloud, Tencent Cloud in China, all have CloudNative services based on containers and Kubernetes, which have standardized and commercialized the deployment and operation of container applications. The developers can easily deploy and use the FATE project by KubeFATE in the public cloud and private cloud. The developers can easily deploy and use the FATE project by KubeFATE in the Public Cloud and Private Cloud.
In addition, FATE v1.1 has upgraded and improved a lot in the algorithm and functions: FATE v1.1 not only provide a general algorithm framework for homogeneous federated learning and add DNN, Regression algorithm, but also it began to support multi-host heterogeneous federated modeling and add spark as computing engine, supporting FATEServing to service governance and secureboost to forecast online. Version 1.1 has improved again the federated learning modeling experience with richer functions and the comprehensive algorithm. It helps more companies and users to join into the deep research of FATE technology and application.
FederatedML: Provide an easily-extensible homogeneous algorithm framework to support the Horizontal algorithm development
In this new version, it is easier to develop for developers. They can focus more on the algorithm and the framework can handle more general transmission communication content. FATE v1.1 provides an easily-extensible homogeneous algorithm framework, supporting Secure Aggregation. The developers can easily realize the homogeneous federated learning algorithm framework by encapsulating the main process of homogeneous federated learning.
As for algorithm, FATE has supported the add the Homogeneous Deep Neural Network, Heterogeneous Linear Regression and Heterogeneous Poisson Regression, which enriches more modeling scenarios and improves the practicability of FATE. It is worth mentioning that it’s very useful to use Linear Regression in the scenario of predicting a continuous label. Poisson Regression can help developers better to forecast the counts and frequency, such as the frequency of purchasing Insurance and the prediction of the accidents’ frequency.
In this version, FATE also begins to support multi-host heterogeneous federated modeling, which can enable multiple data providers to train the federated model under a heterogeneous scenario.
At last, FATE also tries to connect Spark. FATE v1.1 supports the developers who already have had a Spark cluster to reuse the existing resource. In this version, you can choose Spark as a computing engine to configure flexibly according to your actual situation. Move to GitHub to learn more information: https://github.com/FederatedAI/FATE/tree/master/federatedml
FATEFlow: High performance federated learning Pipeline production service
FATEFlow is a Pipeline scheduling and lifecycle management tool for federated learning modeling for users to establish end to end federated learning Pipeline production service. In version 1.1, FATEFlow improves stability and usability. For example:
* Upload and Download support CLI for querying job status
* Support for canceling waiting job
* Support for setting job timeout
* Support for Optimize the job log and store it in the log folder named by job ID to improve the efficiency of troubleshooting
FATEBorad: Simple and efficient, visual modeling process of federated learning
FATEBorad is a visualization tool for federated learning modeling, the entire process for end-users to visualize and measure the training models, helping users to explore and understand models more easily and efficiently. In this new version, the job workflow presentation is further optimized and support the input and output port separation of component data and models, providing a visual model of data and model transmission;
Besides, now FATEBorad has supported the visualization of model training evaluation results to follow and track the training process and result in real-time. It also provides the visual tree models of Heterogeneous Secureboost, which is not only clear to monitor every decision trees of models but also check the tree model of different labels.
FATEServing: service governance, automatically restore the loaded model when service restarts
When the model is loaded successfully, it will save in the local folder. The new version support automatically to restore the loaded model when service restarts.
Besides, the zookeeper is the registration center in version 1.1, providing the limited function of service governance. It enables to dynamically registry grpc interface and automatically switches the flow when some machines are down.
KubeFATE：the ability of FATE deployment is upgraded
FATE v1.1 provides the packaged Docker container image to reduce the threshold of using FATE, avoiding the developers to lose at the beginning. The companies’ developers can realize that the ability of off-line deploying FATE has been improved. By the Harbor open-source container image warehouse, it can automatically sync the online image for the operation and maintenance of decompression.
KubeFATE mainly provides two deployment methods: Docker compose and Kubernetes (Helm Chart).
Docker-Compose can deploy every component of FATE on a single node and support multi-party deployment. The developers don’t need any compiled code to use Docker compose to set up a test environment quickly. Now Docker compose can deploy FATE on one or more nodes, which is better for developers to get familiar with the FATE functions.
Single node deployment of Docker-Compose is used to test. In the production environment, multiple node deployments are required. But it is better to use Kubernetes. KubeFATE provides the method that using Helm Charts to deploy FATE on Kubernetes. It supports to deploy FATE on the Cloud of Kubernetes. Besides, it can customize the details of the deployment as required, such as deploying a computation module on the node with GPU.
Harbor is an open-source image warehouse, providing access control, remote synchronization and security vulnerability scanning of the image. Most of the Chinese users use Harbor to manage the image. Harbor is integrated into the KubeFATE project to provide the management ability of the local image. It doesn’t rely on Docker Hub and other cloud services, which greatly increased the efficiency and security. In addition, Harbor can copy the remote image, which can be copied between Public Cloud or data center and automatically recovered from failures, thus simplifying the operation and maintenance complexity.
Overall, FATE v1.1 added multiple federated learning algorithms and functional modules, bringing richer and powerful features for federated learning modeling. Meanwhile, it launched the KubeFATE in conjunction With VMware, simplifying the use thresholed for FATE, which is more friendly for beginners. Anyone interested in federated learning is welcome to contribute code or documents and submit issues or Pull Requests. . Please refer to the FATE project contribution guide for details: https://fate.fedai.org/contribute/