论文合集 | 联邦学习 x KDD'2023

DM-PFL: Hitchhiking Generic Federated Learning for Efficient Shift-Robust Personalization

Authors: Wenhao Zhang; Zimu Zhou; Yansheng Wang; Yongxin Tong

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599311

Abstract: Personalized federated learning collaboratively trains client-specific models, which holds potential for various mobile and IoT applications with heterogeneous data. However, existing solutions are vulnerable to distribution shifts between training and test data, and involve high training workloads on local devices. These two shortcomings hinder the practical usage of personalized federated learning on real-world mobile applications. To overcome these drawbacks, we explore efficient shift-robust personalization for federated learning. The principle is to hitchhike the global model to improve the shift-robustness of personalized models with minimal extra training overhead. To this end, we present DM-PFL, a novel framework that utilizes a dual masking mechanism to train both global and personalized models with weight-level parameter sharing and end-to-end sparse training. Evaluations on various datasets show that our methods not only improve the test accuracy in presence of test-time distribution shifts but also save the communication and computation costs compared to state-of-the-art personalized federated learning schemes.

abstractTranslation: 个性化联邦学习协作训练特定于客户的模型，这为具有异构数据的各种移动和物联网应用程序提供了潜力。然而，现有的解决方案很容易受到训练和测试数据之间分布变化的影响，并且涉及本地设备上的高训练工作负载。这两个缺点阻碍了个性化联邦学习在现实世界移动应用程序上的实际使用。为了克服这些缺点，我们探索了联邦学习的高效、鲁棒的个性化。其原理是搭上全局模型的便车，以最小的额外训练开销来提高个性化模型的稳健性。为此，我们提出了 DM-PFL，这是一种新颖的框架，它利用双重掩码机制通过权重级参数共享和端到端稀疏训练来训练全局和个性化模型。对各种数据集的评估表明，与最先进的个性化联邦学习方案相比，我们的方法不仅提高了存在测试时间分布变化的测试准确性，而且还节省了通信和计算成本。

Notes:

CODE (https://github.com/garyzhang99/DM-PFL)

Navigating Alignment for Non-identical Client Class Sets: A Label Name-Anchored Federated Learning Framework

Authors: Jiayun Zhang; Xiyuan Zhang; Xinyang Zhang; Dezhi Hong; Rajesh K. Gupta; Jingbo Shang

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599443

Abstract: Traditional federated classification methods, even those designed for non-IID clients, assume that each client annotates its local data with respect to the same universal class set. In this paper, we focus on a more general yet practical setting, non-identical client class sets, where clients focus on their own (different or even non-overlapping) class sets and seek a global model that works for the union of these classes. If one views classification as finding the best match between representations produced by data/label encoder, such heterogeneity in client class sets poses a new significant challenge-local encoders at different clients may operate in different and even independent latent spaces, making it hard to aggregate at the server. We propose a novel framework, FedAlign1, to align the latent spaces across clients from both label and data perspectives. From a label perspective, we leverage the expressive natural language class names as a common ground for label encoders to anchor class representations and guide the data encoder learning across clients. From a data perspective, during local training, we regard the global class representations as anchors and leverage the data points that are close/far enough to the anchors of locally-unaware classes to align the data encoders across clients. Our theoretical analysis of the generalization performance and extensive experiments on four real-world datasets of different tasks confirm that FedAlign outperforms various state-of-the-art (non-IID) federated classification methods.

abstractTranslation: 传统的联邦分类方法，即使是为非 IID 客户端设计的方法，也假设每个客户端都根据相同的通用类集注释其本地数据。在本文中，我们关注更一般但实用的设置，不同的客户端类集，其中客户端关注他们自己的（不同甚至不重叠的）类集，并寻求适用于这些类的联邦的全局模型。如果人们将分类视为寻找数据/标签编码器产生的表示之间的最佳匹配，那么客户端类集中的这种异质性提出了一个新的重大挑战——不同客户端的本地编码器可能在不同甚至独立的潜在空间中运行，从而难以聚合在服务器上。我们提出了一个新颖的框架 FedAlign1，从标签和数据的角度调整跨客户的潜在空间。从标签的角度来看，我们利用富有表现力的自然语言类名称作为标签编码器的共同点来锚定类表示并指导跨客户端的数据编码器学习。从数据的角度来看，在本地训练期间，我们将全局类表示视为锚点，并利用与本地不知道类的锚点足够近/远的数据点来跨客户端对齐数据编码器。我们对泛化性能的理论分析以及对不同任务的四个现实数据集的广泛实验证实，FedAlign 优于各种最先进的（非 IID）联邦分类方法。

Notes:

PDF (https://arxiv.org/abs/2301.00489)

CODE (https://github.com/jiayunz/fedalign)

FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy

Authors: Jianqing Zhang; Yang Hua; Hao Wang; Tao Song; Zhengui Xue; Ruhui Ma; Haibing Guan

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599345

Abstract: Recently, personalized federated learning (pFL) has attracted increasing attention in privacy protection, collaborative learning, and tackling statistical heterogeneity among clients, e.g., hospitals, mobile smartphones, etc. Most existing pFL methods focus on exploiting the global information and personalized information in the client-level model parameters while neglecting that data is the source of these two kinds of information. To address this, we propose the Federated Conditional Policy (FedCP) method, which generates a conditional policy for each sample to separate the global information and personalized information in its features and then processes them by a global head and a personalized head, respectively. FedCP is more fine-grained to consider personalization in a sample-specific manner than existing pFL methods. Extensive experiments in computer vision and natural language processing domains show that FedCP outperforms eleven state-of-the-art methods by up to 6.69%. Furthermore, FedCP maintains its superiority when some clients accidentally drop out, which frequently happens in mobile settings. Our code is public at https://github.com/TsingZ0/FedCP(https://github.com/TsingZ0/FedCP).

abstractTranslation: 近年来，个性化联邦学习（pFL）在隐私保护、协作学习和解决客户（例如医院、移动智能手机等）之间的统计异质性方面受到越来越多的关注。大多数现有的 pFL 方法侧重于利用全局信息和个性化信息。客户端层面的模型参数，而忽略了数据是这两种信息的来源。为了解决这个问题，我们提出了联邦条件策略（FedCP）方法，该方法为每个样本生成一个条件策略，将其特征中的全局信息和个性化信息分开，然后分别由全局头和个性化头进行处理。与现有的 pFL 方法相比，FedCP 更细粒度地以特定于样本的方式考虑个性化。计算机视觉和自然语言处理领域的大量实验表明，FedCP 的性能比 11 种最先进的方法高出 6.69%。此外，当某些客户端意外退出时（这种情况在移动环境中经常发生），FedCP 仍保持其优势。我们的代码在 https://github.com/TsingZ0/FedCP(https://github.com/TsingZ0/FedCP) 上公开。

Notes:

PDF (https://arxiv.org/abs/2307.01217)

CODE (https://github.com/tsingz0/fedcp)

FLAMES2Graph: An Interpretable Federated Multivariate Time Series Classification Framework

Authors: Raneen Younis; Zahra Ahmadi; Abdul Hakmeh; Marco Fisichella

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599354

Abstract: Increasing privacy concerns have led to decentralized and federated machine learning techniques that allow individual clients to consult and train models collaboratively without sharing private information. Some of these applications, such as medical and healthcare, require the final decisions to be interpretable. One common form of data in these applications is multivariate time series, where deep neural networks, especially convolutional neural networks based approaches, have established excellent performance in their classification tasks. However, promising results and performance of deep learning models are a black box, and their decisions cannot always be guaranteed and trusted. While several approaches address the interpretability of deep learning models for multivariate time series data in a centralized environment, less effort has been made in a federated setting. In this work, we introduce FLAMES2Graph, a new horizontal federated learning framework designed to interpret the deep learning decisions of each client. FLAMES2Graph extracts and visualizes those input subsequences that are highly activated by a convolutional neural network. Besides, an evolution graph is created to capture the temporal dependencies between the extracted distinct subsequences. The federated learning clients only share this temporal evolution graph with the centralized server instead of trained model weights to create a global evolution graph. Our extensive experiments on various datasets from well-known multivariate benchmarks indicate that the FLAMES2Graph framework significantly outperforms other state-of-the-art federated methods while keeping privacy and augmenting network decision interpretation.

abstractTranslation: 日益增长的隐私问题催生了去中心化和联邦的机器学习技术，这些技术允许个人客户在不共享私人信息的情况下协作咨询和训练模型。其中一些应用（例如医疗和保健）要求最终决定具有可解释性。这些应用中的一种常见数据形式是多元时间序列，其中深度神经网络，尤其是基于卷积神经网络的方法，在其分类任务中已经建立了出色的性能。然而，深度学习模型的有希望的结果和性能是一个黑匣子，它们的决策不能总是得到保证和信任。虽然有几种方法解决了集中式环境中多元时间序列数据的深度学习模型的可解释性，但在联邦环境中所做的努力较少。在这项工作中，我们引入了 FLAMES2Graph，这是一种新的水平联邦学习框架，旨在解释每个客户的深度学习决策。FLAMES2Graph 提取并可视化那些被卷积神经网络高度激活的输入子序列。此外，还创建了一个演化图来捕获提取的不同子序列之间的时间依赖性。联邦学习客户端仅与集中服务器共享此时间演化图，而不是共享经过训练的模型权重来创建全局演化图。我们对来自知名多变量基准的各种数据集进行的广泛实验表明，FLAMES2Graph 框架显着优于其他最先进的联邦方法，同时保持隐私并增强网络决策解释。

Notes:

PDF (https://arxiv.org/abs/2306.03834)

UA-FedRec: Untargeted Attack on Federated News Recommendation

Authors: Jingwei Yi; Fangzhao Wu; Bin Zhu; Jing Yao; Zhulin Tao; Guangzhong Sun; Xing Xie

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599923

Abstract: News recommendation is essential for personalized news distribution. Federated news recommendation, which enables collaborative model learning from multiple clients without sharing their raw data, is a promising approach for preserving users' privacy. However, the security of federated news recommendation is still unclear. In this paper, we study this problem by proposing an untargeted attack on federated news recommendation called UA-FedRec. By exploiting the prior knowledge of news recommendation and federated learning, UA-FedRec can effectively degrade the model performance with a small percentage of malicious clients. First, the effectiveness of news recommendation highly depends on user modeling and news modeling. We design a news similarity perturbation method to make representations of similar news farther and those of dissimilar news closer to interrupt news modeling, and propose a user model perturbation method to make malicious user updates in opposite directions of benign updates to interrupt user modeling. Second, updates from different clients are typically aggregated with a weighted average based on their sample sizes. We propose a quantity perturbation method to enlarge sample sizes of malicious clients in a reasonable range to amplify the impact of malicious updates. Extensive experiments on two real-world datasets show that UA-FedRec can effectively degrade the accuracy of existing federated news recommendation methods, even when defense is applied. Our study reveals a critical security issue in existing federated news recommendation systems and calls for research efforts to address the issue. Our code is available at https://github.com/yjw1029/UA-FedRec.

abstractTranslation: 新闻推荐对于个性化新闻发布至关重要。联邦新闻推荐可以在不共享原始数据的情况下从多个客户端进行协作模型学习，是保护用户隐私的一种有前景的方法。然而，联邦新闻推荐的安全性仍不清楚。在本文中，我们通过提出一种名为 UA-FedRec 的联邦新闻推荐无目标攻击来研究这个问题。通过利用新闻推荐和联邦学习的先验知识，UA-FedRec 可以在少量恶意客户端的情况下有效降低模型性能。首先，新闻推荐的有效性高度依赖于用户建模和新闻建模。我们设计了一种新闻相似性扰动方法，使相似新闻的表示更加接近中断新闻建模，并提出了一种用户模型扰动方法，使恶意用户更新与良性更新相反，从而中断用户建模。其次，来自不同客户的更新通常根据其样本大小进行加权平均值聚合。我们提出了一种数量扰动方法，将恶意客户端的样本量扩大在合理范围内，以放大恶意更新的影响。对两个真实世界数据集的大量实验表明，即使应用防御，UA-FedRec 也可以有效降低现有联邦新闻推荐方法的准确性。我们的研究揭示了现有联邦新闻推荐系统中的一个关键安全问题，并呼吁开展研究工作来解决该问题。我们的代码可在 https://github.com/yjw1029/UA-FedRec 获取。

Notes:

PDF (https://arxiv.org/abs/2202.06701)

CODE (https://github.com/yjw1029/ua-fedrec)

CriticalFL: A Critical Learning Periods Augmented Client Selection Framework for Efficient Federated Learning

Authors: Gang Yan; Hao Wang; Xu Yuan; Jian Li

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599293

Abstract: Federated learning (FL) is a distributed optimization paradigm that learns from data samples distributed across a number of clients. Adaptive client selection that is cognizant of the training progress of clients has become a major trend to improve FL efficiency but not yet well-understood. Most existing FL methods such as FedAvg and its state-of-the-art variants implicitly assume that all learning phases during the FL training process are equally important. Unfortunately, this assumption has been revealed to be invalid due to recent findings on critical learning periods (CLP), in which small gradient errors may lead to an irrecoverable deficiency on final test accuracy. In this paper, we develop CriticalFL, a CLP augmented FL framework to reveal that adaptively augmenting exiting FL methods with CLP, the resultant performance is significantly improved when the client selection is guided by the discovered CLP. Experiments based on various machine learning models and datasets validate that the proposed CriticalFL framework consistently achieves an improved model accuracy while maintains better communication efficiency as compared to state-of-the-art methods, demonstrating a promising and easily adopted method for tackling the heterogeneity of FL training.

abstractTranslation: 联邦学习 (FL) 是一种分布式优化范例，它从分布在多个客户端的数据样本中学习。了解客户训练进度的自适应客户选择已成为提高 FL 效率的主要趋势，但尚未得到充分理解。大多数现有的 FL 方法（例如 FedAvg 及其最先进的变体）隐含地假设 FL 训练过程中的所有学习阶段都同等重要。不幸的是，由于最近对关键学习期（CLP）的发现，这一假设被证明是无效的，其中小的梯度误差可能会导致最终测试准确性的不可挽回的缺陷。在本文中，我们开发了 CriticalFL，一个 CLP 增强 FL 框架，以揭示利用 CLP 自适应增强现有 FL 方法，当客户端选择由发现的 CLP 指导时，最终性能得到显着提高。基于各种机器学习模型和数据集的实验验证了所提出的 CriticalFL 框架始终能够提高模型精度，同时与最先进的方法相比保持更好的通信效率，展示了一种有前途且易于采用的方法来解决异构性问题联邦学习训练。

Notes:

PDF (https://arxiv.org/abs/2109.05613)

Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Mining

Authors: Xidong Wu; Zhengmian Hu; Jian Pei; Heng Huang

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599499

Abstract: To address the big data challenges, serverless multi-party collaborative training has recently attracted attention in the data mining community, since they can cut down the communications cost by avoiding the server node bottleneck. However, traditional serverless multi-party collaborative training algorithms were mainly designed for balanced data mining tasks and are intended to optimize accuracy (e.g., cross-entropy). The data distribution in many real-world applications is skewed and classifiers, which are trained to improve accuracy, perform poorly when applied to imbalanced data tasks since models could be significantly biased toward the primary class. Therefore, the Area Under Precision-Recall Curve (AUPRC) was introduced as an effective metric. Although multiple single-machine methods have been designed to train models for AUPRC maximization, the algorithm for multi-party collaborative training has never been studied. The change from the single-machine to the multi-party setting poses critical challenges. For example, existing single-machine-based AUPRC maximization algorithms maintain an inner state for local each data point, thus these methods are not applicable to large-scale multi-party collaborative training due to the dependence on each local data point. To address the above challenge, in this paper, we reformulate the serverless multi-party collaborative AUPRC maximization problem as a conditional stochastic optimization problem in a serverless multi-party collaborative learning setting and propose a new ServerLess biAsed sTochastic gradiEnt (SLATE) algorithm to directly optimize the AUPRC. After that, we use the variance reduction technique and propose ServerLess biAsed sTochastic gradiEnt with Momentum-based variance reduction (SLATE-M) algorithm to improve the convergence rate, which matches the best theoretical convergence result reached by the single-machine online method. To the best of our knowledge, this is the first work to solve the multi-party collaborative AUPRC maximization problem. Finally, extensive experiments show the advantages of directly optimizing the AUPRC with distributed learning methods and also verify the efficiency of our new algorithms (i.e., SLATE and SLATE-M).

abstractTranslation: 为了应对大数据挑战，无服务器多方协作训练最近引起了数据挖掘界的关注，因为它们可以通过避免服务器节点瓶颈来降低通信成本。然而，传统的无服务器多方协作训练算法主要是为平衡数据挖掘任务而设计的，旨在优化准确性（例如交叉熵）。许多现实应用中的数据分布是倾斜的，经过训练以提高准确性的分类器在应用于不平衡数据任务时表现不佳，因为模型可能明显偏向于主要类别。因此，引入了精确召回曲线下面积（AUPRC）作为一种有效的指标。尽管已经设计了多种单机方法来训练 AUPRC 最大化的模型，但从未研究过多方协作训练的算法。从单机环境到多方环境的转变带来了严峻的挑战。例如，现有的基于单机的AUPRC最大化算法为本地每个数据点维护一个内部状态，因此由于对每个本地数据点的依赖，这些方法不适用于大规模多方协作训练。为了解决上述挑战，本文将无服务器多方协作 AUPRC 最大化问题重新表述为无服务器多方协作学习环境中的条件随机优化问题，并提出了一种新的无服务器双向随机梯度（SLATE）算法优化 AUPRC。之后，我们使用方差缩减技术，提出基于动量方差缩减的ServerLess biAsed sTochastic GradiEnt（SLATE-M）算法来提高收敛速度，这与单机在线方法达到的最佳理论收敛结果相匹配。据我们所知，这是第一个解决多方协作 AUPRC 最大化问题的工作。最后，大量的实验展示了使用分布式学习方法直接优化 AUPRC 的优势，并验证了我们的新算法（即 SLATE 和 SLATE-M）的效率。

Notes:

PDF (https://arxiv.org/abs/2308.03035)

CODE (https://github.com/xidongwu/D-AUPRC)

Personalized Federated Learning with Parameter Propagation

Authors: Jun Wu; Wenxuan Bao; Elizabeth Ainsworth; Jingrui He

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599464

Abstract: With decentralized data collected from diverse clients, a personalized federated learning paradigm has been proposed for training machine learning models without exchanging raw data from local clients. We dive into personalized federated learning from the perspective of privacy-preserving transfer learning, and identify the limitations of previous personalized federated learning algorithms. First, previous works suffer from negative knowledge transferability for some clients, when focusing more on the overall performance of all clients. Second, high communication costs are required to explicitly learn statistical task relatedness among clients. Third, it is computationally expensive to generalize the learned knowledge from experienced clients to new clients. To solve these problems, in this paper, we propose a novel federated parameter propagation (FEDORA) framework for personalized federated learning. Specifically, we reformulate the standard personalized federated learning as a privacy-preserving transfer learning problem, with the goal of improving the generalization performance for every client. The crucial idea behind FEDORA is to learn how to transfer and whether to transfer simultaneously, including (1) adaptive parameter propagation: one client is enforced to adaptively propagate its parameters to others based on their task relatedness (e.g., explicitly measured by distribution similarity), and (2) selective regularization: each client would regularize its local personalized model with received parameters, only when those parameters are positively correlated with the generalization performance of its local model. The experiments on a variety of federated learning benchmarks demonstrate the effectiveness of the proposed FEDORA framework over state-of-the-art personalized federated learning baselines.

abstractTranslation: 利用从不同客户端收集的分散数据，提出了一种个性化的联邦学习范式，用于训练机器学习模型，而无需交换本地客户端的原始数据。我们从隐私保护迁移学习的角度深入研究个性化联邦学习，并找出以往个性化联邦学习算法的局限性。首先，当更多地关注所有客户的整体表现时，以前的工作对某些客户来说会遭受负面的知识可转移性。其次，明确了解客户之间的统计任务相关性需要很高的沟通成本。第三，将经验丰富的客户学到的知识推广到新客户的计算成本很高。为了解决这些问题，在本文中，我们提出了一种用于个性化联邦学习的新型联邦参数传播（FEDORA）框架。具体来说，我们将标准的个性化联邦学习重新表述为隐私保护的迁移学习问题，目标是提高每个客户的泛化性能。FEDORA 背后的关键思想是学习如何传输以及是否同时传输，包括（1）自适应参数传播：强制一个客户端根据任务相关性（例如，通过分布相似性显式测量）将其参数自适应地传播给其他客户端，（2）选择性正则化：只有当这些参数与其本地模型的泛化性能正相关时，每个客户端才会使用接收到的参数对其本地个性化模型进行正则化。对各种联邦学习基准的实验证明了所提出的 FEDORA 框架相对于最先进的个性化联邦学习基准的有效性。

Notes:

Theoretical Convergence Guaranteed Resource-Adaptive Federated Learning with Mixed Heterogeneity

Authors: Yangyang Wang; Xiao Zhang; Mingyi Li; Tian Lan; Huashan Chen; Hui Xiong; Xiuzhen Cheng; Dongxiao Yu

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599521

Abstract: In this paper, we propose an adaptive learning paradigm for resource-constrained cross-device federated learning, in which heterogeneous local submodels with varying resources can be jointly trained to produce a global model. Different from existing studies, the submodel structures of different clients are formed by arbitrarily assigned neurons according to their local resources. Along this line, we first design a general resource-adaptive federated learning algorithm, namely RA-Fed, and rigorously prove its convergence with asymptotically optimal rate O(1/√Γ*TQ) under loose assumptions. Furthermore, to address both submodels heterogeneity and data heterogeneity challenges under non-uniform training, we come up with a new server aggregation mechanism RAM-Fed with the same theoretically proved convergence rate. Moreover, we shed light on several key factors impacting convergence, such as minimum coverage rate, data heterogeneity level, submodel induced noises. Finally, we conduct extensive experiments on two types of tasks with three widely used datasets under different experimental settings. Compared with the state-of-the-arts, our methods improve the accuracy up to 10% on average. Particularly, when submodels jointly train with 50% parameters, RAM-Fed achieves comparable accuracy to FedAvg trained with the full model.

abstractTranslation: 在本文中，我们提出了一种用于资源受限的跨设备联邦学习的自适应学习范式，其中可以联邦训练具有不同资源的异构局部子模型以产生全局模型。与现有研究不同的是，不同客户端的子模型结构是由根据其本地资源任意分配的神经元形成的。沿着这个思路，我们首先设计了一种通用的资源自适应联邦学习算法，即 RA-Fed，并严格证明了其在松散假设下具有渐近最优速率 O(1/√Γ*TQ) 的收敛性。此外，为了解决非均匀训练下的子模型异构性和数据异构性挑战，我们提出了一种新的服务器聚合机制RAM-Fed，具有相同的理论证明收敛速度。此外，我们还阐明了影响收敛的几个关键因素，例如最小覆盖率、数据异质性水平、子模型引起的噪声。最后，我们在不同的实验设置下使用三个广泛使用的数据集对两种类型的任务进行了广泛的实验。与最先进的方法相比，我们的方法将准确率平均提高了 10%。特别是，当子模型使用 50% 的参数联邦训练时，RAM-Fed 的准确度与使用完整模型训练的 FedAvg 相当。

Notes:

Federated Few-shot Learning

Authors: Song Wang; Xingbo Fu; Kaize Ding; Chen Chen; Huiyuan Chen; Jundong Li

Conference : KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599347

Abstract: Federated Learning (FL) enables multiple clients to collaboratively learn a machine learning model without exchanging their own local data. In this way, the server can exploit the computational power of all clients and train the model on a larger set of data samples among all clients. Although such a mechanism is proven to be effective in various fields, existing works generally assume that each client preserves sufficient data for training. In practice, however, certain clients can only contain a limited number of samples (i.e., few-shot samples). For example, the available photo data taken by a specific user with a new mobile device is relatively rare. In this scenario, existing FL efforts typically encounter a significant performance drop on these clients. Therefore, it is urgent to develop a few-shot model that can generalize to clients with limited data under the FL scenario. In this paper, we refer to this novel problem as federated few-shot learning. Nevertheless, the problem remains challenging due to two major reasons: the global data variance among clients (i.e., the difference in data distributions among clients) and the local data insufficiency in each client (i.e., the lack of adequate local data for training). To overcome these two challenges, we propose a novel federated few-shot learning framework with two separately updated models and dedicated training strategies to reduce the adverse impact of global data variance and local data insufficiency. Extensive experiments on four prevalent datasets that cover news articles and images validate the effectiveness of our framework compared with the state-of-the-art baselines.

abstractTranslation: 联邦学习 (FL) 使多个客户端能够协作学习机器学习模型，而无需交换自己的本地数据。通过这种方式，服务器可以利用所有客户端的计算能力，并在所有客户端中的更大的数据样本集上训练模型。尽管这种机制被证明在各个领域都是有效的，但现有的工作通常假设每个客户端都保留足够的数据用于训练。然而，在实践中，某些客户端只能包含有限数量的样本（即少数样本）。例如，特定用户使用新移动设备拍摄的可用照片数据相对较少。在这种情况下，现有的 FL 工作通常会在这些客户端上遇到性能显着下降的情况。因此，迫切需要开发一种能够泛化到 FL 场景下数据有限的客户端的少样本模型。在本文中，我们将这个新问题称为联邦小样本学习。然而，由于两个主要原因，该问题仍然具有挑战性：客户端之间的全局数据差异（即客户端之间数据分布的差异）和每个客户端的本地数据不足（即缺乏足够的本地数据进行训练）。为了克服这两个挑战，我们提出了一种新颖的联邦少样本学习框架，该框架具有两个单独更新的模型和专用训练策略，以减少全局数据方差和本地数据不足的不利影响。对涵盖新闻文章和图像的四个流行数据集进行的广泛实验验证了我们的框架与最先进的基线相比的有效性。

Notes:

PDF (https://arxiv.org/abs/2306.10234)

CODE (https://github.com/songw-sw/f2l)

ShapleyFL: Robust Federated Learning Based on Shapley Value

Authors: Qiheng Sun; Xiang Li; Jiayao Zhang; Li Xiong; Weiran Liu; Jinfei Liu; Zhan Qin; Kui Ren

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599500

Abstract: Federated Learning (FL) allows clients to form a consortium to train a global model under the orchestration of a central server while keeping data on the local client without sharing it, thus mitigating data privacy issues. However, training a robust global model is challenging since the local data is invisible to the server. The local data of clients are naturally heterogeneous, while some clients can use corrupted data or send malicious updates to interfere with the training process artificially. Meanwhile, communication and computation costs are inevitable challenges in designing a practical FL algorithm. In this paper, to improve the robustness of FL, we propose a Shapley value-inspired adaptive weighting mechanism, which regards the FL training as sequential cooperative games and adjusts clients' weights according to their contributions. We also develop a client sampling strategy based on importance sampling, which can reduce the communication cost by optimizing the variance of the global updates according to the weights of clients. Furthermore, to diminish the computation cost of the server, we propose a weight calculation method by estimating differences between the Shapley value of clients. Our experimental results on several real data sets demonstrate the effectiveness of our approaches.

abstractTranslation: 联邦学习（FL）允许客户组成一个联盟，在中央服务器的编排下训练全局模型，同时将数据保留在本地客户端而不共享，从而缓解数据隐私问题。然而，训练鲁棒的全局模型具有挑战性，因为本地数据对服务器不可见。客户端的本地数据天然是异构的，而某些客户端可以使用损坏的数据或发送恶意更新来人为干扰训练过程。同时，通信和计算成本是设计实用的 FL 算法不可避免的挑战。在本文中，为了提高 FL 的鲁棒性，我们提出了一种受 Shapley 值启发的自适应权重机制，将 FL 训练视为顺序合作游戏，并根据客户的贡献调整客户的权重。我们还开发了基于重要性采样的客户端采样策略，该策略可以根据客户端的权重优化全局更新的方差，从而降低通信成本。此外，为了减少服务器的计算成本，我们提出了一种通过估计客户端 Shapley 值之间的差异来计算权重的方法。我们在几个真实数据集上的实验结果证明了我们方法的有效性。

Notes:

Authors: Md Mahmudur Rahman; Sanjay Purushotham

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599348

Abstract: Survival analysis, aka time-to-event analysis, has a wide-ranging impact on patient care. Federated Survival Analysis (FSA) is an emerging Federated Learning (FL) paradigm for performing survival analysis on distributed decentralized data available at multiple medical institutions. FSA enables individual medical institutions, referred to as clients, to improve their survival predictions while ensuring privacy. However, FSA faces challenges due to non-linear and non-IID data distributions among clients, as well as bias caused by censoring. Although recent studies have adapted Cox Proportional Hazards (CoxPH) survival models for FSA, a systematic exploration of these challenges is currently lacking. In this paper, we address these critical challenges by introducing FedPseudo, a pseudo value-based deep learning framework for FSA. FedPseudo uses deep learning models to learn robust representations from non-linear survival data, leverages the power of pseudo values to handle non-uniform censoring, and employs FL algorithms such as FedAvg to learn model parameters. We propose a novel and simple approach for estimating pseudo values for FSA. We provide theoretical proof that the estimated pseudo values, referred to as Federated Pseudo Values, are consistent. Moreover, our empirical results demonstrate that they can be computed faster than traditional methods of deriving pseudo values. To ensure and enhance the privacy of both the estimated pseudo values and the shared model parameters, we systematically investigate the application of differential privacy (DP) on both the federated pseudo values and local model updates. Furthermore, we adapt V -Usable Information metric to quantify the informativeness of a client's data for training a survival model and utilize this metric to show the advantages of participating in FSA. We conducted extensive experiments on synthetic and real-world survival datasets to demonstrate that our FedPseudo framework achieves better performance than other FSA approaches and performs similarly to the best centrally trained deep survival model. Moreover, FedPseudo consistently achieves superior results across different censoring settings.

abstractTranslation: 生存分析，又称事件发生时间分析，对患者护理具有广泛的影响。联邦生存分析 (FSA) 是一种新兴的联邦学习 (FL) 范例，用于对多个医疗机构提供的分布式去中心化数据进行生存分析。FSA 使个体医疗机构（称为客户）能够在确保隐私的同时改善其生存预测。然而，由于客户端之间的非线性和非独立同分布数据分布以及审查造成的偏差，FSA 面临着挑战。尽管最近的研究已经针对 FSA 采用了 Cox 比例风险 (CoxPH) 生存模型，但目前缺乏对这些挑战的系统探索。在本文中，我们通过引入 FedPseudo（一种基于伪值的 FSA 深度学习框架）来解决这些关键挑战。FedPseudo 使用深度学习模型从非线性生存数据中学习稳健的表示，利用伪值的力量来处理非均匀审查，并采用 FedAvg 等 FL 算法来学习模型参数。我们提出了一种新颖且简单的方法来估计 FSA 的伪值。我们提供了理论证明，证明估计的伪值（称为联邦伪值）是一致的。此外，我们的实证结果表明，它们的计算速度比推导伪值的传统方法更快。为了确保和增强估计伪值和共享模型参数的隐私，我们系统地研究了差分隐私（DP）在联邦伪值和本地模型更新上的应用。此外，我们采用 V 可用信息度量来量化客户数据的信息量，以训练生存模型，并利用该度量来展示参与 FSA 的优势。我们对合成和现实世界的生存数据集进行了广泛的实验，以证明我们的 FedPseudo 框架比其他 FSA 方法具有更好的性能，并且与最好的集中训练的深度生存模型类似。此外，FedPseudo 在不同的审查设置中始终取得优异的结果。

Notes:

FedDefender: Client-Side Attack-Tolerant Federated Learning

Authors: Sungwon Park; Sungwon Han; Fangzhao Wu; Sundong Kim; Bin Zhu; Xing Xie; Meeyoung Cha

Conference : KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599346

Abstract: Federated learning enables learning from decentralized data sources without compromising privacy, which makes it a crucial technique. However, it is vulnerable to model poisoning attacks, where malicious clients interfere with the training process. Previous defense mechanisms have focused on the server-side by using careful model aggregation, but this may not be effective when the data is not identically distributed or when attackers can access the information of benign clients. In this paper, we propose a new defense mechanism that focuses on the client-side, called FedDefender, to help benign clients train robust local models and avoid the adverse impact of malicious model updates from attackers, even when a server-side defense cannot identify or remove adversaries. Our method consists of two main components: (1) attack-tolerant local meta update and (2) attack-tolerant global knowledge distillation. These components are used to find noise-resilient model parameters while accurately extracting knowledge from a potentially corrupted global model. Our client-side defense strategy has a flexible structure and can work in conjunction with any existing server-side strategies. Evaluations of real-world scenarios across multiple datasets show that the proposed method enhances the robustness of federated learning against model poisoning attacks.

abstractTranslation: 联邦学习可以在不损害隐私的情况下从分散的数据源中进行学习，这使其成为一项关键技术。然而，它很容易受到模型中毒攻击，即恶意客户端干扰训练过程。以前的防御机制主要通过使用仔细的模型聚合来集中在服务器端，但是当数据分布不均匀或攻击者可以访问良性客户端的信息时，这可能不会有效。在本文中，我们提出了一种专注于客户端的新防御机制，称为 FedDefender，以帮助良性客户端训练稳健的本地模型，并避免攻击者恶意模型更新的不利影响，即使服务器端防御无法识别或消灭对手。我们的方法由两个主要部分组成：（1）耐攻击的本地元更新和（2）耐攻击的全局知识蒸馏。这些组件用于查找抗噪声模型参数，同时从可能损坏的全局模型中准确提取知识。我们的客户端防御策略具有灵活的结构，可以与任何现有的服务器端策略结合使用。对多个数据集的真实场景的评估表明，所提出的方法增强了联邦学习针对模型中毒攻击的鲁棒性。

Notes:

PDF (https://arxiv.org/abs/2307.09048)

CODE (https://github.com/deu30303/feddefender)

Revisiting Personalized Federated Learning: Robustness Against Backdoor Attacks

Authors: Zeyu Qin; Liuyi Yao; Daoyuan Chen; Yaliang Li; Bolin Ding; Minhao Cheng

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599898

Abstract: In this work, besides improving prediction accuracy, we study whether personalization could bring robustness benefits to backdoor attacks. We conduct the first study of backdoor attacks in the pFL framework, testing 4 widely used backdoor attacks against 6 pFL methods on benchmark datasets FEMNIST and CIFAR-10, a total of 600 experiments. The study shows that pFL methods with partial model-sharing can significantly boost robustness against backdoor attacks. In contrast, pFL methods with full model-sharing do not show robustness. To analyze the reasons for varying robustness performances, we provide comprehensive ablation studies on different pFL methods. Based on our findings, we further propose a lightweight defense method, Simple-Tuning, which empirically improves defense performance against backdoor attacks. We believe that our work could provide both guidance for pFL application in terms of its robustness and offer valuable insights to design more robust FL methods in the future. We open-source our code to establish the first benchmark for black-box backdoor attacks in pFL: https://github.com/alibaba/FederatedScope/tree/backdoor-bench.

abstractTranslation: 在这项工作中，除了提高预测准确性之外，我们还研究个性化是否可以为后门攻击带来鲁棒性优势。我们首次在 pFL 框架中进行后门攻击研究，在基准数据集 FEMNIST 和 CIFAR-10 上针对 6 种 pFL 方法测试了 4 种广泛使用的后门攻击，总共 600 次实验。研究表明，具有部分模型共享的 pFL 方法可以显着提高针对后门攻击的鲁棒性。相比之下，具有完全模型共享的 pFL 方法没有表现出鲁棒性。为了分析鲁棒性性能不同的原因，我们对不同的 pFL 方法提供了全面的消融研究。基于我们的发现，我们进一步提出了一种轻量级的防御方法——Simple-Tuning，它根据经验提高了针对后门攻击的防御性能。我们相信，我们的工作可以为 pFL 应用的稳健性提供指导，并为未来设计更稳健的 FL 方法提供有价值的见解。我们开源代码，以建立 pFL 中黑盒后门攻击的第一个基准：https://github.com/alibaba/FederatedScope/tree/backdoor-bench。

Notes:

PDF (https://arxiv.org/abs/2302.01677)

CODE (https://github.com/alibaba/FederatedScope/tree/backdoor-bench)

FedAPEN: Personalized Cross-silo Federated Learning with Adaptability to Statistical Heterogeneity

Authors: Zhen Qin; Shuiguang Deng; Mingyu Zhao; Xueqiang Yan

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599344

Abstract: In cross-silo federated learning (FL), the data among clients are usually statistically heterogeneous (aka not independent and identically distributed, non-IID) due to diversified data sources, lowering the accuracy of FL. Although many personalized FL (PFL) approaches have been proposed to address this issue, they are only suitable for data with specific degrees of statistical heterogeneity. In the real world, the heterogeneity of data among clients is often immeasurable due to privacy concern, making the targeted selection of PFL approaches difficult. Besides, in cross-silo FL, clients are usually from different organizations, tending to hold architecturally different private models. In this work, we propose a novel FL framework, FedAPEN, which combines mutual learning and ensemble learning to take the advantages of private and shared global models while allowing heterogeneous models. Within FedAPEN, we propose two mechanisms to coordinate and promote model ensemble such that FedAPEN achieves excellent accuracy on various data distributions without prior knowledge of data heterogeneity, and thus, obtains the adaptability to data heterogeneity. We conduct extensive experiments on four real-world datasets, including: 1) Fashion MNIST, CIFAR-10, and CIFAR-100, each with ten different types and degrees of label distribution skew; and 2) eICU with feature distribution skew. The experiments demonstrate that FedAPEN almost obtains superior accuracy on data with varying types and degrees of heterogeneity compared with baselines.

abstractTranslation: 在跨竖井联邦学习（FL）中，由于数据来源多样化，客户端之间的数据通常在统计上是异构的（又名非独立同分布，非独立同分布），从而降低了FL的准确性。尽管已经提出了许多个性化 FL (PFL) 方法来解决这个问题，但它们仅适用于具有特定程度的统计异质性的数据。在现实世界中，由于隐私问题，客户之间的数据异质性往往是无法衡量的，这使得有针对性地选择 PFL 方法变得困难。此外，在跨筒仓 FL 中，客户通常来自不同的组织，往往持有架构上不同的私有模型。在这项工作中，我们提出了一种新颖的 FL 框架 FedAPEN，它结合了相互学习和集成学习，以利用私有和共享全局模型的优势，同时允许异构模型。在FedAPEN中，我们提出了两种机制来协调和促进模型集成，使得FedAPEN在无需先验数据异构性的情况下在各种数据分布上实现出色的准确性，从而获得对数据异构性的适应性。我们对四个真实世界的数据集进行了广泛的实验，包括：1）Fashion MNIST、CIFAR-10 和 CIFAR-100，每个数据集都有十种不同类型和程度的标签分布偏斜；2）具有特征分布偏差的 eICU。实验表明，与基线相比，FedAPEN 在不同类型和异质性程度的数据上几乎获得了更高的准确性。

Notes:

CODE (https://github.com/zhenqincn/FedAPEN)

PrivateRec: Differentially Private Model Training and Online Serving for Federated News Recommendation

Authors: Ruixuan Liu; Yang Cao; Yanlin Wang; Lingjuan Lyu; Yun Chen; Hong Chen

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599889

Abstract: Federated recommendation can potentially alleviate the privacy concerns in collecting sensitive and personal data for training personalized recommendation systems. However, it suffers from a low recommendation quality when a local serving is inapplicable due to the local resource limitation and the data privacy of querying clients is required in online serving. Furthermore, a theoretically private solution in both the training and serving of federated recommendation is essential but still lacking. Naively applying differential privacy (DP) to the two stages in federated recommendation would fail to achieve a satisfactory trade-off between privacy and utility due to the high-dimensional characteristics of model gradients and hidden representations. In this work, we propose a federated news recommendation method for achieving better utility in model training and online serving under a DP guarantee. We first clarify the DP definition over behavior data for each round in the pipeline of federated recommendation systems. Next, we propose a privacy-preserving online serving mechanism under this definition based on the idea of decomposing user embeddings with public basic vectors and perturbing the lower-dimensional combination coefficients. We apply a random behavior padding mechanism to reduce the required noise intensity for better utility. Besides, we design a federated recommendation model training method, which can generate effective and public basic vectors for serving while providing DP for training participants. We avoid the dimension-dependent noise for large models via label permutation and differentially private attention modules. Experiments on real-world news recommendation datasets validate that our method achieves superior utility under a DP guarantee in both training and serving of federated news recommendations.

abstractTranslation: 联邦推荐可以潜在地缓解收集敏感数据和个人数据以训练个性化推荐系统时的隐私问题。然而，当由于本地资源限制而不适用本地服务并且在线服务需要查询客户端的数据隐私时，其推荐质量较低。此外，在联邦推荐的训练和服务中理论上的私有解决方案是必要的，但仍然缺乏。由于模型梯度和隐藏表示的高维特性，将差分隐私（DP）简单地应用于联邦推荐的两个阶段将无法在隐私和效用之间实现令人满意的权衡。在这项工作中，我们提出了一种联邦新闻推荐方法，以在 DP 保证下在模型训练和在线服务中实现更好的效用。我们首先澄清联邦推荐系统流程中每轮行为数据的 DP 定义。接下来，我们基于用公共基本向量分解用户嵌入并扰动低维组合系数的思想，在此定义下提出了一种保护隐私的在线服务机制。我们应用随机行为填充机制来降低所需的噪声强度，以获得更好的实用性。此外，我们设计了一种联邦推荐模型训练方法，可以生成有效的、公共的服务基本向量，同时为训练参与者提供DP。我们通过标签排列和差异私人注意模块避免大型模型的维度相关噪声。对现实世界新闻推荐数据集的实验验证了我们的方法在 DP 保证下在联邦新闻推荐的训练和服务方面实现了卓越的实用性。

Notes:

PDF (https://arxiv.org/abs/2204.08146)

NEWS(http://info.ruc.edu.cn/xwgg/xyxw/e4c838332c3a46cd8b959be49c021bb1.htm)

FedMultimodal: A Benchmark for Multimodal Federated Learning

Authors: Tiantian Feng; Digbalay Bose; Tuo Zhang; Rajat Hebbar; Anil Ramakrishna; Rahul Gupta; Mi Zhang; Salman Avestimehr; Shrikanth Narayanan

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599825

Abstract: Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal(https://github.com/usc-sail/fed-multimodal).

abstractTranslation: 在过去的几年里，联邦学习（FL）已成为一种新兴的机器学习技术，通过协作训练来应对数据隐私挑战。在联邦学习算法中，客户端提交本地训练的模型，服务器聚合这些参数直到收敛。尽管人们在计算机视觉、音频和自然语言处理等领域为 FL 做出了巨大的努力，但利用多模态数据流的 FL 应用在很大程度上仍未得到探索。众所周知，多模态学习在情感识别、医疗保健、多媒体和社交媒体等领域具有广泛的现实应用，而用户隐私仍然是一个关键问题。具体来说，目前还没有针对多模式应用或相关任务的 FL 基准测试。为了促进多模态 FL 的研究，我们引入了 FedMultimodal，这是第一个多模态学习的 FL 基准，涵盖来自 10 个常用数据集的 5 个代表性多模态应用，总共有 8 个独特模态。FedMultimodal 提供系统化的 FL 管道，支持从数据分区和特征提取到 FL 基准算法和模型评估的端到端建模框架。与现有的 FL 基准不同，FedMultimodal 提供了一种标准化方法来评估 FL 针对现实多模式应用中三种常见数据损坏的稳健性：缺失模态、缺失标签和错误标签。我们希望 FedMultimodal 能够加速许多未来的研究方向，包括针对极端数据异构性设计多模态 FL 算法、鲁棒性多模态 FL 和高效多模态 FL。数据集和基准测试结果可访问：https://github.com/usc-sail/fed-multimodal。

Notes:

PDF (https://arxiv.org/abs/2306.09486)

CODE (https://github.com/usc-sail/fed-multimodal)

FS-REAL: Towards Real-World Cross-Device Federated Learning

Authors: Daoyuan Chen; Dawei Gao; Yuexiang Xie; Xuchen Pan; Zitao Li; Yaliang Li; Bolin Ding; Jingren Zhou

Conference : Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599829(https://dl.acm.org/doi/10.1145/3580305.3599829)

Abstract: Federated Learning (FL) aims to train high-quality models in collaboration with distributed clients while not uploading their local data, which attracts increasing attention in both academia and industry. However, there is still a considerable gap between the flourishing FL research and real-world scenarios, mainly caused by the characteristics of heterogeneous devices and its scales. Most existing works conduct evaluations with homogeneous devices, which are mismatched with the diversity and variability of heterogeneous devices in real-world scenarios. Moreover, it is challenging to conduct research and development at scale with heterogeneous devices due to limited resources and complex software stacks. These two key factors are important yet underexplored in FL research as they directly impact the FL training dynamics and final performance, making the effectiveness and usability of FL algorithms unclear. To bridge the gap, in this paper, we propose an efficient and scalable prototyping system for real-world cross-device FL, FS-REAL. It supports heterogeneous device runtime, contains parallelism and robustness enhanced FL server, and provides implementations and extensibility for advanced FL utility features such as personalization, communication compression and asynchronous aggregation. To demonstrate the usability and efficiency of FS-REAL, we conduct extensive experiments with various device distributions, quantify and analyze the effect of the heterogeneous device and various scales, and further provide insights and open discussions about real-world FL scenarios. Our system is released to help to pave the way for further real-world FL research and broad applications involving diverse devices and scales.

abstractTranslation: 联邦学习（FL）旨在与分布式客户端协作训练高质量模型，而不上传本地数据，这引起了学术界和工业界越来越多的关注。然而，蓬勃发展的FL研究与现实场景之间仍然存在相当大的差距，这主要是由于异构设备及其规模的特点造成的。大多数现有工作都是使用同质设备进行评估，这与现实场景中异构设备的多样性和可变性不匹配。此外，由于资源有限和软件堆栈复杂，利用异构设备进行大规模研究和开发具有挑战性。这两个关键因素在 FL 研究中很重要，但尚未得到充分探索，因为它们直接影响 FL 训练动态和最终性能，使得 FL 算法的有效性和可用性不清楚。为了弥补这一差距，在本文中，我们提出了一种用于现实世界跨设备 FL（FS-REAL）的高效且可扩展的原型系统。它支持异构设备运行时，包含并行性和鲁棒性增强的 FL 服务器，并为高级 FL 实用功能（例如个性化、通信压缩和异步聚合）提供实现和可扩展性。为了证明 FS-REAL 的可用性和效率，我们对各种设备分布进行了广泛的实验，量化和分析异构设备和各种规模的影响，并进一步提供有关现实世界 FL 场景的见解和公开讨论。我们的系统发布是为了帮助为进一步的现实世界 FL 研究和涉及不同设备和规模的广泛应用铺平道路。

Notes:

PDF (https://arxiv.org/abs/2303.13363)

Privacy Matters: Vertical Federated Linear Contextual Bandits for Privacy Protected Recommendation

Authors: Zeyu Cao; Zhipeng Liang; Bingzhe Wu; Shu Zhang; Hangyu Li; Ouyang Wen; Yu Rong; Peilin Zhao

Conference : KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Url: https://dl.acm.org/doi/10.1145/3580305.3599475

Abstract: Recent awareness of privacy protection and compliance requirement resulted in a controversial view of recommendation system due to personal data usage. Therefore, privacy-protected recommendation emerges as a novel research direction. In this paper, we first formulate this problem as a vertical federated learning problem, i.e., features are vertically distributed over different departments. We study a contextual bandit learning problem for recommendation in the vertical federated setting. To this end, we carefully design a customized encryption scheme named orthogonal matrix-based mask mechanism (O3M). O3M mechanism, a tailored component for contextual bandits by carefully exploiting their shared structure, can ensure privacy protection while avoiding expensive conventional cryptographic techniques. We further apply the mechanism to two commonly-used bandit algorithms, LinUCB and LinTS, and instantiate two practical protocols for online recommendation. The proposed protocols can perfectly recover the service quality of centralized bandit algorithms while achieving a satisfactory runtime efficiency, which is theoretically proved and analysed in this paper. By conducting extensive experiments on both synthetic and real-world datasets, we show the superiority of the proposed method in terms of privacy protection and recommendation performance.

abstractTranslation: 最近人们对隐私保护和合规要求的认识导致了由于个人数据的使用而对推荐系统产生了争议。因此，隐私保护推荐成为一个新颖的研究方向。在本文中，我们首先将这个问题表述为纵向联邦学习问题，即特征纵向分布在不同的部门。我们研究了纵向联邦环境中推荐的上下文强盗学习问题。为此，我们精心设计了一种定制的加密方案，称为基于正交矩阵的掩码机制（O3M）。O3M机制是通过仔细利用其共享结构为上下文强盗量身定制的组件，可以确保隐私保护，同时避免昂贵的传统加密技术。我们进一步将该机制应用于两种常用的老虎机算法LinUCB和LinTS，并实例化了两个实用的在线推荐协议。所提出的协议能够完美恢复集中式强盗算法的服务质量，同时达到令人满意的运行效率，这在本文中得到了理论证明和分析。通过对合成数据集和真实数据集进行广泛的实验，我们展示了所提出的方法在隐私保护和推荐性能方面的优越性。

1.深入浅出零知识证明（四）：交互式零知识证明和计算复杂度

2.会议信息 | 2023年9月截稿的密码学与信息安全会议整理

3.多方安全计算之Beaver三元组

4. 笔记分享｜山东大学隐私计算暑期课：Encryption: Computational security 1-4