Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models

Jun Luo¹, Chen Chen², Shandong Wu¹,

¹University of Pittsburgh, ²University of Central Florida

ICLR 2025

arXiv Code Poster Slides

Overall pipeline of pFedMoAP

Local workflow of pFedMoAP with proposed attention-based gating network at a client

Abstract

Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods are habitually restricted to the traditional FL paradigm, where the participating clients are generally only allowed to download a single globally aggregated model from the server. While justifiable for training full-sized models under federated settings, in this work, we argue that this paradigm is ill-suited for lightweight prompts. By facilitating the clients to download multiple pre-aggregated prompts as fixed non-local experts, we propose Personalized Federated Mixture of Adaptive Prompts (pFedMoAP), a novel FL framework that personalizes the prompt learning process through the lens of Mixture of Experts (MoE). pFedMoAP implements a local attention-based gating network that learns to generate enhanced text features for better alignment with local image data, benefiting from both local and downloaded non-local adaptive prompt experts. Extensive experiments on 9 datasets under various federated settings demonstrate the efficacy of the proposed pFedMoAP algorithm. The code is available at https://github.com/ljaiverson/pFedMoAP.

BibTeX

@article{luo2024mixture,
      title={Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models},
      author={Luo, Jun and Chen, Chen and Wu, Shandong},
      journal={arXiv preprint arXiv:2410.10114},
      year={2024}
  }
  

PGFed: Personalize Each Client's Global Objective for Federated Learning

Jun Luo¹, Matias Mendieta², Chen Chen², Shandong Wu¹,

¹University of Pittsburgh, ²University of Central Florida

ICCV 2023 Oral

arXiv Code Poster Slides

Workflow and mechanism of PGFed

Abstract

Personalized federated learning has received an upsurge of attention due to the mediocre performance of conventional federated learning (FL) over heterogeneous data. Unlike conventional FL which trains a single global consensus model, personalized FL allows different models for different clients. However, existing personalized FL algorithms only implicitly transfer the collaborative knowledge across the federation by embedding the knowledge into the aggregated model or regularization. We observed that this implicit knowledge transfer fails to maximize the potential of each client's empirical risk toward other clients. Based on our observation, in this work, we propose Personalized Global Federated Learning (PGFed), a novel personalized FL framework that enables each client to personalize its own global objective by explicitly and adaptively aggregating the empirical risks of itself and other clients. To avoid massive (O(N^2)) communication overhead and potential privacy leakage while achieving this, each client's risk is estimated through a first-order approximation for other clients' adaptive risk aggregation. On top of PGFed, we develop a momentum upgrade, dubbed PGFedMo, to more efficiently utilize clients' empirical risks. Our extensive experiments on four datasets under different federated settings show consistent improvements of PGFed over previous state-of-the-art methods. The code is publicly available at https://github.com/ljaiverson/pgfed.

BibTeX

@inproceedings{luo2023pgfed,
      title={Pgfed: Personalize each client's global objective for federated learning},
      author={Luo, Jun and Mendieta, Matias and Chen, Chen and Wu, Shandong},
      booktitle={Proceedings of the IEEE/CVF international conference on computer vision},
      pages={3946--3956},
      year={2023}
    }
  

FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learning

Guanyu Sun¹, Matias Mendieta¹, Jun Luo², Shandong Wu², Chen Chen¹,

¹University of Central Florida, ²University of Pittsburgh

ICCV 2023

arXiv Code Poster

Workflow of FedPerfix

Abstract

Personalized Federated Learning (PFL) represents a promising solution for decentralized learning in heterogeneous data environments. Partial model personalization has been proposed to improve the efficiency of PFL by selectively updating local model parameters instead of aggregating all of them. However, previous work on partial model personalization has mainly focused on Convolutional Neural Networks (CNNs), leaving a gap in understanding how it can be applied to other popular models such as Vision Transformers (ViTs). In this work, we investigate where and how to partially personalize a ViT model. Specifically, we empirically evaluate the sensitivity to data distribution of each type of layer. Based on the insights that the self-attention layer and the classification head are the most sensitive parts of a ViT, we propose a novel approach called FedPerfix, which leverages plugins to transfer information from the aggregated model to the local client as a personalization. Finally, we evaluate the proposed approach on CIFAR-100, OrganAMNIST, and Office-Home datasets and demonstrate its effectiveness in improving the model's performance compared to several advanced PFL methods.

BibTeX

@inproceedings{sun2023fedperfix,
        title={Fedperfix: Towards partial model personalization of vision transformers in federated learning},
        author={Sun, Guangyu and Mendieta, Matias and Luo, Jun and Wu, Shandong and Chen, Chen},
        booktitle={Proceedings of the IEEE/CVF international conference on computer vision},
        pages={4988--4998},
        year={2023}
      }
    

Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning

Jun Luo¹, Shandong Wu¹,

¹University of Pittsburgh

IJCAI 2022

arXiv Code Slides

Workflow of APPLE

Abstract

Conventional federated learning (FL) trains one global model for a federation of clients with decentralized data, reducing the privacy risk of centralized training. However, the distribution shift across non-IID datasets, often poses a challenge to this one-model-fits-all solution. Personalized FL aims to mitigate this issue systematically. In this work, we propose APPLE, a personalized cross-silo FL framework that adaptively learns how much each client can benefit from other clients' models. We also introduce a method to flexibly control the focus of training APPLE between global and local objectives. We empirically evaluate our method's convergence and generalization behaviors, and perform extensive experiments on two benchmark datasets and two medical imaging datasets under two non-IID settings. The results show that the proposed personalized FL framework, APPLE, achieves state-of-the-art performance compared to several other personalized FL approaches in the literature. The code is publicly available at https://github.com/ljaiverson/pFL-APPLE.

BibTeX

@inproceedings{luo2022adapt,
      title={Adapt to adaptation: Learning personalization for cross-silo federated learning},
      author={Luo, Jun and Wu, Shandong},
      booktitle={IJCAI: proceedings of the conference},
      volume={2022},
      pages={2166},
      year={2022}
    }
    

FedSLD: Federated Learning with Shared Label Distribution for Medical Image Classification

Jun Luo¹, Shandong Wu¹,

¹University of Pittsburgh

ISBI 2022

arXiv Slides

Workflow of FedSLD

Abstract

Machine learning in medical research, by nature, needs careful attention on obeying the regulations of data privacy, making it difficult to train a machine learning model over gathered data from different medical centers. Failure of leveraging data of the same kind may result in poor generalizability for the trained model. Federated learning (FL) enables collaboratively training a joint model while keeping the data decentralized for multiple medical centers. However, federated optimizations often suffer from the heterogeneity of the data distribution across medical centers. In this work, we propose Federated Learning with Shared Label Distribution (FedSLD) for classification tasks, a method that assumes knowledge of the label distributions for all the participating clients in the federation. FedSLD adjusts the contribution of each data sample to the local objective during optimization given knowledge of the distribution, mitigating the instability brought by data heterogeneity across all clients. We conduct extensive experiments on four publicly available image datasets with different types of non-IID data distributions. Our results show that FedSLD achieves better convergence performance than the compared leading FL optimization algorithms, increasing the test accuracy by up to 5.50 percentage points.

BibTeX

@inproceedings{luo2022fedsld,
        title={Fedsld: Federated learning with shared label distribution for medical image classification},
        author={Luo, Jun and Wu, Shandong},
        booktitle={2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI)},
        pages={1--5},
        year={2022},
        organization={IEEE}
      }