與單一模態資料相比，多模態感測資料能提升複雜任務的模型表現。聯邦學習（Federated Learning）進一步增強了這一點，既保護數據隱私又確保模型訓練效果。然而，現有的FL算法往往忽視了部分用戶分享特定資料模態的意願，並且因缺乏大規模公開資料集而難以獲取足夠的標記資料。我們提出了模態感知聯邦半監督學習（MAFS）架構，允許各個客戶端選擇他們認為不敏感且願意與FL伺服器分享的資料模態。MAFS從這些未標記的不敏感資料中提取有用訊息，以減輕標記資料匱乏的問題。我們在情感識別（Emotion Recognition）和人類活動識別（Human Activity Recognition）兩個任務上評估了MAFS，並將其與多種最先進的FL算法進行比較。實驗結果顯示，在情感識別任務中，當標記資料比例僅為30\%時，MAFS將準確率提高了至少6.94%，F1-Score提高了至少9.49%，僅比完全標記資料的結果低0.63%和0.46%。在人類活動識別任務中，MAFS也有良好表現，例如，準確率提高了至少3.37%，且F1-Score沒有顯著下降。

Compared to unimodal data, multimodal sensor data improves model performance for complex tasks. Federated Learning (FL) further enhances this by preserving data privacy while ensuring well-trained models. However, existing FL algorithms often overlook some users' willingness to share certain data modalities and struggle to acquire sufficient labeled data due to a scarcity of large-scale public datasets. We propose Modality-Aware Federated Semi-Supervised Learning (MAFS) paradigm, allowing individual clients to select which data modalities they consider insensitive and are willing to share with the FL server. MAFS then extracts useful information from those unlabeled insensitive data to mitigate labeled data scarcity. We evaluate MAFS on two tasks: Emotion Recognition (ER) and Human Activity Recognition (HAR), and compare it with several state-of-the-art FL algorithms. The experimental results show that, in the ER task, when the labeled data rate is only 30%, MAFS improves the accuracy by at least 6.94% and F1-score by at least 9.49%, which are merely 0.63% and 0.46% away from those from a fully labeled dataset. MAFS also performs well in the HAR task, e.g., it improves the accuracy by at least 3.37% with no significant drop in F1-score.

[1] L. Agbley, J. Li, A. Haq, K. Bankas, S. Ahmad, O. Agyemang, D. Kulevome,
D. Ndiaye, B. Cobbinah, and S. Latipova. Multimodal melanoma detection with
federated learning. In Proc. of 18th International Computer Conference on Wavelet
Active Media Technology and Information Processing (ICCWAMTIP), pages 238–
244, Chengdu, China, December 2021.
[2] S. Amershi, M. Cakmak, B. Knox, and T. Kulesza. Power to the people: The role of
humans in interactive machine learning. AI magazine, 35(4):105–120, 2014.
[3] P. Atrey, A. Hossain, A. El Saddik, and M. Kankanhalli. Multimodal fusion for
multimedia analysis: a survey. Multimedia systems, 16:345–379, 2010.
[4] T. Baltrusaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A
survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelli-
gence, 41(2), 2018.
[5] T. Baltruˇsaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A sur-
vey and taxonomy. IEEE transactions on pattern analysis and machine intelligence,
41(2):423–443, 2018.
[6] M. Belkin, P. Niyogi, and V. Sindhwani. Manifold regularization: A geometric
framework for learning from labeled and unlabeled examples. Journal of machine
learning research, 7(11), 2006.
[7] T. Bernecker, A. Peters, C. Schlett, F. Bamberg, F. Theis, D. Rueckert, J. Weiß, and
S. Albarqouni. Fednorm: Modality-based normalization in federated learning for
multi-modal liver segmentation. arXiv preprint arXiv:2205.11096, 2022.
[8] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. Chang,
S. Lee, and S. Narayanan. IEMOCAP: Interactive emotional dyadic motion cap-
ture database. Language Resources and Evaluation, 42(4), 2008.
[9] E. Celebi and K. Aydin. Unsupervised learning algorithms, volume 9. Springer,
2016.
[10] K. Chaitanya, N. Karani, C. Baumgartner, A. Becker, O. Donati, and E. Konukoglu.
Semi-supervised and task-driven data augmentation. In Proc. of 26th Information
Processing in Medical Imaging International Conference, pages 29–41, Hong Kong,
China, 2019.
[11] F. Chen, M. Luo, Z. Dong, Z. Li, and X. He. Federated meta-learning with fast
convergence and efficient communication. arXiv preprint arXiv:1802.07876, 2018.
[12] Y. Chen, C.-F. Hsu, C.-C. Tsai, and C.-H. Hsu. Hpfl: Federated learning by fusing
multiple sensor modalities with heterogeneous privacy sensitivity levels. In Proc.
of the 1st International Workshop on Methodologies for Multimedia, pages 5–14,
Lisboa, Portugal, 2022. ACM.
[13] Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao. Fedhealth: A federated transfer
learning framework for wearable healthcare. IEEE Intelligent Systems, 35(4):83–
93, 2020.
[14] L. Corinzia, A. Beuret, and J. Buhmann. Variational federated multi-task learning.
arXiv preprint arXiv:1906.06268, 2019.
[15] E. Diao, J. Ding, and V. Tarokh. Semifl: Semi-supervised federated learning for
unlabeled clients with alternate training. Advances in Neural Information Processing
Systems, 35:17871–17884, 2022.
[16] A. M. Elbir, S. Coleri, and K. V. Mishra. Hybrid federated and centralized learning.
In European Signal Processing Conference (EUSIPCO), pages 1541–1545, Dublin,
Ireland, 2021. IEEE.
[17] A. Fallah, A. Mokhtari, and A. Ozdaglar. Personalized federated learning: A meta-
learning approach. arXiv preprint arXiv:2002.07948, 2020.
[18] C. Fan, J. Hu, and J. Huang. Private semi-supervised federated learning. In Proc. of
International Joint Conference on Artificial Intelligence, pages 2009–2015, Vienna,
Austria, July 2022.
[19] T. Feng, D. Bose, T. Zhang, R. Hebbar, A. Ramakrishna, R. Gupta, M. Zhang,
S. Avestimehr, and S. Narayanan. Fedmultimodal: A benchmark for multimodal
federated learning. In Proc. of the 29th ACM SIGKDD Conference on Knowledge
Discovery and Data Mining, pages 4035–4045, 2023.
[20] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation.
In International conference on machine learning, pages 1180–1189, Lille, France,
2015.
[21] A. Ghosh, J. Chung, D. Yin, and K. Ramchandran. An efficient framework for
clustered federated learning. Advances in Neural Information Processing Systems,
33:19586–19597, 2020.
[22] J. Gou, B. Yu, S. Maybank, and D. Tao. Knowledge distillation: A survey. Springer
International Journal of Computer Vision, 129(6):1789–1819, 2021.
[23] N. Grira, M. Crucianu, and N. Boujemaa. Unsupervised and semi-supervised clus-
tering: a brief survey. A review of machine learning techniques for processing mul-
timedia content, 1(2004):9–16, 2004.
[24] N. Guha, A. Talwalkar, and V. Smith. One-shot federated learning. arXiv preprint
arXiv:1902.11175, 2019.
[25] T. Guo, S. Guo, and J. Wang. pfedprompt: Learning personalized prompt for vision-
language models in federated learning. In Proc. of the ACM Web Conference, pages
1364–1374, Austin, TX, April 2023.
[26] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network.
arXiv preprint arXiv:1503.02531, 2015.
[27] W. Hong, X. Luo, Z. Zhao, M. Peng, and T. Quek. Optimal design of hybrid feder-
ated and centralized learning in the mobile edge computing systems. In ICC, pages
1–6. IEEE, 2021.
[28] L. Huang, Y. Yin, Z. Fu, S. Zhang, H. Deng, and D. Liu. LoAdaBoost: Loss-based
AdaBoost federated machine learning with reduced computational complexity on
iid and non-iid intensive care data. Plos One, 15(4), 2020.
[29] Y. Huang, L. Chu, Z. Zhou, L. Wang, J. Liu, J. Pei, and Y. Zhang. Personalized
cross-silo federated learning on non-iid data. In Proc. of the AAAI conference on
artificial intelligence, volume 35, pages 7865–7873, Virtual, 2021.
[30] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim. Communication-
efficient on-device machine learning: Federated distillation and augmentation under
non-iid private data. arXiv preprint arXiv:1811.11479, 2018.
[31] E. Jeong, S. Oh, J. Park, H. Kim, M. Bennis, and S.-L. Kim. Hiding in the crowd:
Federated data augmentation for on-device learning. IEEE Intelligent Systems,
36(5):80–87, 2020.
[32] W. Jeong, J. Yoon, E. Yang, and S. J. Hwang. Federated semi-supervised learning
with inter-client consistency & disjoint learning. arXiv preprint arXiv:2006.12097,
2020.
[33] V. Kulkarni, M. Kulkarni, and A. Pant. Survey of personalization techniques for
federated learning. In WorldS4, pages 794–797. IEEE, July 2020.
[34] D. Li and J. Wang. Fedmd: Heterogenous federated learning via model distillation.
arXiv preprint arXiv:1910.03581, 2019.
[35] T. Li, K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated opti-
mization in heterogeneous networks. Proceedings of Machine learning and systems,
2:429–450, 2020.
[36] X. Li, W. Yang, Z. Zhang, K. Huang, and S. Wang. On the convergence of fedavg
on non-iid data. arXiv preprint arXiv:1907.02189, 2019.
[37] Y.-F. Li and Z.-H. Zhou. Towards making unlabeled data never hurt. IEEE transac-
tions on pattern analysis and machine intelligence, 37(1):175–188, 2014.
[38] P. Liang, T. Liu, L. Ziyin, N. Allen, R. Auerbach, D. Brent, R. Salakhutdinov, and
L.-P. Morency. Think locally, act globally: Federated learning with local and global
representations. arXiv preprint arXiv:2001.01523, 2020.
[39] X. Liang, Y. Lin, H. Fu, L. Zhu, and X. Li. Rscfed: Random sampling consensus
federated semi-supervised learning. In Proc. of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition, pages 10154–10163, New Orleans, Louisiana,
June 2022.
[40] T. Lin, L. Kong, S. Stich, and M. Jaggi. Ensemble distillation for robust model
fusion in federated learning. Advances in Neural Information Processing Systems,
33:2351–2363, 2020.
[41] B. Liu, M. Ding, S. Shaham, W. Rahayu, F. Farokhi, and Z. Lin. When machine
learning meets privacy: A survey and outlook. ACM Computing Surveys (CSUR),
54(2):1–36, 2021.
[42] F. Liu, X. Wu, S. Ge, W. Fan, and Y. Zou. Federated learning for vision-and-
language grounding problems. In Proc. of the AAAI Conference on Artificial In-
telligence, volume 34, pages 11572–11579, New York, NY, February 2020.
[43] Y. Liu, Y. Kang, C. Xing, T. Chen, and Q. Yang. A secure federated transfer learning
framework. IEEE Intelligent Systems, 35(4):70–82, 2020.
[44] Z. Liu, Y. Shen, V. B. Lakshminarasimhan, P. Liang, A. Zadeh, and L.-P. Morency.
Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint
arXiv:1806.00064, 2018.
[45] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Arcas. Communication-
efficient learning of deep networks from decentralized data. In Proc. of PMLR Inter-
national Conference on Artificial Intelligence and Statistics (AISTATS), pages 1273–
1282, FL, USA, 2017.
[46] R. Mendes and J. P. Vilela. Privacy-preserving data mining: methods, metrics, and
applications. IEEE Access, 5:10562–10582, 2017.
[47] L. Nagalapatti and R. Narayanam. Game of gradients: Mitigating irrelevant clients
in federated learning. In Proceedings of the AAAI Conference on Artificial Intelli-
gence, volume 35, pages 9046–9054, Virtual, 2021.
[48] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. Multimodal deep
learning. In Proc. of the 28th international conference on machine learning (ICML-
11), pages 689–696, Bellevue, Washington, 2011.
[49] J. Park, S. Wang, A. Elgabli, S. Oh, E. Jeong, H. Cha, H. Kim, S.-L. Kim, and
M. Bennis. Distilling on-device intelligence at the network edge. arXiv preprint
arXiv:1908.05895, 2019.
[50] M. Phuong and C. Lampert. Towards understanding knowledge distillation. In
International Conference on Machine Learning, pages 5142–5151, Long Beach,
CA, 2019. PMLR.
[51] A. Ratner, S. Bach, H. Ehrenberg, and C. R´e. Snorkel: Fast training set generation
for information extraction. In Proc. of the 2017 ACM international conference on
management of data, pages 1683–1686, Hilton, Chicago, 2017.
[52] Y. Ruan and C. Joe-Wong. Fedsoft: Soft clustered federated learning with prox-
imal local updating. In Proc. of the AAAI Conference on Artificial Intelligence,
volume 36, pages 8124–8131, Virtual, 2022.
[53] S. Sharma, C. Xing, Y. Liu, and Y. Kang. Secure and efficient federated transfer
learning. In IEEE International Conference on Big Data (Big Data), pages 2569–
2576, Los Angeles, CA, 2019. IEEE.
[54] N. Sikder and A.-A. Nahid. Ku-har: An open dataset for heterogeneous human
activity recognition. Pattern Recognition Letters, 146:46–54, 2021.
[55] K. Sindhu Meena and S. Suriya. A survey on supervised and unsupervised learn-
ing techniques. In Pro. of international conference on artificial intelligence, smart
grid and smart city applications: AISGSC 2019, pages 627–644, Coimbatore, India,
2020.
[56] K. Sohn, W. Shang, and H. Lee. Improved multimodal deep learning with variation
of information. Advances in neural information processing systems, 27, 2014.
[57] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu. A survey on deep transfer
learning. In International conference on artificial neural networks, pages 270–279,
Rhodes, Greece, 2018. Springer.
[58] I. Wagner and D. Eckhoff. Technical privacy metrics: a systematic survey. ACM
Computing Surveys (Csur), 51(3):1–38, 2018.
[59] H. Wang, L. Munoz-Gonzalez, D. Eklund, and S. Raza. Non-iid data re-balancing
at IoT edge with peer-to-peer federated learning for anomaly detection. In WiSec,
pages 153–163. ACM, 2021.
[60] H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni. Federated
learning with matched averaging. arXiv preprint arXiv:2002.06440, 2020.
[61] K. Wang, R. Mathews, C. Kiddon, H. Eichner, F. Beaufays, and D. Ramage. Fed-
erated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252,
2019.
[62] Y. Wang. Survey on deep multi-modal data analytics: collaboration, rivalry, and
fusion. ACM Transactions on Multimedia Computing, Communications, and Appli-
cations, 17(1s), 2021.
[63] Z. Wu, Y. Xiong, S. Yu, and D. Lin. Unsupervised feature learning via non-
parametric instance discrimination. In Proc. of the IEEE conference on computer
vision and pattern recognition, pages 3733–3742, Salt Lake City, UT, 2018.
[64] B. Xiong, X. Yang, F. Qi, and C. Xu. A unified framework for multi-modal federated
learning. Elsevier, Neurocomputing, 480:110–118, 2022.
[65] Y. Xu, L. Wang, H. Xu, J. Liu, Z. Wang, and L. Huang. Enhancing federated learn-
ing with server-side unlabeled data by adaptive client and data selection. IEEE
Transactions on Mobile Computing, 2023.
[66] H. Yang, H. He, W. Zhang, and X. Cao. Fedsteg: A federated transfer learning
framework for secure image steganalysis. IEEE Transactions on Network Science
and Engineering, 8(2):1084–1094, 2020.
[67] X. Yang, B. Xiong, Y. Huang, and C. Xu. Cross-modal federated human activity
recognition via modality-agnostic and modality-specific representation learning. In
Proc. of the AAAI Conference on Artificial Intelligence, volume 36, pages 3063–
3071, Virtual, February 2022.
[68] N. Yoshida, T. Nishio, M. Morikura, K. Yamamoto, and R. Yonetani. Hybrid-FL
for wireless networks: Cooperative learning mechanism using non-iid data. In ICC,
pages 1–7. IEEE, 2020.
[69] T. Yu, E. Bagdasaryan, and V. Shmatikov. Salvaging federated learning by local
adaptation. arXiv preprint arXiv:2002.04758, 2020.
[70] Y. Zhao, P. Barnaghi, and H. Haddadi. Multimodal federated learning on iot data. In
Proc. of IEEE/ACM Seventh International Conference on Internet-of-Things Design
and Implementation (IoTDI), pages 43–54, Milan, Italy, May 2022.
[71] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra. Federated learning with
non-iid data. arXiv preprint arXiv:1806.00582, 2018.
[72] X. J. Zhu. Semi-supervised learning literature survey. 2005.
[73] Z. Zhu, J. Hong, and J. Zhou. Data-free knowledge distillation for heterogeneous
federated learning. In International Conference on Machine Learning, pages 12878–
12889, Virtual, 2021. PMLR.
[74] L. Zong, Q. Xie, J. Zhou, P. Wu, X. Zhang, and B. Xu. Fedcmr: Federated cross-
modal retrieval. In Proc. of the 44th International ACM SIGIR Conference on Re-
search and Development in Information Retrieval, pages 1672–1676, Virtual, July
2021.