遠程光體積變化描計圖（rPPG）是一種非接觸式方法，用於臉部影片計算生理信號。如果沒有大量的監督數據集，那麼學習一個可靠的rPPG預估模型會變得非常具有挑戰性。因此，我們認為把數據集增大以讓模型學習的更好這件事對於計算rPPG信號至關重要。在本文中，我們提出了一種新穎的多任務學習方式，在學習rPPG估計模型的同時增加訓練數據集。我們設計了三個聯合學習網絡：(1) rPPG估計網絡：從臉部影片估計rPPG信號。 (2) 圖像到影片網絡：根據原始圖片和指定的rPPG信號生成影片。 (3) 影片到影片網絡：根據原始影片和指定的rPPG信號生成影片。我們測試在三個數據集：COHFACE，UBFC-RPPG和PURE上，其實驗結果表明我們的方法成功生成了與原始影片外表相似度極高但不同rPPG信號的影片，並且預測rPPG信號的效果大大優於現有方法。

Remote photoplethysmography (rPPG) is a contactless method for estimating physiological signals from facial videos. Without large supervised datasets, learning a robust rPPG estimation model is extremely challenging. Instead of merely focusing on model learning, we believe data augmentation may be of greater importance for this task. In this thesis, we propose a novel multi-task learning framework to simultaneously augment training data while learning the rPPG estimation model. We design three joint-learning networks: rPPG estimation network, Image-to-Video network, and Video-to-Video network, to estimate rPPG signals from face videos, to generate synthetic videos from a source image and a specified rPPG signal, and to generate synthetic videos from a source video and a specified rPPG signal, respectively. Experimental results on three benchmark datasets, COHFACE, UBFC, and PURE, show that our method successfully generates photo-realistic videos and significantly outperforms existing methods with a large margin.

[1] Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in International Conference on Computer Vision (ICCV), 2019.
[2] X. Niu, H. Han, S. Shan, and X. Chen, “Synrhythm: Learning a deep heart rate estimator from general to specific,” in 2018 24th International Conference on Pattern Recognition (ICPR), pp. 3580–3585, Aug 2018.
[3] X. Li, I. Alikhani, J. Shi, T. Seppanen, J. Junttila, K. Majamaa-Voltti, M. Tulppo, and G. Zhao, “The obf database: A large face video database for remote physiological signal measurement and atrial fibrillation detection,” in 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pp. 242– 249, May 2018.
[4] Z. Yu, X. Li, and G. Zhao, “Recovering remote photoplethysmograph signal from facial videos using spatio-temporal convolutional networks,” CoRR, vol. abs/1905.02419, 2019.
[5] W.ChenandD.McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in The European Conference on Computer Vision (ECCV), pp. 356–373, Springer International Publishing, 2018.
[6] W. Chen and D. J. McDuff, “Deepmag: Source specific motion magnification using gradient ascent,” CoRR, vol. abs/1808.03338, 2018.
[7] Z.-K. Wang, Y. Kao, and C.-T. Hsu, “Vision-based Heart Rate Estimation via a Two-stream CNN,” in 2019 IEEE International Conference on Image Processing (ICIP), pp. 3327–3331, Sep 2019.
[8] G. de Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Transactions on Biomedical Engineering, vol. 60, pp. 2878–2886, Oct 2013.
[9] J. Hernandez-Ortega, J. Fierrez, A. Morales, and P. Tome, “Time Analysis of Pulse-Based Face Anti-Spoofing in Visible and NIR,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[10] Y.Liu, A.Jourabloo, and X.Liu, “LearningDeepModelsforFaceAnti-Spoofing: Binary or Auxiliary Supervision,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 389–398, Aug 2018.
[11] S. Liu, X. Lan, and P. C. Yuen, “Remote photoplethysmography correspondence feature for 3d mask face presentation attack detection,” 09 2018.
[12] S.Liu,P.C.Yuen,S.Zhang,andG.Zhao,“3dmaskfaceanti-spoofingwithremote photoplethysmography,” vol. 9911, pp. 85–100, 10 2016.
[13] R. Spetlík, V. Franc, J. Cech, and J. Matas, “Visual Heart Rate Estimation with Convolutional Neural Network,” in Proceedings of British Machine Vision Conference, 2018.
[14] S. Bobbia, R. Macwan, Y. Benezeth, A. Mansouri, and J. Dubois, “Unsupervised skin tissue segmentation for remote photoplethysmography,” Pattern Recognition Letters, Oct 2017.
[15] R.Stricker,S.Müller,andH.-M.Gross,“Non-contactvideo-basedpulseratemea- surement on a mobile service robot,” vol. 2014, pp. 1056–1062, Aug 2014.
[16] G. Heusch, A. Anjos, and S. Marcel, “A reproducible study on remote heart rate measurement,” CoRR, vol. abs/1709.00962, 2017.
[17] Y. Benezeth, S. Bobbia, K. Nakamura, R. Gomez, and J. Dubois, “Probabilistic signal quality metric for reduced complexity unsupervised remote photoplethysmography,” pp. 1–5, May 2019.
[18] P. Li, K. N. Yannick Benezeth, R. Gomez, and F. Yang, “Model-based region of interest segmentation for remote photoplethysmography,” in 14th International Conference on Computer Vision Theory and Applications, pp. 383–388, Feb 2019.
[19] X. Li, J. Chen, G. Zhao, and M. Pietikäinen, “Remote heart rate measurement from face videos under realistic situations,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271, June 2014.
[20] R. Macwan, Y. Benezeth, and A. Mansouri, “Heart rate estimation using remote photoplethysmography with multi-objective optimization,” Biomedical Signal Processing and Control, vol. 49, pp. 24–33, March 2019.
[21] R. Macwan, S. Bobbia, Y. Benezeth, J. Dubois, and A. Mansouri, “Periodic variance maximization using generalized eigenvalue decomposition applied to remote photoplethysmography estimation,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1413–14138, June 2018.
[22] W. Wang, S. Stuijk, and G. de Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE Transactions on Biomedical Engineering, vol. 63, pp. 1974–1984, Sep 2016.
[23] Y.-Y. Tsou, Y.-A. Lee, C.-T. Hsu, and S.-H. Chang, “Siamese-rppg network: Remote photoplethysmography signal estimation from face video,” in The 35th ACM/SIGAPP Symposium on Applied Computing (SAC’20), 2020.
[24] N.Dvornik, J.Mairal, and C.Schmid, “Modeling visual context is key to augmenting object detection datasets,” in The European Conference on Computer Vision (ECCV), Sep 2018.
[25] M. Frid-Adar, E. Klang, M. Amitai, J. Goldberger, and H. Greenspan, “Synthetic data augmentation using gan for improved liver lesion classification,” pp. 289– 293, April 2018.
[26] Y. Qiu, Y. Liu, J. Arteaga-Falconi, H. Dong, and A. E. Saddik, “Evm-CNN: Real-time contactless heart rate estimation from facial video,” IEEE Transactions on Multimedia, vol. 21, pp. 1778–1787, July 2019.
[27] D. E. King, “Dlib-ml: A machine learning toolkit,” Journal of Machine Learning Research, vol. 10, pp. 1755–1758, 2009.