指紋辨識技術在近年來已經廣泛運用在行動裝置上，進行更高安全的身份辨識且維持便利性，並且能確保行動支付、資料安全保護。

隨著智慧型手機螢幕面積占比更高的趨勢，許多廠商為了輕量化，會將指紋感測器的尺寸縮小並置於手機側邊。但是，由於較小的感測器所截取到的指紋覆蓋範圍有限，為指紋辨識帶來更大的挑戰。此外，水滴、汗水與污漬都會導致指紋影像變得模糊，進而導致辨識失敗並提高了驗證難度。

目前有許多研究專注在局部濕指紋影像的還原，並針對紋路、結構細節進行影像增強與恢復。然而，這些方法雖然還原了圖像細節，卻會丟失大部分的身份辨識資訊，進而導致誤判率（FRR）無法有效降低。

為了解決這個問題，我們提出了 IPFormer，該模型使用 Transformer 架構搭配 U 型架構，結合辨識身份特徵擷取與提示學習，還原模糊與潮濕的局部指紋。我們的方法旨在保留身份辨識資訊下，同時還原紋路與結構細節，進而讓還原指紋依舊可被辨識，降低誤判率（FRR）。我們提出的辨識特徵提取模組，可以有效的從輸入指紋中提取身份相關特徵，並且融合進入模型解碼器架構中，讓還原指紋可以同時含有身份特徵資訊。我們使用提示學習，針對多種等級的濕度與模糊退化進行學習，可讓模型適應多種退化種類的指紋還原，增加泛化能力。為了解決在現實世界採集指紋數據的不對齊問題，我們使用 CycleGAN 生成對齊的濕指紋數據，提高模型訓練穩定度與表現。通過真實世界數據下的實驗，我們的方法超越現有方法，並在 FRR 評估上取得 11.89% 的結果。

本篇研究提出了以 Transformer 為基底的神經網路 IPFormer，可以在還原局部濕指紋結構細節特同時保留身份辨識資訊。我們的實驗結果呈現了在局部濕指紋修復的優勢，降低了 FRR 並且超越其他研究的成果。與其他指紋去噪研究相比，我們的 IPFormer 比 FPD-M-Net 降低了 25.50%，比 DenseUNet 降低了 17.28%。此外，與其他局部濕指紋去噪研究相比，我們比 FPN-ResUNet 降低了 13.34%，比 PGT-Net 降低了 11.33%。

In recent years, fingerprint recognition technology has been widely used in mobile devices to enhance security while maintaining convenience, ensuring the safety of mobile payments and data protection.

As the trend of increasing smartphone screen size continues, many manufacturers are reducing the size of fingerprint sensors and positioning them on the side of the device to achieve a lighter design. However, the smaller size of these sensors limits the fingerprint coverage area, making fingerprint recognition more challenging. Additionally, factors such as water droplets, sweat, and smudges can blur the fingerprint image, resulting in recognition failures and further complicating the verification process.

Current research has focused on restoring partial wet fingerprint images and enhancing image details such as ridges and structures. However, while these methods restore image details, they often lose critical identification information, leading to an ineffective reduction in the False Rejection Rate (FRR).

To address this issue, we propose IPFormer, a model that utilizes a Transformer architecture combined with a U-shaped structure. It integrates recognition identity feature extraction and prompt learning to restore blurred or wet partial fingerprints. Our approach aims to retain identification information while restoring ridge and structural details, allowing the restored fingerprints to remain identifiable and reducing the FRR. The proposed feature extraction module effectively extracts identity-related features from the input fingerprint and integrates them into the model's decoder architecture, ensuring that the restored fingerprint retains identity information. We employ prompt learning to train the model to handle various degradation types, such as different levels of humidity and blurriness, enhancing its generalization ability. To address misalignment issues in real-world fingerprint data collection, we use CycleGAN to generate aligned wet fingerprint data, improving the model's training stability and performance. Through experiments on real-world data, our method outperforms existing methods and achieves an FRR result of 11.89%.

This study presents IPFormer, a Transformer-based neural network that restores partial wet fingerprint structure details while preserving identification information. Our experimental results demonstrate the advantages of our approach in restoring partial wet fingerprints, reducing the FRR, and surpassing other research outcomes. Compared to other fingerprint denoising studies, IPFormer reduces FRR by 25.50% compared to FPD-M-Net and by 17.28% compared to DenseUNet. Additionally, compared to other partial wet fingerprint denoising studies, our method reduces FRR by 13.34% compared to FPN-ResUNet and by 11.33% compared to PGT-Net.

[1] S.AdigaVandJ.Sivaswamy,“Fpd-m-net:Fingerprintimagedenoisingandinpaintingus- ing m-net based convolutional neural networks,” in Inpainting and denoising challenges, pp. 51–61, Springer, 2019.
[2] P.Qian,A.Li,andM.Liu,“Latentfingerprintenhancementbasedondenseunet,”in2019 international conference on biometrics (ICB), pp. 1–6, IEEE, 2019.
[3] A.-T. Hsieh, C.-T. Chiu, T.-C. Chen, M.-H. Hsu, and L. Wenyong, “Feature points based residual unet with nonlinear decay rate for partial wet fingerprint restoration and recogni- tion,” in 2024 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5, IEEE, 2024.
[4] Y.-T. Li, C.-T. Chiu, A.-T. Hsieh, M.-H. Hsu, L. Wenyong, and J.-M. Hsu, “Pgt-net: Pro- gressive guided multi-task neural network for small-area wet fingerprint denoising and recognition,” arXiv preprint arXiv:2308.07024, 2023.
[5] N. D. S. Cunha, H. M. Gomes, and L. V. Batista, “Residual m-net with frequency-domain loss function for latent fingerprint enhancement,” in 2022 35th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), vol. 1, pp. 198–203, IEEE, 2022.
[6] Y. Zhu, X. Yin, and J. Hu, “Fingergan: a constrained fingerprint generation scheme for latent fingerprint enhancement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8358–8371, 2023.
[7] Z. Huang, J. Zhang, and H. Shan, “When age-invariant face recognition meets face age synthesis: A multi-task learning framework,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7282–7291, 2021.
[8] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690–4699, 2019.
[9] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international confer- ence on computer vision, pp. 2223–2232, 2017.
[10] G. Fang, S. N. Srihari, H. Srinivasan, and P. Phatak, “Use of ridge points in partial finger- print matching,” in Biometric Technology for Human Identification IV, vol. 6539, pp. 115– 123, SPIE, 2007.
[11] F. Zhang, S. Xin, and J. Feng, “Combining global and minutia deep features for partial high-resolution fingerprint matching,” Pattern Recognition Letters, vol. 119, pp. 139–147, 2019.
[12] F. Chen, M. Li, and Y. Zhang, “A fusion method for partial fingerprint recognition,” In- ternational Journal of Pattern Recognition and Artificial Intelligence, vol. 27, no. 06, p. 1356009, 2013.
[13] T.-Y. Jea and V. Govindaraju, “A minutia-based partial fingerprint recognition system,” Pattern recognition, vol. 38, no. 10, pp. 1672–1684, 2005.
[14] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International journal of computer vision, vol. 60, pp. 91–110, 2004.
[15] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” in Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I 9, pp. 404–417, Springer, 2006.
[16] S.TandonandA.Namboodiri,“Transformerbasedfingerprintfeatureextraction,”in2022 26th International Conference on Pattern Recognition (ICPR), pp. 870–876, IEEE, 2022.
[17] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Effi- cient transformer for high-resolution image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5728–5739, 2022.
[18] T.Wolf,L.Debut,V.Sanh,J.Chaumond,C.Delangue,A.Moi,P.Cistac,T.Rault,R.Louf, M. Funtowicz, et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45, 2020.
[19] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., “Huggingface’s transformers: State-of-the-art natural lan- guage processing,” arXiv preprint arXiv:1910.03771, 2019.
[20] K. S. Kalyan, A. Rajasekharan, and S. Sangeetha, “Ammus: A survey of transformer-based pretrained models in natural language processing,” arXiv preprint arXiv:2108.05542, 2021.
[21] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” Advances in neural information processing systems, vol. 34, pp. 15908–15919, 2021.
[22] A. DOSOVITSKIY, “An image is worth 16x16 words: Transformers for image recogni- tion at scale,” arXiv preprint arXiv:2010.11929, 2020.
[23] H.Touvron,M.Cord,M.Douze,F.Massa,A.Sablayrolles,andH.Jégou,“Trainingdata- efficient image transformers & distillation through attention,” in International conference on machine learning, pp. 10347–10357, PMLR, 2021.
[24] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” in Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 558–567, 2021.
[25] W.Wang,E.Xie,X.Li,D.-P.Fan,K.Song,D.Liang,T.Lu,P.Luo,andL.Shao,“Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 568–578, 2021.
[26] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Advances in neural information processing systems, vol. 34, pp. 12077–12090, 2021.
[27] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al., “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6881–6890, 2021.
[28] Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. U. Li, “A general u-shaped transformer for image restoration. arxiv 2021,” arXiv preprint arXiv:2106.03106.
[29] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restora- tion using swin transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 1833–1844, 2021.
[30] X. Chen, X. Wang, W. Zhang, X. Kong, Y. Qiao, J. Zhou, and C. Dong, “Hat: Hybrid attention transformer for image restoration,” arXiv preprint arXiv:2309.05239, 2023.
[31] H. Zhao, Y. Gou, B. Li, D. Peng, J. Lv, and X. Peng, “Comprehensive and delicate: An efficient transformer for image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14122–14132, 2023.
[32] A.Dudhane,S.W.Zamir,S.Khan,F.S.Khan,andM.-H.Yang,“Burstormer:Burstimage restoration and enhancement transformer,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5703–5712, IEEE, 2023.
[33] Y.N.Dauphin,A.Fan,M.Auli,andD.Grangier,“Languagemodelingwithgatedconvo- lutional networks,” in International conference on machine learning, pp. 933–941, PMLR, 2017.
[34] S. Anwar and N. Barnes, “Densely residual laplacian super-resolution,” IEEE Transac- tions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp. 1192–1204, 2020.
[35] A. Dudhane, S. W. Zamir, S. Khan, F. S. Khan, and M.-H. Yang, “Burst image restoration and enhancement,” in Proceedings of the ieee/cvf Conference on Computer Vision and Pattern Recognition, pp. 5759–5768, 2022.
[36] S.W.Zamir,A.Arora,S.Khan,M.Hayat,F.S.Khan,M.-H.Yang,andL.Shao,“Learning enriched features for real image restoration and enhancement,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16, pp. 492–511, Springer, 2020.
[37] Y.Zhang,K.Li,K.Li,L.Wang,B.Zhong,andY.Fu,“Imagesuper-resolutionusingvery deep residual channel attention networks,” in Proceedings of the European conference on computer vision (ECCV), pp. 286–301, 2018.
[38] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2472–2481, 2018.
[39] K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 12, pp. 2341–2353, 2010.
[40] J. Kopf, B. Neubert, B. Chen, M. Cohen, D. Cohen-Or, O. Deussen, M. Uyttendaele, and D. Lischinski, “Deep photo: Model-based photograph enhancement and viewing,” ACM transactions on graphics (TOG), vol. 27, no. 5, pp. 1–10, 2008.
[41] T. Michaeli and M. Irani, “Nonparametric blind super-resolution,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 945–952, 2013.
[42] R. Timofte, V. De Smet, and L. Van Gool, “Anchored neighborhood regression for fast example-based super-resolution,” in Proceedings of the IEEE international conference on computer vision, pp. 1920–1927, 2013.
[43] A. Abuolaim and M. S. Brown, “Defocus deblurring using dual-pixel data,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Pro- ceedings, Part X 16, pp. 111–126, Springer, 2020.
[44] S.-J. Cho, S.-W. Ji, J.-P. Hong, S.-W. Jung, and S.-J. Ko, “Rethinking coarse-to-fine ap- proach in single image deblurring,” in Proceedings of the IEEE/CVF international con- ference on computer vision, pp. 4641–4650, 2021.
[45] O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, “Deblurgan-v2: Deblurring (orders-of- magnitude) faster and better,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 8878–8887, 2019.
[46] Z. Yue, Q. Zhao, L. Zhang, and D. Meng, “Dual adversarial network: Toward real-world noise removal and noise generation,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16, pp. 41–58, Springer, 2020.
[47] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi- stage progressive image restoration,” in Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 14821–14831, 2021.
[48] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 6360–6376, 2021.
[49] R. Mehta and J. Sivaswamy, “M-net: A convolutional neural network for deep brain struc- ture segmentation,” in 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), pp. 437–440, Ieee, 2017.
[50] Z. Ming, J. Xia, M. M. Luqman, J.-C. Burie, and K. Zhao, “Dynamic multi-task learning for face recognition with facial expression,” arXiv preprint arXiv:1911.03281, 2019.
[51] X.YinandX.Liu,“Multi-taskconvolutionalneuralnetworkforpose-invariantfacerecog- nition,” IEEE Transactions on Image Processing, vol. 27, no. 2, pp. 964–975, 2017.
[52] C. Ding, C. Xu, and D. Tao, “Multi-task pose-invariant face recognition,” IEEE Transac- tions on image Processing, vol. 24, no. 3, pp. 980–993, 2015.
[53] Y. Fu, G. Guo, and T. S. Huang, “Age synthesis and estimation via faces: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 32, no. 11, pp. 1955–1976, 2010.
[54] Y. Sun, J. Tang, X. Shu, Z. Sun, and M. Tistarelli, “Facial age synthesis with label distribution-guided generative adversarial network,” IEEE transactions on information forensics and security, vol. 15, pp. 2679–2691, 2020.
[55] Y. Shen, P. Luo, J. Yan, X. Wang, and X. Tang, “Faceid-gan: Learning a symmetry three- player gan for identity-preserving face synthesis,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 821–830, 2018.
[56] F. Mokhayeri, K. Kamali, and E. Granger, “Cross-domain face synthesis using a control- lable gan,” in Proceedings of the IEEE/CVF winter conference on applications of com- puter vision, pp. 252–260, 2020.
[57] Y. Shen, B. Zhou, P. Luo, and X. Tang, “Facefeat-gan: a two-stage approach for identity- preserving face synthesis,” arXiv preprint arXiv:1812.01288, 2018.
[58] A. Vaswani, “Attention is all you need,” arXiv preprint arXiv:1706.03762, 2017.
[59] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” arXiv preprint
arXiv:1606.08415, 2016.
[60] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint
arXiv:1607.06450, 2016.
[61] M. Jaderberg, K. Simonyan, A. Zisserman, et al., “Spatial transformer networks,” Ad-
vances in neural information processing systems, vol. 28, 2015.
[62] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention mod- ule,” in Proceedings of the European conference on computer vision (ECCV), pp. 3–19, 2018.
[63] X.DongandJ.Shen,“Tripletlossinsiamesenetworkforobjecttracking,”inProceedings of the European conference on computer vision (ECCV), pp. 459–474, 2018.
[64] V.Potlapalli,S.W.Zamir,S.H.Khan,andF.ShahbazKhan,“Promptir:Promptingforall- in-one image restoration,” Advances in Neural Information Processing Systems, vol. 36, 2024.
[65] N. Ding, S. Hu, W. Zhao, Y. Chen, Z. Liu, H.-T. Zheng, and M. Sun, “Openprompt: An open-source framework for prompt-learning,” arXiv preprint arXiv:2111.01998, 2021.
[66] M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, “Maple: Multi-modal prompt learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113–19122, 2023.
[67] K.Zhou,J.Yang,C.C.Loy,andZ.Liu,“Conditionalpromptlearningforvision-language models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16816–16825, 2022.
[68] J.Qi,J.Du,S.M.Siniscalchi,X.Ma,andC.-H.Lee,“Onmeanabsoluteerrorfordeepneu- ral network based vector-to-vector regression,” IEEE Signal Processing Letters, vol. 27, pp. 1485–1489, 2020.
[69] D. G. Low, “Distinctive image features from scale-invariant keypoints,” Journal of Com- puter Vision, vol. 60, no. 2, pp. 91–110, 2004.
[70] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention– MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, pro- ceedings, part III 18, pp. 234–241, Springer, 2015.
[71] Z. Shen, Y. Xu, and G. Lu, “Cnn-based high-resolution fingerprint image enhancement for pore detection and matching,” in 2019 IEEE Symposium Series on Computational In- telligence (SSCI), pp. 426–432, IEEE, 2019.