生成對抗網路 (GAN) 在數據合成方面取得了令人矚目的性能，並推動 了許多應用程序的發展。然而，GAN 因其雙層優化目標而難以訓練，這會 導致收斂、模式崩潰和梯度消失等問題。在本文中，我們提出了一種新的 生成模型，稱為生成對抗 NTK (GA-NTK)，它具有單層目標。 GA-NTK 保留 了對抗性學習的精神（這有助於生成合理的數據），同時避免了 GAN 的訓
練困難。這是通過將鑑別器建模為具有神經正切核 (NTK-GP) 的高斯過程 來完成的，其訓練動態可以完全用封閉形式的公式來描述。我們分析了通 過梯度下降訓練的 GA-NTK 的收斂行為，並給出了一些收斂的充分條件。 我們還進行了廣泛的實驗來研究 GA-NTK 的優點和局限性，並提出了一些 使 GA-NTK 更實用的技術。

N/A

Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard G.
Baraniuk. The recurrent neural tangent kernel. In Proc. of ICLR,
2021.
Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function
approximation. In Proc. of ICML, 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In Proc. of ICLR, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proc. of ICML, 2017.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov,
and Ruosong Wang. On exact computation with an infinitely wide neural
net. In Proc. of NeurIPS, 2019.
Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong
Wang, and Dingli Yu. Harnessing the power of infinitely wide deep
nets on small-data tasks. In Proc. of ICLR, 2020.
Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and
Carsten Steger. Improving unsupervised defect segmentation by applying structural similarity to autoencoders. In VISIGRAPP, 2019.
Alberto Bietti and Julien Mairal. On the inductive bias of neural
tangent kernels. In Proc. of NeurIPS, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training
for high fidelity natural image synthesis. arXiv preprint arXiv:
1809.11096, 2018.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li.
Mode regularized generative adversarial networks. In Proc. of ICLR,
2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever,
and Pieter Abbeel. Infogan: Interpretable representation learning
by information maximizing generative adversarial nets. In Proc. of
NeurIPS, 2016.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in
differentiable programming. In Proc. of NeurIPS, 2019.
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang
Zeng. Training gans with optimism. In Proc. of ICLR, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database. In Proc. of
CVPR, 2009.
Ishan P. Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multiadversarial networks. In Proc. of ICLR, 2017.
32
Farzan Farnia and Asuman E. Ozdaglar. Do gans always have nash equilibria? In Proc. of ICML, 2020.
Jean-Yves Franceschi, Emmanuel de Bézenac, Ibrahim Ayed, Mickaël Chen,
Sylvain Lamprier, and Patrick Gallinari. A neural tangent kernel
perspective of gans. CoRR, abs/2106.05566, 2021.
Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison.
Deep convolutional networks as shallow gaussian processes. In Proc.
of ICLR, 2019.
Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs,
and Basri Ronen. On the similarity between the laplace and neural
tangent kernels. In Proc. of NeurIPS, 2020.
Arnab Ghosh, Viveka Kulharia, Vinay P. Namboodiri, Philip H. S. Torr,
and Puneet Kumar Dokania. Multi-agent diverse generative adversarial
networks. In Proc. of CVPR, 2018.
Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks.
arXiv preprint arXiv:1701.00160, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative
adversarial nets. In Proc. of NeurIPS, 2014.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining
and harnessing adversarial examples. In Proc. of ICLR, 2015.
Robert M Gower. Convergence theorems for gradient descent, May 2022.
https://gowerrobert.github.io/pdf/M2_statistique_optimisation/grad_
conv.pdf.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard
Schölkopf, and Alexander J. Smola. A kernel two-sample test. J.
Mach. Learn. Res., 2012.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and
Aaron C. Courville. Improved training of wasserstein gans. In Proc.
of NeurIPS, 2017.
Insu Han, Haim Avron, Neta Shoham, Chaewon Kim, and Jinwoo Shin. Random
features for the neural tangent kernel. CoRR, abs/2104.01351, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler,
and Sepp Hochreiter. Gans trained by a two time-scale update rule
converge to a local nash equilibrium. In Proc. of NeurIPS, 2017.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak.
Infinite attention: NNGP and NTK for deep attention networks. In
Proc. of ICML, 2020.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom,
Brandon Tran, and Aleksander Madry. Adversarial examples are not
bugs, they are features. In Proc. of NeurIPS, 2019.
33
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent
kernel: Convergence and generalization in neural networks. In Proc.
of NeurIPS, 2018.
Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proc. of the Royal Society of London. Series A.
Mathematical and Physical Sciences, 186(1007):453–461, 1946.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of
stylegan. In Proc. of CVPR, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images.
Technical report, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit
database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/
mnist, 2, 2010.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew
Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image
super-resolution using a generative adversarial network. In Proc. of
CVPR, 2017.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as
gaussian processes. In Proc. of ICLR, 2018.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman
Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural
networks of any depth evolve as linear models under gradient descent.
In Proc. of NeurIPS, 2019.
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam,
Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus
infinite neural networks: an empirical study. In Proc. of NeurIPS,
2020.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás
Póczos. MMD GAN: towards deeper understanding of moment matching
network. In Proc. of NeurIPS, 2017.
Yujia Li, Kevin Swersky, and Richard S. Zemel. Generative moment
matching networks. In Proc. of ICML, 2015.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning
face attributes in the wild. In Proc. of ICCV, December 2015.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier
Bousquet. Are gans created equal? A large-scale study. In Proc. of
NeurIPS, 2018.
34
Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, and Ming-Hsuan Yang.
Mode seeking generative adversarial networks for diverse image synthesis. In Proc. of CVPR, 2019.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and
Stephen Paul Smolley. Least squares generative adversarial networks.
In Proc. of ICCV, 2017.
Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner,
and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural
networks. In Proc. of ICLR, 2018.
Lars M. Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In Proc. of ICML, 2018.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled
generative adversarial networks. In Proc. of ICLR, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.
Spectral normalization for generative adversarial networks. In Proc.
of ICLR, 2018.
Aryan Mokhtari, Asuman E. Ozdaglar, and Sarath Pattathil. A unified
analysis of extra-gradient and optimistic gradient methods for saddle
point problems: Proximal point approach. In Proc. of AISTATS, 2020.
Vaishnavh Nagarajan and J. Zico Kolter. Gradient descent GAN optimization is locally stable. In Proc. of NeurIPS, 2017.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A Alemi,
Jascha Sohl-Dickstein, and Samuel S Schoenholz. Neural tangents:
Fast and easy infinite neural networks in python. In Proc. of ICLR,
2019a.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang,
Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, and Jascha SohlDickstein. Bayesian deep convolutional networks with many channels
are gaussian processes. In Proc. of ICLR, 2019b.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein,
and Surya Ganguli. Exponential expressivity in deep neural networks
through transient chaos. In Proc. of NeurIPS, 2016.
Guo-Jun Qi. Loss-sensitive generative adversarial networks on lipschitz
densities. Int. J. Comput. Vis., 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial
networks. In Proc. of ICLR, 2016.
Maithra Raghu, Ben Poole, Jon M. Kleinberg, Surya Ganguli, and Jascha
Sohl-Dickstein. On the expressive power of deep neural networks. In
Proc. of ICML, 2017.
35
Alfréd Rényi et al. On measures of entropy and information. In Proc.
of the 4th Berkeley symposium on mathematical statistics and probability, volume 1, 1961.
Mehdi S. M. Sajjadi, Giambattista Parascandolo, Arash Mehrjou, and
Bernhard Schölkopf. Tempered adversarial networks. In Proc. of ICML,
2018.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec
Radford, and Xi Chen. Improved techniques for training gans. In
Proc. of NeurIPS, 2016.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha SohlDickstein. Deep information propagation. In Proc. of ICLR, 2017.
Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich-Keil, Jonathan
Ragan-Kelley, Ludwig Schmidt, and Benjamin Recht. Neural kernels
without tangents. In Proc. of ICML, 2020.
Kiran Koshy Thekumparampil, Prateek Jain, Praneeth Netrapalli, and
Sewoong Oh. Efficient algorithms for smooth minimax optimization.
In Proc. of NeurIPS, 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating
videos with scene dynamics. In Proc. of NeurIPS, 2016.
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and
Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference
on computer vision and pattern recognition, pp. 8798–8807, 2018.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image
quality assessment: from error visibility to structural similarity.
IEEE transactions on image processing, 2004.
Li Xu, Jimmy S. J. Ren, Ce Liu, and Jiaya Jia. Deep convolutional
neural network for image deconvolution. In Proc. of NeurIPS, 2014.
Greg Yang. Scaling limits of wide neural networks with weight sharing:
Gaussian process behavior, gradient independence, and neural tangent
kernel derivation. CoRR, abs/1902.04760, 2019a.
Greg Yang. Tensor programs I: wide feedforward or recurrent neural
networks of any architecture are gaussian processes. CoRR, abs/
1910.12478, 2019b.
Amir Zandieh, Insu Han, Haim Avron, Neta Shoham, Chaewon Kim, and
Jinwoo Shin. Scaling neural tangent kernels via sketching and random
features. In Proc. of NeurIPS, 2021.