A Scalable and Programmable Continuous Restricted Boltzmann Machine in VLSI

簡易檢索 / 詳目顯示

回結果列表

研究生：	盧峙丞 Lu, Chih-Cheng
論文名稱：	A Scalable and Programmable Continuous Restricted Boltzmann Machine in VLSI 連續值局限型波茲曼模型積體電路系統之模組化及程式化設計
指導教授：	陳新 Chen, Hsin
口試委員:
學位類別：	博士 Doctor
系所名稱：	電機資訊學院 - 電子工程研究所 Institute of Electronics Engineering
論文出版年：	2010
畢業學年度：	98
語文別：	英文
論文頁數：	119
中文關鍵詞：	超大型積體電路、雜訊、機率型模型、波茲曼模型、可程式化及模組化系統、隨機系統
外文關鍵詞：	VLSI Implementation, Noise, Probabilistic Model, Boltzamnn Machine, Scalable and Programmable Systems, Stochastic systems
相關次數：	點閱：2 下載：0
分享至:	分享至facebook 分享至twitter

查詢本校圖書館目錄查詢臺灣博碩士論文知識加值系統勘誤回報

Probabilistic models use stochasticity to generalise the natural variability of data, and have been shown promising for reasoning biomedical data or for solving weakly-constrained problems such as pattern recognition. Realising probabilistic models in the Very-Large-Scale-Integration (VLSI) is thus attractive for the application like intelligent sensor fusion in implantable devices. However, only a few probabilistic models are amenable to VLSI implementation, and most of which relies greatly on precise computation of Bayesian rules or vector products, which becomes infeasible as transistor noise and hardware non-ideality grow.

This study presents the VLSI system of a scalable and programmable Continuous Restricted Boltzmann Machine (CRBM), a probabilistic model proved to be both useful and hardware-amenable. The probabilistic VLSI system utilises noise to induce continuous-valued stochastic behaviour in VLSI, and based on the CRBM algorithm, the stochasticity in VLSI can be adapted to represent the natural variability in data for pattern recognition. Moreover, as the noise injection makes a circuit’s inputs decide its “output probability” only, the stochasticity in VLSI would further discourage the propagation of noise and computation errors. The full system containing 10 stochastic units, and 25 adaptable connections has been designed and fabricated with the TSMC 0.35□m 2P4M CMOS process. Modular design has been employed such that the system can expand its size by inter-connecting multiple chips, extending its application to reasoning high-dimensional, complex data, e.g. real biomedical signals. The system’s capability of modelling and classifying real biomedical data is examined in the context of sorting neuronal spikes. Furthermore, to enhance the system’s performance and flexibility, parameters in the system are not only adaptable by chip-in-a-loop training, but also externally-programmable through a computer.

The capability of the system to model high-dimensional biomedical data, as well as the feasibility of using noise-induced stochastic behaviour to enhance the robustness of analogue computation will be examined and discussed. The latter is especially important when the VLSI technology heads towards the deep-sub-micron era, wherein transistor noise and hardware nonidealities are expected to increase dramatically. The feasibility will then support the suggestion that transistor noise could be used rather than suppressed in future computation, as is what biological neurons have being doing.

機率型演算法利用其隨機性來概括信號資料的變異，已經廣泛的應用在生醫訊號分析與解決如圖形辨識等問題上，所以實現在超大型積體電路上之機率型演算法對於植入式之智慧型感測資料融合系統或生醫檢測儀器是很有潛力的。然而，適合實現在超大型積體電路上的機率型演算法並不多，且大部分的機率型演算法需很準確的計算貝斯定理(Bayesian rule)或向量乘積(vector product)，如此，當元件雜訊或是電路非線性度增加時，硬體實現的困難度便會隨之提升。

本研究提出一個可程式化及模組化之「連續值局限型波茲曼模型」 (Continuous Restricted Boltzmann Machine, CRBM)，此模型已證明適合於硬體。此機率型之超大型積體電路系統利用雜訊注入來產生機率行為，且基於此CRBM 演算法，在超大型積體電路上之隨機性可以被訓練來表示資料感測信號之變異並用在圖形之辨識上。由於電路的輸入只決定其“輸出之機率”，此特性可以阻擋雜訊與計算之錯誤往下一級傳遞。系統使用TSMC 0.35□m 2P4M標準製程實現，包含十個隨機型運算單元與二十五個可調整權重參數之突觸連結。模組化設計使得系統可以多晶片相互連結，擴充運算單元數目來處理高維度與複雜之資料，如真實世界之生醫信號。而系統之重建與生醫信號辨識能力則透過分類神經脈衝展示出來。另外，為了增加系統效能與彈性，系統參數不但可以經由外部電腦來寫入，也可以經由晶片迴路訓練來設定。

在此，我們將會探討系統處理高維生醫信號之能力，並且討論利用雜訊注入之隨機行為來提高類比運算穩健性的可能性。尤其是機率行為，在未來超大型積體電路製造技術邁入深次微米時代會顯得更重要，屆時元件雜訊跟電路非線性度預期會更嚴重。就如同真實生物神經訊號的運算方式，雜訊可以使用在運算過程而非抑制運算的準確度。

Abstract    i
摘要    ii
Acknowledgement    iii
Contents    iv
Chapter 1  Introduction    1
1 Motivation    1
2 Contribution to Knowledge    4
3 Chapter Layout    5
Chapter 2  Literature Review    7
1 Hardware-amenable Probabilistic Models    7
1.1 The Bump Mixture Model    7
1.2 The Boltzmann Machine    11
1.3 The Diffusion Network    13
1.4 The Restricted Boltzmann Machine    16
1.5 The Product of Expert/Restricted Boltzmann Machine    17
2 The CRBM    19
2.1 Introduction    19
2.2 Experiment with artificial data    22
2.3 Modelling real-world data    25
3 Summary for Probabilistic Neural Computation    25
4 The VLSI Implementation of the CRBM    26
4.1 A full system on chip    27
4.2 Minimising-contrastive-divergence training in VLSI    28
4.3 Summary    29
5 Artificial Neural Network in VLSI    30
5.1 Digital versus analog VLSI implementation    30
5.2 A fully analog self-learning neural network in VLSI    31
6 Summary    33
Chapter 3  The Effects of Hardware Non-idealities on the CRBM System    35
1 A Quantitative Index for the Performance of the CRBM    35
2 Quantisation Effects    37
2.1 Required resolution for modelling artificial data    38
2.2 Required resolution for modelling biomedical data    39
3 Maximum Offset Tolerable by the CRBM Model    42
3.1 Modelling artificial data with offsets    43
3.2 Modelling real heartbeat data with offsets    45
3.3 Tolerable offsets for different training strategies    47
4 The Training Circuit Based on Dynamic Current Mirrors    48
4.1 Circuit description    48
4.2 Measurement results    53
5 Modelling Data with Uniformly-Distributed Offsets    59
6 Summary    63
Chapter 4  The Robustness against Environemntal Noise    65
1 The Robustness of a CRBM System against Environmental Noise    65
1.1 The environmental interferences on a CRBM system    65
1.2 Modelling artificial data in the presence of environmental noise    66
1.3 Modelling biomedical data in the presence of environmental noise    68
1.4 Noise-enhanced robustness in the CRBM system    70
2 Reconstructing Data Distribution with External Noise    72
2.1 The updating circuit    72
2.1 Modelling artificial data in the presence of external noise    77
3 Classifying Data under the Existence of External Noise    79
4 Summary    81
Chapter 5  The Scalable and Programmable CRBM in VLSI    82
1 Architecture    82
2 Building Block    84
2.1 Continuous stochastic neurons in VLSI    85
2.2 Programmable parameter array    91
3 Digital Control    92
4 Summary    94
Chapter 6  Data Modelling and Classification with Modular CRBM on VLSI    95
1 The Measurement Setup    95
2 Regenerating Continuous-Valued Data Distributions    95
2.1 Regenerating two-dimensional data with a symmetric distribution    96
2.2 Regenerating two-dimensional data with a non-symmetric distribution    97
2.3 Regenerating three-dimensional data with a symmetric distribution    98
3 Classifying Artificial, Three-Dimensional Data    99
4 Modelling and Classifying Real Biomedical Data    101
5 Summary    105
Chapter 7    106
1 Summary    106
2 Conclusion    107
2 Future Work    109
References    111
Publication List    118

                                

[1]A. Schwartz, "Cortical neural prosthetics," Annual review neuroscience, vol. 27, pp. 487-507, July 2004.
[2]K. Wise, D. J. Anderson, J. F. Hetke, D. R. Kipke, and K. Najfi, "Wireless implantable microsystems: high-density electronic interfaces to the nervous system," Proceedings of the IEEE, vol. 92, no. 1, pp. 76-97, 2004.
[3]Z. S. Zumsteg, R. E. Ahmed1, G. Santhanam, K. V. Shenoy, and T. H. Meng, "Power feasibility of implantable digital spike-sorting circuits for neural prosthetic systems," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 13, no. 3, pp. 272-279, 2004.
[4]N. H. Hamid, A. F. Murray and S. Roy, "Time-domain modeling of low-frequency noise in deep-submicrometer MOSFET," IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 55, no. 1, pp. 245-257, 2008.
[5]E. Johannessen, L. Wang, C. Wyse, D. R. S. Cumming, and J. M. Cooper, "Biocompatibility of a lab-on-a-pill sensor in artificial gastrointestinal environments," IEEE Transactions on Biomedical Engineering, vol. 53, no.11, pp. 2333-2340, 2006.
[6]T. B. Tang, E. Johannessen, L. Wang, A. Astaras, M. Ahmadian, A. F. Murray, J. M. Cooper, S. P. Beaumont, B. W. Flynn, and D. S. R. Cumming, "Towards a miniature wireless integrated multisensor microsystem for industrial and biomedical applications," IEEE Sensors Journal: Special Issue on Integrated Multisensor Systems and Signal Processing, vol. 2, no. 6, pp. 628-635, 2002.
[7]H. Chen and A. Murray, "Continuous restricted Boltzmann machine with an implementable training algorithm," IEE Proceedings-Vision Image and Signal Processing, vol. 150, no. 3, pp. 153-159, 2003.
[8]R. Genov and G. Cauwenberghs, "Kerneltron: support vector "machine" in silicon," IEEE Transactions on Neural Networks, vol. 14, pp. 1426-1434, 2003.
[9]D. Hsu, S. Bridges, M. Figueroa, and C. Diorio, "Adaptive quantization and density estimation in silicon," in Advances in Neural Information Processing Systems (NIPS02), vol. 15, 2003.
[10]B. E. S. Akgul, L. N. Chakrapani, P. Korkmaz, and K. V. Palem, "Probabilistic cmos technology: a survey and future directions," The IFIP International Conference on Very Large Scale Integration (VLSI-SoC), pp. 1-6, 2006.
[11]B. Brown and H. Card, "Stochastic neural computation I: computational elements," IEEE Transactions on Computers, vol. 50, no. 9, pp. 891-905, 2001.
[12]B. Brown and H. Card, "Stochastic neural computation II: soft competitive Learning," IEEE Transactions on Computers, vol. 50, no. 9, pp. 906-920, 2001.
[13]L. N. Chakrapani, B. E. S. Akgul, S. Cheemalavagu, P. Korkmaz, K. V. Palem, and B. Seshasayee, "Ultra-efficient (embedded) SOC architectures based on probabilistic cmos (PCMOS) technology," The Ninth Design Automation and Test in Europe (DATE), pp. 1110-1115, 2006.
[14]L. N. Chakrapani, P. Korkmaz, B. E. S. Akgul, and K. V. Palem, "Probabilistic system-on-a-chip architectures," ACM Transactions on Design Automation of Electronic Systems (TODAES), vol. 12, no. 29, 2007.
[15]J. Alspector, B. Gupta, and R. B. Allen, "Performance of a stochastic learning microchip," in Advances in Neural Information Processing Systems (NIPS88), vol. 1, pp. 748-760, 1989.
[16]J. Alspector, A. Jayakumar, and S. Luma., "Experimental evaluation of learning in a neural microsystem," in Advances in Neural Information Processing Systems (NIPS91), vol. 4, pp. 871-878, 1992.
[17]P. Smolensky, "Information processing in dynamical systems: Foundations of harmony theory," Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, pp. 194-281, 1986.
[18]G. Hinton, "Products of experts," The Ninth International Conference on Artificial Neural Networks (ICANN’99), pp. 1-6, 1999.
[19]G. Hinton, "Training products of experts by minimizing contrastive divergence," Neural Computation, vol. 14, pp. 1771-1800, 2002.
[20]A. Murray, "Novelty detection using products of simple experts--a potential architecture for embedded systems," Neural Networks, vol. 14, pp. 1257-1264, 2001.
[21]J. Movellan and J. McClelland, "Learning continuous probability distributions with symmetric diffusion networks," Cognitive Science, vol. 17, pp. 463-496, 1993.
[22]J. R. Movellan, P. Mineiro, and R. J. Williams, "A Monte Carlo EM Approach for partially observable diffusion processes: theory and applications to neural networks," Neural Computation, vol. 14, pp. 1507-1544, 2002.
[23]H. Chen, P. C. Fleury, and A. F. Murray, "Continuous-valued probabilistic behavior in a VLSI generative model," IEEE Transactions on Neural Networks, vol. 17, no. 3, p. 755, 2006.
[24]C. C. Lu, C. C. Li, and H. Chen, "How robust is a probabilistic neural VLSI system against environmental noise," Artificial Neural Networks in Pattern Recognition, pp. 44-53, 2008.
[25]G. Cauwenberghs and M. Bayoumi, Learning on silicon: Adaptive VLSI neural systems, Kluwer Academic Publishers, 1999.
[26]G. Hinton and T. Sejnowski, "Learning and relearning in Boltzmann machines," MIT Press, Cambridge, Mass., vol. 1, pp. 283-317, 1986.
[27]A. Jayakumar and J. Alspector, "A cascadable neural network chip set with on-chip learning using noise and gain annealing," IEEE Custom Integrated Circuits Conference, pp. 19.5.1-19.5.4, 1992.
[28]J. Movellan, "A learning theorem for networks at detailed stochastic equilibrium," Neural Computation, vol. 10, pp. 1157-1178, 1998.
[29]P. Y. Ting and R. A. Iltis, "Diffusion network architectures for implementation of Gibbs samplers with applications to assignment problems," IEEE Transactions on Neural Networks, vol. 5, no. 4, pp. 622-638, 1994.
[30]G. Martin and J. Pittman, "Recognizing hand-printed letters and digits using backpropagation learning," Neural Computation, vol. 3, pp. 258-267, 1991.
[31]M. Valle, "Analog VLSI implementation of artificial neural networks with supervised on-chip learning," Analog Integrated Circuits and Signal Processing, vol. 33, pp. 263-287, 2002.
[32]T. Morie and Y. Amemiya, "An all-analog expandable neural network LSI with on-chip backpropagation learning," IEEE Journal of Solid-State Circuits, vol. 29, no. 9, pp. 1086-1093, 1994.
[33]P. Moerland and E. Fiesler, "Neural network adaptations to hardware implementations," Handbook of neural computation, pp. 2, 1997.
[34]C. C. Lu, C. Y. Hong, and H. Chen, "A scalable and programmable architecture for the continuous restricted Boltzmann machine in VLSI," IEEE International Symposium on Circuits and Systems, pp. 1297-1300, 2007.
[35]P. Fleury, H. Chen, and A. F. Murray, "On-chip contrastive divergence learning in analogue VLSI," the International Joint Conference on Neural Networks (IJCNN'2004), pp. 1723-1728, 2004.
[36]G. Wegmann and E. Vittoz, "Analysis and improvements of accurate dynamic current mirrors," IEEE Journal of Solid-State Circuits, vol. 25, no. 3, pp. 699-706, 1990.
[37]A. Hastings and R. Hastings, The art of analog layout: Prentice Hall, 2001.
[38]H. Chen, P. Fleury, and A. F. Murray, "Minimizing contrastive divergence in noisy mixed-mode VLSI neurons," Advances in Neural Information Processing System (NIPS2003), vol. 16, 2004.
[39]Y. Teh and G. Hinton, "Rate-coded restricted Boltzmann machines for face recognition," Advances in neural information processing systems, pp. 908-914, 2001.
[40]C. Peterson and J. Anderson, "A mean field theory learning algorithm for neural networks," Complex Systems, vol. 1, pp. 995-1019, 1987.
[41]S. Haykin, Neural networks: a comprehensive foundation, Prentice Hall PTR Upper Saddle River, NJ, USA, 1994.
[42]C. Toumazou, J. B. Hughes, N. C. Battersby, and N. C. Battersby, Analogue IC design: the current-mode approach, Peter Peregrinus Ltd, 1990.
[43]A. F. Murray, "Analogue noise-enhanced learning in neural network circuits," Electronics Letters, vol. 27, no. 17, pp. 1546-1548, 1991.
[44]A. F. Murray and P. Edwards, "Synaptic weight noise during MLP training: Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training," IEEE Transactions Neural Networks, vol. 5, no. 5, pp. 792-802, 1994.
[45]G. Weinan, G. Weinan, and W. M. Snelgrove, "Floating gate charge-sharing: a novel circuit for analog trimming," 4 ed. W. M. Snelgrove, pp. 315-318, 1994.
[46]J. Hyde, T. Humes, C. Diorio, M. Thomas, and M. Figueroa, "A 300-MS/s 14-bit digital-to-analog converter in logic CMOS," IEEE Journal of Solid-State Circuits, vol. 38, no. 5, pp. 734-740, 2003.
[47]N. H. Hamid, A. F. Murray, D. Laurenson, S. Roy, and C. Binjie, "Probabilistic computing with future deep sub-micrometer devices: a modelling approach," IEEE International Symposium on Circuits and Systems, pp. 2510-2513, 2005.
[48]G. Cauwenberghs and A. Yariv, "Fault-tolerant dynamic multilevel storage in analog VLSI," IEEE Transactions on Circuits and Systems II: Analog and Digital Processing, vol. 37, no. 12, pp. 808-814, 1990.
[49]B. Hochet, "Multivalued MOS memory for variable-synapse neural networks," Electronics Letters, vol. 25, pp. 669, 1989.
[50]G. Cauwenberghs, "Delta-sigma cellular automata for analog VLSI random vector generation," IEEE Transactions on Circuits and Systems II: Analog and Digital Processing, vol. 46, no. 3, pp. 240-250, 1999.
[51]M. Reid, E. A. Brown, and S. P. Deweerth, "Subthreshold CMOS array for generating a Gaussian distribution of currents," IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 53, no. 10, pp. 1123-1127, 2006.
[52]G. Cauwenberghs, "An analog VLSI recurrent neural network learning a continuous-time trajectory," IEEE Transactions on Neural Networks, vol. 7, no. 2, pp. 346-361, 1996.
[53]J. Donoghue, "Bridging the brain to the world: a perspective on neural interface systems," Neuron, vol. 60, pp. 511-521, 2008.
[54]R. Olsson III and K. Wise, "A three-dimensional neural recording microsystem with implantable data compression circuitry," IEEE Journal of Solid-State Circuits, vol. 40, no. 12, pp. 2796-2804, 2005.
[55]C. F. Liu, T. Y. Pu, Y. C. Huang, K. F. Lin, Y. Y. Chen, W. T. Huang, Y. J. Chang, and C. H. Chen, "Wireless Systems for the Sorting of Neural Signals Using Onboard Processing," presented at the Biomedical Engineering Symposium on Biosignal, Biosensor, Bioelectronic, and Bioengineering, 2009.
[56]M. S. Chae, Z. Yang, M. R. Yuce, L. Hoang, and W. Liu, "A 128-channel 6 mW wireless neural recording IC with on-the-fly spike sorting and UWB transmitter," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 42, no. 4, pp.312-321, 2009.
[57]R. R. Harrison, P. T. Watkins, R. J. Kier, R. O. Lovejoy, D. J. Black, B. Greger, and F. Solzbacher, "A low-power integrated circuit for a wireless 100-electrode neural recording system," IEEE Journal of Solid-State Circuits, vol. 42, no. 1, pp. 123-133, 2007.
[58]E. Salinas, "Noisy neurons can certainly compute," Nature neuroscience, vol. 9, pp. 1349-1350, 2006.
[59]C. Diorio, D. Hsu, and M. Figueroa, "Adaptive CMOS: from biological inspiration to systems-on-a-chip," Proceedings of the IEEE, vol. 90, no. 3, pp. 345-357, 2002.
[60]G. Serrano, P. D. Smith, H. J. Lo, R. Chawla, T. S. Hall, C. M. Twigg and P. Hasler, "Automatic rapid programming of large arrays of floating-gate elements," IEEE International Symposium on Circuits and Systems, pp. 373-376, 2004.

全文公開日期本全文未授權公開 (校內網路)
全文公開日期本全文未授權公開 (校外網路)

簡易檢索 / 詳目顯示

相關論文