在智慧場域中，隨著串流監視影像技術的興起，許多創新的分析應用也應運而生，其中包含了各式各樣能將純影像轉化為對使用者來說具有意義的結果。除了即時的串流服務，這些監視影像也被保存在存儲伺服器中，以提供未來使用者隨選式、自定義的分析請求。不同於現有專注於最大化影片品質的隨選式影音串流服務，監視影像存儲伺服器需要面對的是：在有限的空間和運算資源下決定並最大化儲存影片的資訊含量，同時還要替未來新進的影片預留空間。在本論文中，我們設計、實作、優化、並評估了一個多級特徵驅動存儲系統，該系統能提供各種規模的智慧場域使用，如智慧校園、建築、社區或城市。我們專注於此系統的設計和實作，並且解決了兩個核心的問題：（一）有效率地捕捉新進影片的資訊含量（二）聰明地決定影片保存的品質。我們首先採取近似分析的方式推算出影片的資訊含量，並使得伺服器免於過載的問題。此近似分析的算法基於多級特徵（語意及視覺特徵）正式定義所謂的「資訊量」。接著，我們根據此量化的數值決定最佳降採樣的方法和影片的目標保存品質，目的在於最大化存儲系統中的的總資訊量。我們嚴格地制定上述兩個研究問題並以數學方法將其轉化為最佳化問題。對此兩個問題，我們分別給出了最佳、近似、和高效共六個算法。除了一系列優化的算法，我們也利用清華大學的物聯網測試平台中的影片評估我們提出的系統。此平台包含了八支裝有各式感測器的智慧路燈，其中四支裝了監視錄影機，並且所有捕捉的影片都會送回至機房做儲存和分析。在實驗階段，我們以實際的影片評估我們提出的演算法的效能，而我們提出的解決方案在多方面都勝過了目前業界的做法，例如：（一）在捕捉資訊量上，達成了和最佳解僅有7\%的差距（二）在一週的實驗後存下了近三倍的影片數量（三）減少平均58\%的請求誤差（四）能在100毫秒內做出決定所有影片保存的品質（五）不超出系統可負荷的儲存空間（六）能適應各種規模的儲存空間。

Surveillance videos in smart environments have become commodities nowadays, which enable many novel applications, including various video analytics that turn videos into semantic results. In addition to live feeds, the surveillance videos may be saved in a storage server for on-demand user-defined queries in the future. Different from on-demand video streaming servers whose design objective is to maximize the user-perceived video quality, a surveillance video storage server has limited space and must retain as much information as possible, while reserving sufficient space for incoming videos.
In this thesis, we design, implement, optimize, and evaluate a multi-level feature driven storage server for diverse-scale smart environments, for example buildings, campuses, communities, and cities.
We focus on the design and implementation of the storage server and solve two key research problems in it, namely: (i) efficiently determining the information amount of incoming videos and (ii) intelligently deciding the qualities of videos to be kept. In particular, we first analyze the videos to derive approximate information amount without overloading our storage server. This is done by formally defining the information amount based on multi-level (semantic and visual) features of videos. We then leverage the information amounts to determine the optimal downsampling approach and target quality level of videos to save storage space, while preserving as much information amount as possible. We rigorously formulate the above two research problems into mathematical optimization problems, and propose optimal, approximate, and efficient algorithms to solve them. Besides the suite of optimization algorithms, we also implement our proposed system on a smart campus testbed at NTHU, Taiwan, which consists of eight smart street lamps. The street lamps are equipped with a wide spectrum of sensors, network devices, analytics servers, and a storage server. We compare the performance of our proposed algorithms against the current practices using real surveillance videos from our smart campus testbed. Our efficient algorithms outperform the current practices in multiple dimensions, meaning we: (i) achieve a mere $7\%$ approximation gap on captured information amount compared to the optimal solutions, (ii) save almost 3 times more clips after a week, (iii) achieve $58\%$ less per-query error on average, (iv) always terminate in less than 100 ms, (v) do not consume excessive storage space, and (vi) scale well with larger storage spaces.

[1] G. Ananthanarayanan, P. Bahl, P. Bod ́ık, K. Chintalapudi, M. Philipose, L. Ravindranath, and S. Sinha. Real-time video analytics: The killer app for edge computing. IEEE Computer, 50(10):58–67, October 2017.
[2] A. Anjomshoaa, F. Duarte, D. Rennings, T. Matarazzo, P. deSouza, and C. Ratti. City scanner: Building and scheduling a mobile sensing platform for smart city services. IEEE Internet of things Journal, 5(6):4567–4579, May 2018.
[3] K. Ashton et al. That Internet of Things thing. LLC RFID, 22(7):97–114, June 2009.
[4] A. Astaras, H. Lewy, C. James, A. Katasonov, D. Ruschin, and P. Bamidis. Unobtrusive smart environments for independent living and the role of mixed methods in elderly healthcare delivery: the USEFIL approach. In Handbook of Research on Innovations in the Diagnosis and Treatment of Dementia, chapter 15, pages 290–305. IGI Global, May 2015.
[5] Azure. Azure status history, 2021.
[6] E. Bingham and H. Mannila. Random projection in dimensionality reduction: applications to image and text data. In Proc. of ACM International Conference on Knowledge discovery and data mining (KDD’01), pages 245–250, San Francisco, California, August 2001.
[7] L. Bittencourt, R. Immich, R. Sakellariou, N. Fonseca, E. Madeira, M. Curado, L. Villas, L. DaSilva, C. Lee, and O. Rana. The Internet of Things, fog and cloud continuum: Integration and challenges. Elsevier Internet of Things, 3(4):134–155, October 2018.
[8] A. Bochkovskiy. YOLO-v3 and YOLO-v2 for Windows and Linux, 2019.
[9] B. Burns, J. Beda, and K. Hightower. In Kubernetes: up and running: dive into the future of infrastructure. O’Reilly Media, May 2015.
[10] H. Cai, L. Da Xu, B. Xu, C. Xie, S. Qin, and L. Jiang. IoT-based configurable information service platform for product lifecycle management. IEEE Transactions on Industrial Informatics, 10(2):1558–1567, February 2014.
[11] L. Carnevale, A. Celesti, A. Galletta, S. Dustdar, and M. Villari. From the cloud to edge and IoT: a smart orchestration architecture for enabling osmotic computing. In Proc. of IEEE International Conference on Advanced Information Networking and Applications Workshops (WAINA’18), pages 419–424, Krakow, Poland, May 2018.
[12] G. Chandan, A. Jain, H. Jain, et al. Real time object detection and tracking using deep learning and OpenCV. In Proc. of IEEE International Conference on Inventive Research in Computing Applications (ICIRCA’18), pages 1305–1308, Coimbatore, India, July 2018.
[13] T.Chen, L.Ravindranath, S.Deng, P.Bahl, and.Balakrishnan.Glimpse: Continuous, real-time object recognition on mobile devices. In Proc. of ACM Conference on Embedded Networked Sensor Systems (SenSys’15), pages 155–168, New York, NY, November 2015.
[14] X. Chen, L. Pu, L. Gao, W. Wu, and D. Wu. Exploiting massive D2D collaboration for energy-efficient mobile edge computing. IEEE Wireless Communications, 24(4):64–71, August 2017.
[15] T.-c. Chiueh and S. Nanda. A survey on virtualization technologies. Rpe Report, 142, June 2005.
[16] Christopher Ingraham. Nine days on the road. Average commute time reached a new record last year, 2019.
[17] V. Cimino and S. E. Rombo. Design and prototyping of a smart university campus. In Handbook of Research on Implementation and Deployment of IoT Projects in Smart Cities, chapter 14, pages 228–252. IGI Global, May 2019.
[18] D. Cook, G. Duncan, G. Sprint, and R. Fritz. Using smart city technology to make healthcare smarter. Proceedings of the IEEE, 106(4):708–722, 2018.
[19] M. Cox and T. Cox. Multidimensional scaling. In Handbook of data visualization, pages 315–347. Springer, June 2008.
[20] S. Dey, A. Chakraborty, S. Naskar, and P. Misra. Smart city surveillance: Leveraging benefits of cloud data stores. In Proc. of IEEE Conference on Local Computer Networks Workshops (LCNW), pages 868–876, Clearwater, Florida, October 2012.
[21] S. Dutta, T. Taleb, P. Frangoudis, and A. Ksentini. On-the-fly QoE-Aware transcoding in the mobile edge. In Proc. of IEEE Global Communications Conference (GLOBECOM’16), Washington, DC, USA, December 2016.
[22] FFmpeg Developers. FFmpeg Documentation, 2019.
[23] FFmpeg Developers. FFmpeg documentation black frame, 2020.
[24] Y. Fu, Y. Guo, Y. Zhu, F. Liu, C. Song, and Z. Zhou. Multi-view video summarization. IEEE Transactions on Multimedia, 12(7):717–729, June 2010.
[25] N. Funde, P. Paranjape, K. Ram, P. Magde, and M. Dhabu. Object detection and tracking approaches for video surveillance over camera network. In Proc. of IEEE International Conference on Advanced Computing & Communication Systems (ICACCS’19), pages 1171–1176, Coimbatore, India, 2019.
[26] G. Gao, W. Zhang, Y. Wen, Z. Wang, W. Zhu, and Y. Tan. Cost optimal video transcoding in media cloud: Insights from user viewing pattern. In Proc. of IEEE Intl. Conf. on Multimedia and Expo (ICME’14), pages 1–6, Chengdu, China, July 2014.
[27] G. Gens and E. Levner. An approximate binary search algorithm for the multiple-choice knapsack problem. Elsevier Information Processing Letters, 67(5):261– 265, September 1998.
[28] B. Ghazal, K. ElKhatib, K. Chahine, and M. Kherfan. Smart traffic light con- trol system. In Proc. of IEEE International Conference on Electrical, Electronics, Computer engineering and their Applications (EECEA’16), pages 140–145, Beirut, Lebanon, April 2016.
[29] A. Godbehere, A. Matsukawa, and K. Goldberg. Visual tracking of human visitors under variable-lighting conditions for a responsive audio art installation. In Proc. of IEEE American Control Conference (ACC’12), pages 4305–4312, Montreal, QC, Canada, June 2012.
[30] Google. DeepMind, 2010.
[31] Google. Google brain team: Make machines intelligent. improve people’s lives, 2011.
[32] Google.Machinelearningfindsnewwaysforourdatacenterstosaveenergy,2019.
[33] Google. Tensorflow lite, 2021.
[34] A. Harris, V. Khanna, G. Tuncay, R. Want, and R. Kravets. Bluetooth low energy in dense IoT environments. IEEE Communications Magazine, 54(12):30–36, December 2016.
[35] C. He, J. Leung, K. Lee, and M. Pinedo. An improved binary search algorithm for the multiple-choice knapsack problem. EDP Sciences RAIRO-Operations Research, 50(4-5):995–1001, December 2016.
[36] J. Herrman. Amazon’s big breakdown, 2020.
[37] R. High. The era of cognitive systems: An inside look at IBM Watson and how it works. IBM Corporation, Redbooks, 1:16, December 2012.
[38] W. Hlaing, S. Thepphaeng, V. Nontaboot, N. Tangsunantham, T. Sangsuwan, and C. Pira. Implementation of wifi-based single phase smart meter for Internet of Things (IoT). In Proc. of IEEE International Electrical Engineering Congress (IEECON’17), pages 1–4, Pattaya, Thailand, October 2017.
[39] L. Horwitz. The future of IoT miniguide: The burgeoning IoT market continues, 2019.
[40] S. Hossain, M. Hassan, M. Al, and A. Alghamdi. Resource allocation for service composition in cloud-based video surveillance platform. In Proc. of IEEE International Conference on Multimedia and Expo Workshops (ICMEW’12), pages 408–412, Melbourne, Australia, July 2012.
[41] F. Hsu. IBM's deep blue chess grandmaster chips. IEEE Micro, 19(2):70–81, April 1999.
[42] A. Ikpehai, B. Adebisi, K. Rabie, K. Anoh, R. Ande, M. Hammoudeh, H. Gacanin, and U. Mbanaso. Low-power wide-area network technologies for Internet-of-Things: A comparative review. IEEE Internet of Things Journal, 6(2):2225–2240, November 2018.
[43] InfluxDB. InfluxDB official website, 2019.
[44] Time-of-use tariffs: Innovation landscape brief. Standard, 2019.
[45] The Internet of things. Standard, International Telecommunication Union, 2005.
[46] S. Javaid, A. Sufian, S. Pervaiz, and M. Tanveer. Smart traffic management system using Internet of Things. In Proc. of IEEE International Conference on Advanced Communication Technology (ICACT’18), pages 393–398, Chuncheon, South Ko- rea, March 2018.
[47] H. Kellerer, U. Pferschy, and D. Pisinger. Multidimensional knapsack problems. In Knapsack problems, chapter 9, pages 235–283. Springer, 2004.
[48] D. Kiritsis. Closed-loop PLM for intelligent products in the era of the Internet of things. Elsevier Computer-Aided Design, 43(5):479–501, May 2011.
[49] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. ACM Communications, 60(6):84–90, June 2017.
[50] E. Lawler. Fast approximation algorithms for knapsack problems. INFORMS Mathematics of Operations Research, 4(4):339–356, November 1979.
[51] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436–444, May 2015.
[52] P. Li, L. Prieto, D. Mery, and P. Flynn. On low-resolution face recognition in the wild: Comparisons and new techniques. IEEE Transactions on Information Forensics and Security, 14(8):2000–2012, January 2019.
[53] S. Li, L. Da Xu, and S. Zhao. The Internet of Things: a survey. Springer Information Systems Frontiers, 17(2):243–259, April 2015.
[54] X. Li, M. Salehi, and M. Bayoumi. High performance on-demand video transcoding using cloud services. In Proc. of IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid’16), pages 600–603, Cartagena, Colombia, May 2016.
[55] Y. Li, M. Hou, H. Liu, and Y. Liu. Towards a theoretical framework of strategic decision, supporting capability, and information sharing under the context of Internet of Things. Springer Information Technology and Management, 13(4):205–216, April 2012.
[56] P. Liu, B. Qi, and S. Banerjee. Edgeeye: An edge service framework for real-time intelligent video analytics. In Proc. of ACM International Workshop on Edge Systems, Analytics and Networking (EdgeSys’18), pages 1–6, Munich, Germany, June 2018.
[57] L. Maaten and G. Hinton. Visualizing data using t-SNE. JMLR Journal of machine learning research, 9(11):2579–2605, November 2008.
[58] F. Mehboob and M. Abbas. Intelligent video surveillance. In A. Neves, editor, Video Surveillance-Based Intelligent Traffic Management in Smart Cities, chapter 2, pages 1–9. IntechOpen, November 2018.
[59] D. Merkel. Docker: lightweight Linux containers for consistent development and deployment. ACM Linux journal, 2014(239):2, May 2014.
[60] J. Michel. Massive outage hits google services worldwide, 2020.
[61] L. Minchala-Avila, J. Armijos, D. Pesa ́ntez, and Y. Zhang. Design and implementation of a smart meter with demand response capabilities. Elsevier Energy Procedia, 103:195–200, December 2016.
[62] S.Mo ̈llerandA.Raake.InQualityofexperience: advanced concepts, applications, and methods. Springer, March 2014.
[63] K. Muhammad, T. Hussain, M. Tanveer, G. Sannino, and V. H. C. de Albuquerque. Cost-effective video summarization using deep CNN with hierarchical weighted fusion for IoT surveillance networks. IEEE Internet of Things Journal, 7(5):4455– 4463, November 2019.
[64] A. Murugan, K. Devi, A. Sivaranjani, and P. Srinivasan. A study on various methods used for video summarization and moving object detection for video surveillance applications. Springer Multimedia Tools and Applications, 77(18):23:273– 23:290, September 2018.
[65] The NIST definition of cloud computing. Standard, 2011.
[66] S. Nunna, A. Kousaridas, M. Ibrahim, M. Dillinger, C. Thuemmler, H. Feussner, and A. Schneider. Enabling real-time context-aware collaboration through 5G and mobile edge computing. In Proc. of IEEE International Conference on Information Technology-New Generations (ITNG’15), pages 601–605, April 2015.
[67] E. Nygren, R. K. Sitaraman, and J. Sun. The akamai network: a platform for high-performance internet applications. ACM SIGOPS Operating Systems Review, 44(3):2–19, July 2010.
[68] R. Panda and A. Roy-Chowdhury. Multi-view surveillance video summarization via joint embedding and sparse optimization. IEEE Transactions on Multimedia, 19(9):2010–2021, December 2017.
[69] B. Patt-Shamir and D. Rawitz. Vector bin packing with multiple-choice. Elsevier Discrete Applied Mathematics, 160(10-11):1591–1600, July 2012.
[70] A. Prati, C. Shan, and K. Wang. Sensors, vision, and networks: From video surveillance to activity recognition and health monitoring. IOS Press Journal of Ambient Intelligence and Smart Environments, 11(1):5–22, January 2019.
[71] P. Rasti, T. Uiboupin, S. Escalera, and G. Anbarjafari. Convolutional neural network super-resolution for face recognition in surveillance monitoring. In Proc. of Springer of International conference on articulated motion and deformable objects (AMDO’16), pages 175–184, Palma de Mallorca, Spain, July 2016.
[72] D. Rodr ́ıguez-Silva, L. Adkinson-Orellana, F. Gonz’lez-Castano, I. Armino- Franco, and D. Gonz’lez-Martinez. Video surveillance based on cloud storage. In Proc. of IEEE International Conference on Cloud Computing (CLOUD’12), pages 991–992, Honolulu, Hawaii, June 2012.
[73] E. Ronen, A. Shamir, A.-O. Weingarten, and C. O’Flynn. IoT goes nuclear: Creating a ZigBee chain reaction. In Proc. of IEEE Symposium on Security and Privacy (SP’17), pages 195–212, San Jose, CA, May 2017.
[74] M. Saifuzzaman, N. Moon, and F. Nur. IoT based street lighting and traffic management system. In Proc. of IEEE Region 10 Humanitarian Technology Conference (R10-HTC’17), pages 121–124, Dhaka, Bangladesh, December 2017.
[75] M. Satyanarayanan, P. Simoens, Y. Xiao, P. Pillai, Z. Chen, K. Ha, W. Hu, and B. Amos. Edge analytics in the Internet of Things. IEEE Pervasive Computing, 14(2):24–31, April-June 2015.
[76] J. Seo, S. Han, S. Lee, and H. Kim. Computer vision techniques for construction safety and health monitoring. Elsevier Advanced Engineering Informatics, 29(2):239–251, April 2015.
[77] M. Shafi, A. Molisch, P. Smith, T. Haustein, P. Zhu, P. Silva, F. Tufvesson, A. Ben- jebbour, and G. Wunder. 5G: A tutorial overview of standards, trials, challenges, deployment, and practice. IEEE Journal on Selected Areas in Communications, 35(6):1201–1221, April 2017.
[78] Z. Shao, J. Cai, and Z. Wang. Smart monitoring cameras driven intelligent processing to big surveillance video data. IEEE Transactions on Big Data, 4(1):105–116, June 2017.
[79] W.Shi, J.Cao, Q.Zhang, Y.Li,andL.Xu.Edge computing: Visionandchallenges. IEEE Internet of Things Journal, 3(5):637–646, June 2016.
[80] B. Siregar, A. Nasution, and F. Fahmi. Integrated pollution monitoring system for the smart city. In Proc. of IEEE International Conference on ICT For Smart Society (ICISS’16), pages 49–52, Surabaya, Indonesia, July 2016.
[81] G. Sullivan, J. Ohm, W. Han, and T. Wiegand. Overview of the high-efficiency video coding (HEVC) standard. IEEE Transactions on Circuits and Systems for Video Technology, 22(12):1649–1668, September 2012.
[82] R. Sutton and A. Barto. In Reinforcement learning: An introduction. MIT Press, November 2018.
[83] F. Tao, Y. Cheng, L. Da, L. Zhang, and B. Li. CCIoT-CMfg: cloud computing and internet of things-based cloud manufacturing service system. IEEE Transactions on industrial informatics, 10(2):1435–1442, February 2014.
[84] T. Taylor. How machine learning language is driving innovation at Uber, 2018.
[85] J. Tenenbaum, V. De, and C. Langford. A global geometric framework for non-linear dimensionality reduction. AAAS Science, 290(5500):2319–2323, December 2000.
[86] S. Thomas, S. Gupta, and K. Subramanian. Smart surveillance based on video summarization. In Proc. of IEEE Region 10 Symposium (TENSYMP’17), pages 1–5, Cochin, India, July 2017.
[87] M. Tsai, N. Venkatasubramanian, and C. Hsu. Analytics-aware storage of surveillance videos: Implementation and optimization. In Proc. of IEEE International Conference on Smart Computing (SMARTCOMP’20), pages 25–32, Bologna, Italy, September 2020.
[88] A. Usman, R. Usman, and S. Shin. An intrusion-oriented heuristic for efficient re-source management in end-to-end wireless video surveillance systems. In Proc. of IEEE Annual Consumer Communications & Networking Conference (CCNC’18), pages 1–6, Las Vegas, Nevada, January 2018.
[89] L. Vangelista, A. Zanella, and M. Zorzi. Long-range IoT technologies: The dawn of LoRa. In Proc. of Springer Future access enablers of ubiquitous and intelligent infrastructures (FABULOUS’15), pages 51–58, Ohrid, Republic of Macedonia, September 2015.
[90] L. Vaquero, F. Cuadrado, Y. Elkhatib, J. BernalBernabe, S. Srirama, and M. Zhani. Research challenges in nextgen service orchestration. Elsevier Future Generation Computer Systems, 90:20–38, January 2019.
[91] B. Vejlgaard, M. Lauridsen, H. Nguyen, I. Z. Kova ́cs, P. Mogensen, and M. Sorensen. Coverage and capacity analysis of SigFox, LORA, GPRS, and NB- IoT. In Proc. of IEEE Vehicular Technology Conference (VTC’17), pages 1–5, Sydney, Australia, June 2017.
[92] C. Wang, Z. Bi, and L. Da. IoT and cloud computing in automation of assembly modeling systems. IEEE Transactions on Industrial Informatics, 10(2):1426–1434, January 2014.
[93] M. Weiss, A. Helfenstein, F. Mattern, and T. Staake. Leveraging smart meter data to recognize home appliances. In Proc. of IEEE International Conference on Pervasive Computing and Communications (PerCom’12), pages 190–197, Lugano, Switzerland, March 2012.
[94] S. Wold, K. Esbensen, and P. Geladi. Principal component analysis. Elsevier Chemometrics and intelligent laboratory systems, 2(1-3):37–52, August 1987.
[95] R. Xu, S. Y. Nikouei, Y. Chen, A. Polunchenko, S. Song, C. Deng, and T. R. Faughnan. Real-time human objects tracking for smart surveillance at the edge. In Proc. of IEEE International Conference on Communications (ICC’18), pages 1–6, Kansas City, MO, May 2018.
[96] J. Yoon, P. Liu, and S. Banerjee. Low-cost video transcoding at the wireless edge. In Proc. of IEEE/ACM Symposium on Edge Computing (SEC’16), pages 129–141, Washington DC, USA, October 2016.
[97] P. Yuan, Y. Cai, X. Huang, S. Tang, and X. Zhao. Collaboration improves the capacity of mobile edge computing. IEEE Internet of Things Journal, 6(6):10610– 10619, September 2019.
[98] M. Zakerinasab and M. Wang. Dependency-aware distributed video transcoding in the cloud. In Proc. of IEEE Conference on Local Computer Networks (LCN’15), pages 245–252, Clearwater Beach, FL, USA, October 2015.
[99] T. Zhang, A. Chowdhery, P. Bahl, K. Jamieson, and S. Banerjee. The design and implementation of a wireless video surveillance system. In Proc. of ACM Annual International Conference on Mobile Computing and Networking (MobiCom’15), pages 426–238, Paris, France, September 2015.
[100] W. Zhang, Y. Wen, and H. Chen. Toward transcoding as a service: energy-efficient offloading policy for green mobile cloud. IEEE Network, 28(6):67–73, November 2014.
[101] Y. Zhang, Q. Shang, and G. Zhang. pyDRMetrics - a python toolkit for dimensionality reduction quality assessment. Elsevier Heliyon, 7(2):e06199, February 2021.
[102] Z. Zhang. Microsoft Kinect sensor and its effect. IEEE MultiMedia, 19(2):4–10, April 2012.