本研究探討半導體後端組裝排程問題（Semiconductor Back-End Assembly Scheduling Problem, SBESP），這是半導體封測廠製程中的關鍵環節。半導體封測廠主要採用訂貨型生產和代工製造模式[53]，客戶期望在無需大量庫存的情況下滿足需求，使得週期時間成為客戶選擇供應商的重要指標。SBESP可以視為混合式流程生產（Hybrid Flow Shop, HFS）問題的延伸，涉及多階段和多類型平行機器環境，被認為是NP-hard問題。過去針對小規模問題的研究多採用解析解，但面對規模和複雜性提升的問題時，則轉向使用啟發式算法和模擬為基礎的最佳化（Simulation-Based Optimization, SBO）來尋求較佳解。然而，SBO在處理龐大解空間時需要大量計算資源。為解決此問題，近期研究將排名與選擇方法（Ranking & Selection, R&S）與元啟發式演算法相結合，構建模擬最佳化框架，以改善資源分配和提升解的品質。
隨著決策細緻度的提升，單純依賴元啟發式演算法會消耗更多計算資源，並且局部最佳解隨細緻度上升而增加，使搜索過程更容易提早收斂。為了在有限時間內獲得更優解，本研究引入強化學習作為動態派工決策者，使系統能根據當前環境快速做出決策，以減少使用元啟發式演算法之探索時間成本。本研究延續Chiu等人所提出的ORAS方法作為模擬最佳化框架，保留原先批次分割大小及機台分配之決策外，並結合強化學習的動態派工策略，以提高決策細緻度。透過模擬最佳化和強化學習的結合，能夠在不耗費過多計算資源的情況下，進一步降低訂單之平均週期時間。最後實驗結果中，模擬最佳化和強化學習結合之系統框架下，在12個資料集中皆能夠有效降低平均週期時間。

This study investigates the Semiconductor Back-End Assembly Scheduling Problem (SBESP), a critical aspect of semiconductor backend manufacturing processes. Semiconductor backend facilities primarily adopt make-to-order and outsourcing modes[53], where minimizing lead time becomes crucial for customer satisfaction without excessive inventory. SBESP extends from the Hybrid Flow Shop (HFS) problem, involving multi-stage and multi-type parallel machine environments, recognized as an NP-hard problem. Previous studies focused on small-scale problems often utilized analytical solutions. However, with increasing scale and complexity, heuristic algorithms and Simulation-Based Optimization (SBO) are now preferred for seeking better solutions. Nevertheless, SBO requires significant computational resources when handling large solution spaces. To address this challenge, recent research combines Ranking & Selection (R&S) methods with metaheuristic algorithms to construct a simulation-based optimization framework, aiming to improve resource allocation and enhance solution quality.
As decision granularity increases, relying solely on metaheuristic algorithms consumes more computational resources, and the prevalence of local optima rises with granularity, leading to premature convergence in the search process. To obtain superior solutions within limited time, this study introduces Reinforcement Learning (RL) as a dynamic dispatching decision maker, enabling the system to make rapid decisions based on current conditions, thereby reducing exploration time costs associated with metaheuristic algorithms. Building upon the ORAS method proposed by Chiu et al. [20] as a simulation-based optimization framework, this study integrates RL-based dynamic dispatching strategies to enhance decision granularity. The integration of simulation-based optimization and RL allows for further reduction in average lead times of orders without excessive computational resource consumption. Experimental results demonstrate the effectiveness of the integrated framework across 12 datasets in lowering average cycle times.

[1] Allaoui, H., A. Artiba, 2004. Integrating simulation and optimization to schedule a hybrid flow shop with maintenance constraints. Computers & Industrial Engineering, 47, 431–450.
[2] Balin, S. (2011). Parallel machine scheduling with fuzzy processing times using a robust genetic algorithm and simulation, Information Sciences, 181, 3551-3569.
[3] Bang, J.Y., Y.D., Kim and S.W., Choi, 2012. Multiproduct lot merging–splitting algorithms for semiconductor wafer fabrication. IEEE Transactions on Semiconductor Manufacturing, 25(2), 200–210.
[4] Behera, D.K., 2012. Complexity on parallel machine scheduling: A review, In Emerging Trends in Science, Engineering and Technology, 373-381, Springer India.
[5] Branke, J., and D.C., Mattfeld, 2005. Anticipation and flexibility in dynamic scheduling. International Journal of Production Research, 43(15), 3103-3129.
[6] Branke, J., S.E. Chick, and C. Schmidt, 2007. Selecting a selection procedure, Management Science, 53, 1916–1932.
[7] Carson, Y., and A. Maria, 1997. Simulation optimization: Methods and applications. Proceedings of the 29th Conference on Winter Simulation, Atlanta, GA, 118–126.
[8] Chaari, T., S. Chaabane, T. Loukil, and D. Trentesaux, 2011. A genetic algorithm for robust hybrid flow shop scheduling. International Journal of Computer Integrated Manufacturing, 24(9), 821–833.
[9] Chang, H. C., Chen, Y. P., Liu, T. K., & Chou, J. H. (2015). Solving the flexible job shop scheduling problem with makespan optimization by using a hybrid Taguchi-genetic algorithm. IEEE access, 3, 1740-1754.
[10] Chan, F. T., & Chan, H. K. (2004). A comprehensive survey and future trend of simulation study on FMS scheduling. Journal of Intelligent Manufacturing, 15, 87-102.
[11] Chan, F. T., and K. L. Choy, 2011. A genetic algorithm-based scheduler for multiproduct parallel machine sheet metal job shop. Expert Systems with Applications, 38(7), 8703–8715.
[12] Chen, C.H., and L.H. Lee, 2011. Stochastic simulation optimization – an optimal computing budget allocation, World Scientific Publishing Co. Ptd. Ltd. Singapore.
[13] Chen, C.H., J. Y. Lin, E. Ucesan, and S.E. Chick, 2000. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic Systems, 10(3), 251–270.
[14] Chen, J.C., L. Su, C. Sun, and M. Hsu, 2010. Infinite capacity planning for IC packaging plants. International Journal of Production Research, 48(19), 5729–5748.
[15] Chen, R., Yang, B., Li, S., & Wang, S. (2020). A self-learning genetic algorithm based on reinforcement learning for flexible job-shop scheduling problem. Computers & industrial engineering, 149, 106778.
[16] Cheng, M., N.J. Mukherjee, and S.C. Sarin, 2013. A review of lot streaming. International Journal of Production Research, 51(23-24), 7023–7045.
[17] Chien, C. F., & Lan, Y. B. (2021). Agent-based approach integrating deep reinforcement learning and hybrid genetic algorithm for dynamic scheduling for Industry 3.5 smart production. Computers & Industrial Engineering, 162, 107782.
[18] Chiu, C.C., J.T. Lin, 2017. Novel hybrid approach with elite group optimal computing budget allocation for the stochastic multimodal problem. Neurocomputing, 260, 449–465.
[19] Chiu, C.C., J.T. Lin, 2018. A novel optimal replication allocation strategy for particle swarm optimization algorithms applied to simulation optimization problem. Applied Soft Computing, 71: 591–607.
[20] Chiu, C. C., Lai, C. M., & Chen, C. M. (2023). An evolutionary simulation-optimization approach for the problem of order allocation with flexible splitting rule in semiconductor assembly. Applied Intelligence, 53(3), 2593-2615.
[21] Choe, R., Kim, J., & Ryu, K. R. (2016). Online preference learning for adaptive dispatching of AGVs in an automated container terminal. Applied Soft Computing, 38, 647-660.
[22] Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2009). Introduction to algorithms. 2001.
[23] de Sousa Junior, W. T., Montevechi, J. A. B., de Carvalho Miranda, R., & Campos, A. T. (2019). Discrete simulation-based optimization methods for industrial engineering problems: A systematic literature review. Computers & Industrial Engineering, 128, 526-540.
[24] Frantzén, M., A.H. Ng, and P. Moore, 2011. A simulation-based scheduling system for real-time optimization and decision making support. Robotics and Computer-Integrated Manufacturing, 27(4), 696–705.
[25] French, S., 1982. Sequencing and scheduling: an introduction to the mathematics of the job-shop. Chichester: Ellis Horwood.
[26] Fu, M., 2002. Optimization for simulation: theory vs. practice. INFORMS Journal on Computing, 14(3), 192–215.
[27] Fu, M., R. Askin, J. Fowler, M. Haghnevis, N. Keng, J. S. Pettinato, and M. Zhang, 2011. Batch production scheduling for semiconductor back-end operations. Semiconductor Manufacturing, IEEE Transactions on, 24(2), 249-260.
[28] Fu, M.C., F.W. Glover, and J. April, 2005. Simulation optimization: a review, new developments, and applications. In Proceedings of the 37th conference on Winter simulation, 83–95.
[29] Garey, M., and D. Johnson, 1997. Computers and intractability: A guide to the theory of NP-completeness, W.H. Freeman, New York.
[30] Gendreau, M., & Potvin, J. Y. (Eds.). (2010). Handbook of metaheuristics (Vol. 2, p. 9). New York: Springer.
[31] Grillo, Samuele, Antonio Pievatolo, and Enrico Tironi. "Optimal storage scheduling using Markov decision processes." IEEE Transactions on Sustainable Energy 7.2 (2015): 755-764.
[32] Herroelen, W., & Leus, R. (2005). Project scheduling under uncertainty: Survey and research potentials. European journal of operational research, 165(2), 289-306.
[33] Holland, J.H., 1975. Adaptation in Natural and Artificial Systems, The University of Michigan Press, Ann Arbor, MI.
[34] Jemmali, M., Hidri, L., & Alourani, A. (2022). Two-stage hybrid flowshop scheduling problem with independent setup times. International Journal of Simulation Modelling, 21(1), 5-16.
[35] Johnson, D.J. 2003. A framework for reducing manufacturing throughput time. Journal of Manufacturing Systems, 22(4), 283–298.
[36] Karimi-Mamaghan, M., Mohammadi, M., Meyer, P., Karimi-Mamaghan, A. M., & Talbi, E. G. (2022). Machine learning at the service of meta-heuristics for solving combinatorial optimization problems: A state-of-the-art. European Journal of Operational Research, 296(2), 393-422.
[37] Kayhan, B. M., & Yildiz, G. (2023). Reinforcement learning applications to machine scheduling problems: a comprehensive literature review. Journal of Intelligent Manufacturing, 34(3), 905-929.
[38] Kennedy, J., and R.C. Eberhart, 1995. Particle swarm optimization. Proceedings of the IEEE International Conference on Neural Networks, 4, 1942–1948.
[39] Kim, G. H., & Lee, C. G. (1996, April). Genetic reinforcement learning for scheduling heterogeneous machines. In Proceedings of IEEE International Conference on Robotics and Automation (Vol. 3, pp. 2798-2803). IEEE.
[40] Kravchenko, S.A. and F. Werner, 2001. A heuristic algorithm for minimizing mean flow time with unit setups. Information Processing Letters, 79(6), 291-296.
[41] Lai, C.M. 2018. Multi-objective simplified swarm optimization with weighting scheme for gene selection, Applied Soft Computing, 65, 58-68.
[42] Lai, C.M., 2019. Integrating simplified swarm optimization with AHP for solving capacitated military logistic depot location problem, Applied Soft Computing, 78, 1-12.
[43] Law, A. M. (2007). Simulation Modeling and Analysis, McGraw-Hill, Boston.
[44] Lee, L. H., E.P. Chew, P.I. Frazier, Q.S. Jia, and C.H. Chen, 2013. Advances in simulation optimization and its applications. IIE Transactions, 45(7), 683–684.
[45] Lee, L.H., C.H. Chen, E.P. Chew, J. Li, N.A. Pujowidianto, and S. Zhang, 2010. A review of optimal computing budget allocation algorithms for simulation optimization problem. International Journal of Operations Research, 7(2), 19–31.
[46] Lee, L.H., E.P. Chew, S. Teng, and Y. Chen, 2008. Multi-objective simulation-based evolutionary algorithm for an aircraft spare parts allocation problem. European Journal of Operational Research, 189(2), 476–491.
[47] Li, D., X. Meng, Q. Liang, and J. Zhao, 2015. A heuristic-search genetic algorithm for multi-stage hybrid flow shop scheduling with single processing machines and batch processing machines. Journal of intelligent manufacturing, 26, 873–890.
[48] Li, K. A. Sava, and X. Xie, 2009. Simulation-based discrete optimization of stochastic discrete event systems subject to non closed-form constraints. IEEE Transaction on Automatic Control, 54(12), 2900–2904.
[49] Li, Y., Fadda, E., Manerba, D., Tadei, R., & Terzo, O. (2020, September). Reinforcement learning algorithms for online single-machine scheduling. In 2020 15th Conference on Computer Science and Information Systems (FedCSIS) (pp. 277-283). IEEE.
[50] Lin, J. T., and C.M. Chen, 2015. Simulation optimization approach for hybrid flow shop scheduling problem in semiconductor back-end manufacturing. Simulation Modelling Practice and Theory, 51, 100–114.
[51] Lin, J.T., and C.J. Huang, 2014. A Simulation-based Optimization Approach for a Semiconductor Photobay with Automated Material Handling System. Simulation Modelling Practice and Theory, 46, 76–100.
[52] Lin, J.T., C.C. Chiu, 2015. A hybrid particle swarm optimization with local search for stochastic resource allocation problem, Journal of intelligent manufacturing, 1–10.
[53] Lin, J.T., C.C., Chiu, and Y.H. Chang, 2017. Simulation-based optimization approach for simultaneous scheduling of vehicles and machines with processing time uncertainty in FMS, Flexible Service Manufacturing Journal. in press, doi.org/10.1007/s10696-017-9302-x.
[54] Lin, J.T., C.M. Chen, C.C. Chiu, and H.Y. Fang, 2013. Simulation optimization with PSO and OCBA for semiconductor back-end assembly. Journal of Industrial and Production Engineering, 30(7), 452–460.
[55] Lin, J. T., & Chen, C. M. (2015, March). Simulation optimization with GA and OCBA for semiconductor back-end assembly scheduling. In 2015 International Conference on Industrial Engineering and Operations Management (IEOM) (pp. 1-8). IEEE.
[56] Lin, R.C. M.Y. Sir, and K.S. Pasupathy, 2013. Multi-objective simulation optimization using data envelopment analysis and genetic algorithm: Specific application to determining optimal resource level in surgical services. Omega, 41, 881–892.
[57] Liu, R., Piplani, R., & Toro, C. (2022). Deep reinforcement learning for dynamic scheduling of a flexible job shop. International Journal of Production Research, 60(13), 4049-4069.
[58] Marimuthu, S., S.G. Ponnambalam, and N. Jawahar, 2008. Evolutionary algorithms for scheduling m-machine flow shop with lot streaming. Robotics and Computer-Integrated Manufacturing, 24(1), 125–139.
[59] Martin, C.H., 2009. A hybrid genetic algorithm/mathematical programming approach to the multi-family flowshop scheduling problem with lot streaming. Omega, 37(1), 126–137.
[60] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Hassabis, D. (2015). Human-level control through deep reinforcement learning. nature, 518(7540), 529-533.
[61] Mohammadi, M., and S. M. T. Fatemi Ghomi, 2011. Genetic algorithm-based heuristic for capacitated lotsizing problem in flow shops with sequence-dependent setups. Expert systems with applications, 38(6), 7201–7207.
[62] Nait Tahar, D., F. Yalaoui, C. Chu, and L. Amodeo, 2006. A linear programming approach for identical parallel machine scheduling with job splitting and sequence-dependent setup times. International Journal of Production Economics, 99(1), 63–73.
[63] Ni, Y., Li, Y., Yao, J., & Li, J. (2014). Development of an integrated real time dispatching system: A case study at a semiconductor assembly and test factory. Journal of Manufacturing Technology Management, 25(7), 980-997.
[64] Osman, I. H., & Laporte, G. (1996). Metaheuristics: A bibliography. Annals of Operations research, 63, 511-623.[64]
[65] Pan, H. L. Wang, and B. Liu, 2006. Particle swarm optimization for function optimization in noisy environment. Applied Mathematics and Computation, 181, 908–919.
[66] Pan, Q. K., & Ruiz, R. (2012). An estimation of distribution algorithm for lot-streaming flow shop problems with setup times. Omega, 40(2), 166–180.
[67] Paternina-Arboleda, C. D., & Das, T. K. (2001). Intelligent dynamic control policies for serial production lines. Iie Transactions, 33(1), 65-77.
[68] Pearn, W. L., Chung, S. H., & Lai, C. M. (2007) Scheduling integrated circuit assembly operations on die bonder. IEEE Transactions on Electronics Packaging Manufacturing, 30(2), 97–105.
[69] Pickardt, C.W., T., Hildebrandt, J., Branke, J., Heger, and B. Scholz-Reiter, 2013. Evolutionary generation of dispatching rule sets for complex dynamic scheduling problems. International Journal of Production Economics, 145(1), 67–77.
[70] Pinedo, M. 1995. Scheduling: theory, algorithms, and systems, Prentice-Hall, Englewood Cliffs, NJ.
[71] Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
[72] Qin, W., J. Zhang, and D. Song 2015. An improved ant colony algorithm for dynamic hybrid flow shop scheduling with uncertain processing time. Journal of intelligent manufacturing, 1–14.
[73] Ribas, I., R. Leisten, and J. M. Framiñan, 2010. Review and classification of hybrid flow shop scheduling problems from a production system and a solutions procedure perspective. Computers & Operations Research, 37(8), 1439–1454.
[74] Ruiz, R., and J.A. Vázquez-Rodríguez, 2010. The hybrid flow shop scheduling problem, European Journal of Operational Research, 205(1), 1–18.
[75] Ruiz, R., F.S., Şerifoğlu, and T. Urlings, 2008. Modeling realistic hybrid flexible flowshop scheduling problems, Computers & Operations Research, 35(4), 1151–1175.
[76] Schmidt, C., J. Branke, and S.E. Chick, 2006. In Applications of Evolutionary Computing, Integrating techniques from statistical ranking into evolutionary algorithms (Springer Berlin Heidelberg), 752–763.
[77] Shi, L., Guo, G., & Song, X. (2021). Multi-agent based dynamic scheduling optimisation of the sustainable hybrid flow shop in a ubiquitous environment. International Journal of Production Research, 59(2), 576-597.
[78] Song, H., Triguero, I., & Özcan, E. (2019). A review on the self and dual interactions between machine learning and optimisation. Progress in Artificial Intelligence, 8(2), 143-165.
[79] Sun, X., Vogel‐Heuser, B., Bi, F., & Shen, W. (2022). A deep reinforcement learning based approach for dynamic distributed blocking flowshop scheduling with job insertions. IET Collaborative Intelligent Manufacturing, 4(3), 166-180.
[80] Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
[81] Talbi, E. G. (2009). Metaheuristics: from design to implementation. John Wiley & Sons.
[82] Talbi, E.G. (2016). Combining metaheuristics with mathematical programming, constraint programming and machine learning. Annals of Operations Research, 240(1), 171-215.
[83] Tekin, E., and I. Sabuncuoglu, 2004. Simulation optimization: A comprehensive review on theory and applications. IIE Transactions, 36(11), 1067–1081.
[84] Tseng, C. T., and C. J. Liao, 2008. A discrete particle swarm optimization for lot-streaming flowshop scheduling problem. European Journal of Operational Research, 19(2), 360–373.
[85] Vallada, E., and R. Ruiz, 2011, A genetic algorithm for the unrelated parallel machine scheduling problem with sequence dependent setup times. European Journal of Operational Research, 211(3), 612–622.
[86] Van Hasselt, H., Guez, A., & Silver, D. (2016, March). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 30, No. 1).
[87] Ventura, J.A., and S.H. Yoon, 2013. A new genetic algorithm for lot-streaming flow shop scheduling with limited capacity buffers. Journal of Intelligent Manufacturing, 24(6), 1185–1196.
[88] Walker, J., 2014. Forecast Analysis: Semiconductor Assembly and Test Services. Worldwide, 4Q13 Update, Gartner.
[89] Wang, L., Hu, X., Wang, Y., Xu, S., Ma, S., Yang, K., ... & Wang, W. (2021). Dynamic job-shop scheduling in smart manufacturing using deep reinforcement learning. Computer Networks, 190, 107969.
[90] Wang, S., L. Wang, M. Liu, and Y. Xu, 2015. A hybrid estimation of distribution algorithm for the semiconductor final testing scheduling problem. Journal of intelligent manufacturing, 26, 861–871.
[91] Waschneck, B., Reichstaller, A., Belzner, L., Altenmüller, T., Bauernhansl, T., Knapp, A., & Kyek, A. (2018). Optimization of global production scheduling with deep reinforcement learning. Procedia Cirp, 72, 1264-1269.
[92] Waschneck, B., Reichstaller, A., Belzner, L., Altenmüller, T., Bauernhansl, T., Knapp, A., & Kyek, A. (2018, April). Deep reinforcement learning for semiconductor production scheduling. In 2018 29th annual SEMI advanced semiconductor manufacturing conference (ASMC) (pp. 301-306). IEEE.
[93] Watkins, C. J., & Dayan, P. (1992). Q-learning. Machine learning, 8, 279-292.
[94] Weigert, D., A. Klemmt, and S. Horn, 2009. Design and validation of heuristic algorithms for simulation-based scheduling of a semiconductor backend facility. International Journal of Production Research, 47(8), 2165–2184.
[95] Yang, T., and Y.A. Shen, 2012. Heuristic algorithms for a practical-size dynamic parallel-machine scheduling problem: integrated-circuit wire bonding. Production Planning & Control, 23(1), 67–78.
[96] Ye, T., 2012. Analysis on multi-stage lot streaming: The effect of transfer. Computers & Industrial Engineering, 62(4), 1046–1054.
[97] Yeh, W.C. 2009. A two-stage discrete particle swarm optimization for the problem of multiple multi-level redundancy allocation in series systems, Expert Systems with Applications, 36, 9192-9200.
[98] Yeh, W.C., 2012. Simplified swarm optimization in disassembly sequencing problems with learning effects, Computers & Operations Research, 39, 2168-2177.
[99] Zhang, M., Lu, Y., Hu, Y., Amaitik, N., & Xu, Y. (2022). Dynamic scheduling method for job-shop manufacturing systems by deep reinforcement learning with proximal policy optimization. sustainability, 14(9), 5177.
[100] Zhang, M.T., S. Niu, S. Deng, Z. Zhang, Q. Li, and L. Zheng, 2007. Hierarchical capacity planning with reconfigurable kits in global semiconductor assembly and test manufacturing, IEEE Transactions on Automation Science and Engineering, 4(4), 543–552.
[101] Zhang, W., & Dietterich, T. G. (1995, August). A reinforcement learning approach to job-shop scheduling. In Ijcai (Vol. 95, pp. 1114-1120).