在許多商業應用中，例如直接行銷(direct marketing)或顧客流失管理(churn management)，資源有限，因此需要從大量潛在對象中篩選出最有價值的目標。監督式機器學習(Supervised machine learning)中的分類模型常用於預測潛在目標，但決策者的行動（如行銷推播或介入措施）僅會針對部分預測對象實施，通常只會鎖定前 n 名(top-n)的資料。因此，傳統的模型評估指標，如準確率（accuracy）與曲線下面積（AUC），無法有效反映這種受限排序下的模型表現。本論文旨在彌合模型表現評估與實務商業需求之間的落差，重點關注lift，一個衡量模型在有限資源下排序效果的指標。儘管lift廣泛用於事後分析，但其做為直接訓練目標仍未被討論。

我們首先以 R 語言的 caret 套件為基礎，設計一套以lift為目標函數的訓練流程。我們接著探討兩個與lift有關的重要問題：其一是資料潛在排序可能導致lift charts產生誤導的資訊，其二是常見的採樣技術對lift的影響。

我們以兩個公開資料集進行實證分析，一為銀行目標行銷資料，另一為電信用戶流失資料。

研究結果指出：（1）直接對於lift進行最佳化可有效提高模型lift的表現，並可能增加模型複雜度（2）重複打亂資料順序可以產生更穩定的lift chart，讓我們能夠觀察lift當中的變異性（3）常見的資料平衡方式（如過採樣與欠採樣)通常無法提升lift的表現，甚至會降低lift的表現。我們的方法和這些發現可以幫助企業更有效地優先排序高價值目標並分配有限資源。

In many business applications—such as direct marketing or churn management—resources are limited, creating a need to identify high-value targets from a large pool of records. Supervised machine learning classifiers are commonly used for predicting likely targets. However, the decision maker’s action (targeting) will only be applied to a subset of the predicted records. Only the top-n records will be targeted. Traditional model performance metrics, such as accuracy and AUC, do not convey the constrained ranking performance. This thesis bridges the gap between standard model evaluation and practical business needs by focusing on lift, a metric that measures constrained ranking performance. Although widely used for post-hoc analysis, lift remains underutilized as a direct training objective.

We start by exploring using lift as an objective function for designing a lift-centric training pipeline, using the popular R caret package. We then studied two lift related issues. The first is the potential for lift charts to mislead due to the data sequence. The second is the effect of common resampling techniques on lift.

We illustrate our method and findings on two public datasets, one on bank direct marketing and the other on telecom customer churn.

Our findings show: (1) Directly optimizing for lift improves lift performance, potentially increasing model complexity. (2) Repeated shuffling of data order leads to a more stable lift chart, allowing us to see the variability in lift values. (3) Common data balancing techniques generally do not improve and often harm lift performance. Our methodology and these findings can help businesses more effectively prioritize high-value targets and allocate limited resources.