A Hybrid Data-Driven Machine Learning Approach for Anomaly Detection

Abstract

Anomaly detection is a vital application field of machine learning. Given the widespread problems of extreme class imbalance and complex nonlinearity in high-dimensional data, traditional anomaly detection methods can hardly achieve both high-precision identification and model interpretability. Taking non-invasive prenatal testing data as the research background, this paper proposes a two-stage expert model and constructs an anomaly detection strategy integrating statistical analysis and machine learning. After appropriate data oversampling, the random forest algorithm is used to preliminarily verify the reliability of the results, and the recall rate of each expert model on the test set exceeds 0.9. SHAP is further introduced to analyze the main effects and interaction effects, indicating that the Z-score of each chromosome plays a core role in its corresponding model. Combined with index quality control in NIPT theory, this paper innovatively applies SHAP single-feature dependence analysis to explore the stability of quality control indicators, constructs a standard quality control score, and divides samples into high, medium, and low risk regions, with an abnormality rate of only 7.6% in the low-risk region. On this basis, the results of multi-index collaborative quality control are analyzed and predicted.