Binary classifier calibration using an ensemble of piecewise linear regression models
Naeini MP, Cooper GF. Binary classifier calibration using an ensemble of regression models. Journal of Knowledge and Information System 2018 Jan;54(1); 151-170. Doi:10.1007/s10115-017-1133-2. Epub 2017 Nov 17. PMID: 29606784. PMCID: PMC5875942.
The discovery of causal relationships from data is an important problem in many fields, including biomedical
science. Algorithms have been developed that learn causal Bayesian networks from observational data, even
when latent confounders are possible. Causal structure discovery from observational data is generally made under
uncertainty. Thus, it is helpful for scientists to know how likely a structure or substructure is to be correct. Previous
researchers have proposed using bootstrap probabilities in Bayesian structure learning; however, the accuracy of
this approach for causal discovery with latent variables has not been studied, to our knowledge. This paper reports
extensive simulation studies to evaluate the accuracy (calibration) of such bootstrap probabilities in causal discovery.
Using a state-of-the-art causal discovery method (RFCI with typical parameter settings), our results show that
bootstrap probabilities are usually well-calibrated for directed edge types in the discovered causal network, especially
for datasets that contain thousands of variables, which is an increasingly common situation in biomedicine and other
fields. Furthermore, the results indicate that the bootstrap RFCI (BRFCI) consistently yields a significantly higher
precision than RFCI in discovering directed causal edges over a wide range of experimental setups, yet it obtains
lower recall. While its recall is lower, BRFCI nonetheless outputs many correct causal relationships, which might
serve as hypotheses that drive experimentation. This suggests that BRFCI may often be a more suitable method for
guiding causal structure discovery in real applications compared to RFCI.