Evaluation of a belief-network-based reminder system that learns from utility feedback

Wagner MM, Cooper GF. Evaluation of a belief-network-based reminder system that learns from utility feedback. In: Proceedings of the Symposium on Computer Applications to Medical Care (1995) 666–672.  PMID:  8563370

PRETRIEVE is a belief-network-based, unsolicited information-retrieval system that performs machine learning based on userfeedback. We report here on the document-ordering and document-retrieval performance ofPRETRIEVE. We developed a test collection of 410 judgments of document utility in a simulated medical orderentry context. We characterized the validity of these judgments, which were elicited from domain experts, by measuring interrater and intrarater reproducibility. We developed a measure of the quality of document orderings similar to the ROCcurve analysis used to evaluate document-retrieval systems. We found that the ordering performance of the PRETRIEVE system was (1) substantially better than random, (2) somewhat less than ideal, and (3) superior to that of versions of the PRETRIEVE system that used relevance feedback instead of utility feedback. Under a set of assumptions, which we make explicit, we found that the documents retrieved by a version of PRETRIEVE that modeled time cost were of higher utility than those retrieved by a similar rule-based system

Publication Year: 
1995
Publication Credits: 
Wagner MM, Cooper GF.
AttachmentSize
PDF icon Wagner.pdf1.55 MB
^