Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government …
Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts
Ben Green, Yiling Chen
CSCW'21: ACM Conference on Computer-Supported Cooperative Work and Social Computing
Session: Algorithms and Decision Making
Abstract
Governments are increasingly turning to algorithmic risk assessments when making important decisions (such as whether to release criminal defendants before trial). Policymakers assert that providing public servants with algorithms will improve human risk predictions and thereby lead to better (e.g., fairer) decisions. Yet because many policy decisions require balancing risk-reduction with competing goals, improving the accuracy of predictions may not necessarily improve the quality of decisions. Through an experiment with 2,140 lay participants simulating two high-stakes government contexts, we interrogate the assumption that improving human prediction accuracy with risk assessments will improve human decisions. We provide the first direct evidence that risk assessments can systematically alter how people factor risk into their decisions. These shifts counteract the potential benefits of improved prediction accuracy. In the pretrial setting of our experiment, the risk assessment made participants more sensitive to increases in perceived risk when making decisions; this shift increased the racial disparity in pretrial detention by 1.9%. In the government home improvement loans setting of our experiment, the risk assessment made participants more risk-averse when making decisions; this shift reduced government aid by 8.3%. These results demonstrate the potential limits and harms of efforts to improve public policy by incorporating predictive algorithms into multifaceted policy decisions. If these observed behaviors occurred in practice, presenting algorithms to public servants would generate unexpected and unjust shifts in public policy without being subject to democratic deliberation or oversight.
DOI:: https://doi.org/10.1145/3479562
WEB:: https://cscw.acm.org/2021/
Pre-recorded presentations of CSCW 2021
Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts
Ben Green, Yiling Chen
CSCW’21: ACM Conference on Computer-Supported Cooperative Work and Social Computing
Session: Algorithms and Decision Making
Abstract
Governments are increasingly turning to algorithmic risk assessments when making important decisions (such as whether to release criminal defendants before trial). Policymakers assert that providing public servants with algorithms will improve human risk predictions and thereby lead to better (e.g., fairer) decisions. Yet because many policy decisions require balancing risk-reduction with competing goals, improving the accuracy of predictions may not necessarily improve the quality of decisions. Through an experiment with 2,140 lay participants simulating two high-stakes government contexts, we interrogate the assumption that improving human prediction accuracy with risk assessments will improve human decisions. We provide the first direct evidence that risk assessments can systematically alter how people factor risk into their decisions. These shifts counteract the potential benefits of improved prediction accuracy. In the pretrial setting of our experiment, the risk assessment made participants more sensitive to increases in perceived risk when making decisions; this shift increased the racial disparity in pretrial detention by 1.9%. In the government home improvement loans setting of our experiment, the risk assessment made participants more risk-averse when making decisions; this shift reduced government aid by 8.3%. These results demonstrate the potential limits and harms of efforts to improve public policy by incorporating predictive algorithms into multifaceted policy decisions. If these observed behaviors occurred in practice, presenting algorithms to public servants would generate unexpected and unjust shifts in public policy without being subject to democratic deliberation or oversight.
DOI:: https://doi.org/10.1145/3479562
WEB:: https://cscw.acm.org/2021/
Pre-recorded presentations of CSCW 2021
