Scientists must choose which among many experiments to perform. We study the epistemic success of experimental choice strategies proposed by philosophers of science or executed by scientists themselves. We develop a multi-agent model of the scientific process that jointly formalizes its core aspects: active experimentation, theorizing, and social learning. We find that agents who choose new experiments at random develop the most accurate theories of the world. The agents aiming to confirm, falsify theories, or resolve theoretical disagreements end up with an illusion of epistemic success: they develop promising accounts for the data they collected, while completely misrepresenting the ground truth that they intended to learn about. Agents experimenting in theory-motivated ways acquire less diverse or less representative samples from the ground truth that are also easier to account for. The novelty-seeking agents suffer from collecting unrepresentative observations which do not allow them to build as successful accounts for reality as enabled by random data collection. Thus, random data collection combines virtues of diverse and representative sampling from a target scientific domain. We suggest that randomization, already a gold standard within experiments, is also beneficial at the level of experiments themselves.
Latest posts by Ryan Watkins (see all)
- POTATO: The Portable Text Annotation Tool - March 27, 2023
- Decision-aid or Controller? Steering Human Decision Makers with Algorithms - March 27, 2023
- Human Uncertainty in Concept-Based AI Systems - March 24, 2023