When learners escape models of their behavior
Authors of the Article
- Jean Czerlinski Whitmore, Software engineer at Google
- Escape pressures prompt agents to escape models of their behaviors, leaving the models at best useless, at worst counterproductive. Examples of escape from models include:
- The spam industry gamed the spam filters for Viagra, using various tricks such as voluntary spelling mistakes, use of images, etc.
- The pet food industry, which artificially increased the measured protein content of their pet food by using components causing the chemical test to produce higher results with lower protein content.
- The academic world, where there is a high pressure to publish a lot of papers without strong controls on quality.
- Such escape pressures are caused by:
- Visible and clearly specified goals
- High difficulty in achieving goals
- Significant gap in terms of outcome (success vs failure lead to considerably different outcomes)
- Modelers should take into account the existence of escape behaviors, and build their models accordingly:
- Use multiple models, not a single one to avoid model-specific limits and biases
- Use updatable or modular models to be able to adapt to new escape behaviors
- Use explainable models to differentiate between bad luck and model failure when predictions are wrong, and hence determine when to update the model
- Limit available information about your model, and limiting the possibility for others to test the model so they can’t easily understand it and game it.
- Escape pressure is a significant factor in finance. As regulations are announced publicly and are slow to be updated, financiers have room for escape. This may justify the increased complexity of financial regulations trying to capture all observed escape behaviors. Pushing agents to have more “skin in the game” (see Taleb) might be a good way to limit escape behaviors in finance.
"Too much striving for measured goals hurts unmeasured goals."
"Every model excludes some aspects of reality—ideally the unimportant aspects. (If it included all of reality, then it would be reality.) But when a model is used to control human behavior, the humans can react by changing what is important and thereby escape the model."
"Modelers should study conditions that make escape from their models likely and develop better meta-modeling techniques to deal with escape."