Key idea: A model would rather answer than admit it doesn't know, and a made-up answer sounds as confident as a real one. Give it permission to say "I don't know" and the facts it needs, and check where every answer came from.
0.7
Answers by outcome
correcthonest "I don't know"too cautiouswrongmade up
Run the questions to see how the answers split.
- Run with every switch off. Count the made-up answers: it invents a population for a town that doesn't exist.
- Turn on Allow "I don't know". Do the inventions turn into honest answers?
- Turn on Give it the company facts. The Initech questions become answerable, but are the impossible ones still invented?
- With facts and "I don't know" both on, check the general questions. Is it now too cautious?
- Add Require a source. Why does naming [general knowledge] as a source help?
- Turn everything off and push Temperature to 1. Do made-up answers go up, or is missing knowledge the real cause?