Meteoric Karma
Magic was destroying their world.
The dwarves had been the first to connect the dots — unchecked releases of arcane energy were pooling in the upper stratosphere. If left unchecked, the consequences to the planet would be catastrophic.
The various races, from the Elves to the Fae, voluntarily gave up their magic in the ultimate self-sacrifice. In doing so, they faded from the world.
One race selfishly kept their magic: the Lizardfolk.
A few million years later, a meteor took care of them, too.