I often wipe the cooker controls when it is hot, and always with a cold wet cloth.
Maybe I've just been lucky, but I wouldn't think they should come off so easily.
Vim and other rought things will remove them over time, but I really don't think you can be blamed for wiping them off!!