To get rid of visas for non-EU partners (spouses) of British and other EU citizens who simply want to enjoy a quick holiday without silly paperwork would be possible if the UK joined the Schengen area (drawback: more asylumseekers to the UK unlessthe EU reforms this along the lines of what I wrote or what Merkel, Juncker etc. are proposing).
An other option would be that an EU spouse permit gives acces to all other EU nations, just as those who have a "family member of an EU/EEA national" residencecard already can. I too find it frustrating that a Dutchy or German who wants to go to the UK for a short holiday, and a Britton who wants to have a short holiday in Spain or Germany, need to go through paperwork. They are rather unlikey to bother anyone and under the Freedom of Movement directives certain rules (saveguards) are in place: it's fine to visit or even move to an other memberstate aslong as you are not an unreasonable burden to that state.
So I really hope that the seperate issue of how to properly deal with asylumseekers (who have false and true refugees amongst them, who would prefer to move to certain nations over other which would burden specific memberstates etc.) on the one hand. This could be done by proper agreements (EU directives?) but they ahve tried this since 2001 or so, but the EU Commission got nowhere as this would give the topic of asylum-migration out of their hands and it would be 'Brussel' that would dictate who ends up where and memberstates don't seem too eager to be willing to do that if this would/could mean an increase in refugees to their own country (NIMBY?). And then for EU citizens and their family we could update the Freedom of Movement directive so that we can travel around freely as if all non-EU spouses (and minors) had a "family member of an EU/EEA national" residencecard. UK and Schengen visa pretty much look alike, if they'd also do that to residencecards and have them in 2-3 languages (national language, English and either Spanish/French/German as a third), that would help the borderguards a lot. The UK could keep it's own visa policy and it's more easily to guard external UK border.