Alan,
You have asked a simple question that involves a very complicated explanation to provide an answer.

EU directives in such areas are not very well understood. Whatever is 'legally' allowed may not in practice be fully understood by those people who have authority to deny.
(eg airline staff/airport staff/immigration officers/border guards /embassy staff etc.)

Firstly, the Family Permit is just a name given to a type of visa that allows the holder to enter the UK as the dependent of an EEA citizen. So in that sense it is not considered as a permit to travel to and enter other states. (although it may be possible in a strictly legal technical argument)
It's much less complicated to travel around when the non-EEA spouse actually holds a family member Residence Card.

To minimise any problems when travelling within EU I suggest the following:-

- Always travel together

- Carry both passports and be sure they are current and valid

- The Non-EEA Spouse to carry the formally issued Residence Card (or your countries equivalent)

- Carry an Original marriage certificate

- Carry original birth-certificates

- Carry a copy of Directive 2004/38/EC (better to include an English language one and others in the languages of your destination(s))

Here's an English Language version. of Directive 2004/38/EC

It's worth taking a couple of minutes to at least review Article 5 "Right of Entry"

Here's another good source of information written in a more easily understood manner. The downside is that it covers so much detailed EU information that it needs time to research.

I'm not suggesting that travel without Schenghen is complicated, just that the law can be complicated and not well understood.

Of course you could decide that the potential risks of issues/problems/delays at airports, ports and borders are too much and you'd rather take the Schenghen Visa route.
Yet here again the EU rules are not well understood and you could end up paying for the visa and waiting for a few weeks, when in fact the visa should be free of charges and the application processed without delay.

I hope this writing helps you and doesn't create too many doubts.

EU directives are full of vagaries