We went to Gibraltar today driving in.

The woman immigration officer looked carefully at my wife's passport and her Residencia Card, although did not appear to check my Irish Passport. She spoke to another officer and then waved us through the border.

So it works.