I can't speak about a 32 or a 36 but I will say I have lived in both ends of our 42 and living in a V berth is not as comfortable for the long term as a cabin with a big enough bed for both of you. This is my opinion and I will bet your wife would think the same.
Lots of things to consider and even though you see the 4' in length there is also the beam which is wider on the 36 so it's a significant increase in interior space.
My suggestion would be choose what your finances can comfortably handle and go with that. You have to consider that the slip fees will be higher with the 36 and most likely you will be paying for a 40' slip. Probably a 35' for the 32 if they offer 35' slips in your area.
Good luck and I know you will love whichever GB you choose. It will be one of the best times of your life.
I lived aboard for 25 years and would do it again.