In the case of off-road vehicles, I'd have to say that it most certainly isn't.
Even the people responsible for the behemoth HUMMER H1 and H2 have recognized this ideal in producing a smaller version of the aforementioned in the H3 and in introducing their Hx concept, which is roughly the size of a Jeep Wrangler. This speaks volumes about HUMMER's change to a such a vehicle because when it comes to off-road pedigree, it's hard to cite a vehicle with a stronger lineage than the Jeep (and the Wrangler in particular).
Why is a smaller vehicle better in off-road applications? Other than the obvious reason of being able to fit in tight spaces, there is one main reason for the success of vehicles such as the Wrangler: Breakover Angle.
Breakover Angle is defined as the largest angle of an obstacle that a vehicle can traverse without getting hung up between the front and rear axles. This depends largely on the wheelbase of the vehicle, with shorter wheelbase vehicles having a greater breakover angle.
Of course ground clearance is also a determining factor in the ability to travel over obstacles, but the breakover angle isn't something that can be adjusted easily as can your ground clearance if you have adjustable suspension. Unless you modify the chassis/wheelbase, your breakover angle is largely predetermined from the point of manufacture, and you can't simply adjust this out in the woods as you can with an adjustable suspension. Therefore, if we are to produce an on/off road capable vehicle, the wheelbase is something we will have to research intensely for the appropriate blend of on and off road performance.