I'm dropped on Skunk2 springs and my 07 fit sport handles 2 yaks on top with a Thule system and all our camping/kayaking gear inside. I've been on some bumpy but hardpacked roads. As long as we go slow and keep an eye out for big bumps/rocks and avoid it carefully I've no problems. Softer terrain is something I wouldn't chance because getting stuck with the fit would probably be less than idea.
And according to scangaugeII I still average 31mpgs on the highway at 65mph with the two kayaks :-) see if the SX4 can do that...