I ran 4.11's and 32's for 5 years and no real problems. My Jeep wasn't as heavy as it is now though. I'm not running 4.56's and 35's and I'm running about the same. (70mph@3000rpm) Cruises with a headwind in 4th isn't too bad. One of the biggest differences though, highway speeds have shot up. With speeds up from 55 mph to 65 mph, people are cruising at 75-80mph on the highways. This is really tough with a Jeep. You're not only underpowered, you're also fighting with aerodynamics. On highways in town, I can get over 80 mph no problem. The noise buffers and buildings help a lot with the wind.
Offroad, you should have no problems with up to medium trails. On a 0-5 scale, you could go a 3.5 without too much trouble. I've been over the Rubicon with a TJ with the only modification was 31" AT tires. The other TJ in the group had a 3" RE lift and 33" MT's. One XJ had 32" TSL and stock gears. All made it with very few problems. Remember that you'll also air down which should drop the overall height 1"-2".