I'm not unhappy with the highway & city street & still-unlocked, off-road performance of my slightly modded 2.5 94 YJ (course, I'm using 29s and will probably not ever go over 31s, at least with this rig). Looks like hell, but with front/rear lockers, think it will soon be an "anybody-got-a-winch?" candidate.
TJ 4s have a little more aftermarket selection for performance upgrades. Easier stuff like doing a decent Cat-back (Borla, Gibson) and maybe one of those AEV shielded intake kits, a Chip (YJs 4s have to use Map Adjusters, instead, though) and maybe one of those performance distributors-type 4 cyl iginition kits will probably be all you need fit your better-performnace bill, if you stay with 31s (I use the same or similar upgrades). All that stuff is about $1000-1200 in parts and the install work you can likely do yourself on a weekend.
From what I read and who I speak with, you have to invest in regearing if you are going to do 33's or 35s with a 4 cylinder, properly. That will run you some money, $800+ for parts/labor. Course, you'll need a suspension lift (or cheaper body lift, which has its own weaknesses) to handle this, which could run you $800-1200+ depending on the parts and your mechanical wherewithall. Then you get in to how you are going to use your Jeep...serious off-road? If so, then you get in to axles, the locker debate. And, then by the time you add that lift, a little more rolling resistance, a little more weight from stronger parts, then you may start thinking about forced Induction or a 8 cylinder swap.
And, of course, some would even say that 31" tires are tall enough to break your D35 rear axle when on the street in some instances, let alone off-road. So, you hopefully have the D44 in the rear, right? :|