Okay, since I can't keep well enough alone I decided to disassemble the mirror and see what makes it tick. Hopefully this will be helpful to others who I'm sure will have the same problem.
If you are trying to tighten the two screws under the ball joint, do not loosen them till they fall out. The point of no return happens if you turn the mirror upside down and the nuts for those bolts falls out of the bracket. You're SOL then. Well, not really.
You'll hear the two nuts bouncing around in the housing. There's no way to get them out or back into place without taking it all apart.
So, I took the mirror apart. I used a feeler gauge to slip between the mirror and the housing. If you're mirrors are like mine they are old. Mine is 23 years old so the adhesive on the mirror is old and should pry loose if you're careful. I started at the bottom.
Inside is a U bracket with a upper ball joint that has several washers on it. If you want it tight replace some of the shims with a washer. Anything that increases the pack thickness will force the ball joint to be tighter. Once your done put the shims back into the upper cup. You can try to put the brass nuts back onto the bracket or do it once it's back in place. It's a bit of a tight fit to get it all in there. You might also want to add a little RTV to the nuts so they won't ever fall out of the bracket.
I ended up flipping some of the shims the opposite direction since they're cupped. It tightened it a bit more but if you really want it tight add a washer.
Once that's done you'll want to scrape the dried adhesive off the back of the mirror as well as the housing. Be careful not to scratch the surface as you'll see those scratches on the other side. I used two line of RTV on the housing o re-attach the mirror.