My mirrors tend to flop inwards at over 100mph (track days obviously). I usually bind them with duck tape but that does not look too slick. My latest fix is to push the rubber boot back and spray liberal amounts of hairspray on the exposed ball. Instantly the flop has gone. Not sure how long it will last bust it is working for now.
Why do I have hairspray (ladies extra hold) in my toolbox? Because it is the best way to fix mountain bike handlebar grips onto the bars - I have little use for hairspray otherwise.