I think loose grips will get looser with shooting. When fired, the grip frame is propelled backward by the shot. Since you are holding the grips steady, the effect is to slam them forward in relation to the grip frame. What ordinarily resists this would be the forward edges of the material remaining on the backsides of the grips that fits a small fraction of an inch into the open, relieved areas of the sides of the frame. This interface should be snug. A rattling fit will allow the grips to be shocked forward every shot, compressing or further eroding the wood necessary for a snug fit.
In order to avoid this "running start" at messing up the grips, I've added a dab of 1-hr. epoxy where necessary to take up the play. To avoid a lot of fitting/filing/sanding after the epoxy has set, I wait until the epoxy isn't sticky any more, but still a bit
lastic". Just the right amount of repair material will be squished out of the way when the grip is squeezed into place, and the thickness of the epoxy shim will be quite precise when left to cure. One smooth layer of HanidWrap is something I'd recommend for protecting the metal's finish.