I have never cleaned under the extractor or ejector...
Never heard NOT taking the grips off, what is important is how you take them off.
They can be fragile (same with shooting, lots of force there); so you take the screw off, push from inside while pulling out with your thumbnail. Just realizing that if the grip is 'sticking' to the frame it may break, so gentleness is required.
Then clean under them once off.
Ed
PS: To those who collect info, you provided it for their databases by posting here