The mag spring is (or at least should be) way stronger than the latch spring, so it should push up easily as long as the mag follower can hit it and move it far enough. You may be on the right track, but I really think your problem is in the movement of the parts.
|