It seems like i've got the same issue with my newly bought TM M4A. But i find if i hold the mag in (so its not wobbling around - is wobbling normal?) it doesn't give me this issue.
Granted my observation is based on only a couple of hundred rounds shot within the last few hours