Illusion is dead on with trying the empty slide (i.e. nothing in it) on the lower. If it slides nicely...DON'T file anything on the slide or frame rails.
With just the blow back unit tightened in the slide...does it slide just as smoothly? If not, where is it rubbing? Those are the specific areas that you'll need to address. You're essentially custom fitting the housing to that frame and that slide.
I used honing stones...because they are flat, easy to handle and I have a number of different grits. The coarser ones did most of the work, the finer ones polished and took off any burrs.
Reference the following picture for the rest of this comment:
Note: It's "upside down" in the pic. When tightened into the slide, my housing was torqued such that the two "arms" (top, far left of the pic) were riding low and rubbing against the frame/hammer assembly. So 95% of the filing I did was to taper those "arms".
Also referencing the picture. I also had a bit of an issue where the "bump" (top, right end of the housing in the pic) was getting hung up with the trigger. The bump would ride back and cock the hammer as normal, but the corner of the hammer would get caught sometimes just inside the bump as the slide was moving forward. I ended up re-profiling the corner of the hammer so it wouldn't catch. I did not remove too much material so that the hammer would fully/reliably cock with each shot, but enough so that the slide will return to battery with little/no-resistance from the hammer.
Hope that helps...
About 4 hours into it...and you're thinking..."I should have left it stock"...but it's worth it in the end.