So I tried what's been mentioned above (aside from filing down the bearing) and tried pushing the bearing in slightly to see if that would make a difference. It made a small difference and doesn't rub as much.
Will keep disassembling and reassembling to see if I get lucky and will keep you all posted.
Thanks for the help everyone!
__________________
|