If you wear issued CADPAT then a crusty old RSM will butt rape you till you cry. [/sarcasm]
Don't worry about getting jacked for having the issued LTWT's (regular shirt or trousers), but you'll be trying your luck when/if you get into stuff like ICE Goretex and some of the 'harder to get' bits of kit. Trust me.
Browse around on eBay or contact Army Issue if you want some good deals on CADPAT bits and bobs. If you're looking at repro CADPAT but want it to 'look real' I'd suggest Frontenac.