Psh, have you ever felt like a badass trying to grasp your rifle awkwardly in one hand, and trying to line up your follow up pistol shots with your other?
The reason I play dress up and blow so much money on toys is so I can LARP as a bad ass spec ops action hero!
But on a more serious/off topic note, I don't see that as a viable alternative to a sling for serious use of a sidearm. Plus I'm getting funny images of someone holding their gun to their chest butt-down-barrel-up, then dropping it butt-first, and having the gun fire when it hits the ground. They'd get a BB up the nose. Even better if the butt lands on their toes. =D
On a more serious/on topic note, a sling will usually be more helpful than hurtful. It's there to catch your gun if it leaves your hands fast (either voluntarily as you switch to sidearm or give your arms a break, or involuntarily as you do a faceplant). If you get a one point sling (which I absolutely love on my MP5s/P90s - an M4 is almost too tall for a shorter guy like me), it's VERY easy to switch shoulders, and is more manouverable going through windows and stuff. The downside is that it doesn't hold your gun as snuggly to your body as a multi-point sling when you're walking/running around.
You can also get slings that detach easily (generally with a buckle), so if you need to get your gun off fast, you just release the buckle (or buckles if it's a multi point sling), and you don't have to go through the extra motion of bringing the sling over your head.
Finally, you can get into integrating the sling into your gear. Some plate carriers come with a one point sling built into it. I personally like to run a small vest in CQB, and have been experimenting with running a 1 point sling through the D loops on my vest and behind my neck, instead of putting my head through the sling, to put the weight on the vest instead of having it rub around the shoulder straps. Other people do cool things like run a one point sling attached to their stock/back of the receiver, and another attachment point with a buckle on their barrel. When they need to let the gun hang for long periods of time, they'll buckle the barrel to its partner that is attached somewhere on the vest.
So you can get creative with your slings. In general, a lot of the creativity just causes more problems (for example, since I don't run that sling as a loop around my neck, it can't rotate, meaning I have to let my gun hang a bit lower since it requires slack to bring it to my shoulder, instead of just being able to rotate the sling up to my shoulder), but it's up to you how you want to do it.