I keep it in the primary for three weeks at least, then keg. If you have a deep enough pipeline, the beer will clarify, regardless to the vessel it is in. Now if you plan to keep it in a fermenter for an extended time, then removing it from the yeast may be beneficial, but for a normal batch you can just let it settle out in the primary. The yeast may still be working some. Also there is one less vessel to clean.
That being said, if you want to secondary, and you feel it makes your beer better, then it does.