I think I read, in one of Palmer's books that you would swirl & pitch it within the 1st 18-24 hours of making the starter. After that decant then pitch.
I usually give myself a 2 or 3 day window for when I brew. Not that busy, just lazy. I spin up my starter in advance and put in the fridge until brew day. I always decant since the starter is cold and the yeast has settled out already. If I missed my fermenter volume by a liter or 2, I would just dump it all in.