If you're only using the specialty grains for flavor, and not to get gravity points, then no worries. Just make sure they are milled of course, and steep them as you slowly heat your initial water up to boiling. Just take care to remove them before the water gets above 170F or so, else you could add some off flavor to your beer. No worries though.