Higher buffer size - higher latency but more stability with the audio signal.Lower buffer size - lower latency but a higher chance of glitches/drop-outs.Using a lower buffer size (and therefore getting less latency) gives your CPU less time to process the audio and therefore the CPU must work much harder to process the audio in that amount of time. You can adjust the buffer size for your requirements. The time this takes is, in part, the buffer size, which you can set yourself. This delay is because the computer needs time to process the audio and route it back out of your interface.
When you are recording audio with your interface, you might notice a slight delay in the audio coming back to you. This article explains the implications of buffer sizes and latency, including how different sample rates have an effect on them.