Yes, I think you basically got it.
Buffer/block size is too small if you get dropouts and it's too big if the latency (delay between playing and instrument and hearing the sound) interferes with your timing. In other words, a bigger buffer allows your computer to do more work without running out of resources, but also takes more time and thus increases audible latency. It's a balance, but priories change whether you are recording or mixing.