If you need a library, you will download it anyway. But you can download it once, in base image, or multiple times. IMHO, the fatter _base_ image is, the better.

Ideally, base image must be full installation of everything, one large image for all. You will just download it, and it will just work.