Runtime downloads (don't download during build)

Yup! But after every cold start it will need to download the model again. It’s been a requested feature especially for dreambooth where people are training lots of models and there’s no API to auto-deploy (yet; I believe its landing very soon). There are some use-cases even after that, but it would be much more useful if we had S3-compatible storage at Banana HQ :smiley:

However, once the model has been downloaded (in a particular container, and until the next cold-boot), it can of course be re-used. Multiple models can be stored on disk, and reloaded into memory if the requested model changes. Could be very useful for those using minimum-replicas, to have one “model” deployed that stays up all the time… vs needing to have minimum-replicas for many different models.

We’ll see what develops. Have further speed improvements planned here too. But “slow” cloud-storage is still currently the most limiting aspect.

1 Like