ChipTuner on Nostr: A "smart" decision based router unlocks better and possibly more dynamic pricing ...

A "smart" decision based router unlocks better and possibly more dynamic pricing routes. The average home hoster is unlikely to be able to run more than a single usable model loaded at a time, and usually not more than a 1-3 concurrent KV caches. I don't think routerly alone solves this issue, but it facilitates a small decision model that can be tuned. It already supports round-robin queuing and a weighted routing scheme. The embedding and decision model could run somewhat local to the router and offer better per-request routing from the pool of hosters. Commercial services are appearing already of course.