Currently using a slightly modified version of ollama and a slightly custom build of llama-cpp stuffed into a container together. Your ollama and llama-cpp performance should be identical (it's both llama-cpp lol)
Don't worry running all local is not great, all of my usable machines are dual processor which means no X8 or X16 slots that are usable in any compo share a single processor (root complex) so I have absolutely abhorrent interconnect speeds, like generally around 500MB/s transfer and it limits my decode to ~50 t/s aggregate (1 model, or 5) in any combination. NCCL can't find a path, and vgpu without a shared root makes SYS P2P routing impossible so im kinda stuck unless I pony up for a different host platform.
Ive been experimenting with embedding and decisions models using a gateway with AI routing to offload some stuff either local, or away from local depending on the task XD
ChipTuner
npub1qd…3fqm7
2026-10-12 00:34:05 UTC
in reply to nevent1q…qrx5
Author Public Key
npub1qdjn8j4gwgmkj3k5un775nq6q3q7mguv5tvajstmkdsqdja2havq03fqm7Seen on
wss://theforest.nostr1.comPublished at
2026-10-12 00:34:05 UTCEvent JSON
{
"id": "1f43821670364877225e93a004735673537616aa0c7be1c23005f6e604505741",
"pubkey": "036533caa872376946d4e4fdea4c1a0441eda38ca2d9d9417bb36006cbaabf58",
"created_at": 1791765245,
"kind": 1,
"tags": [
[
"e",
"39a0931f7f4a8e5cb7e5e5e43335b8c27032933b7ec51ffc6a12f303df00e368",
"wss://theforest.nostr1.com/",
"root",
"036533caa872376946d4e4fdea4c1a0441eda38ca2d9d9417bb36006cbaabf58"
],
[
"e",
"000000e2db5c7a8ffb89c99cde3328893a2f2b954a579313b4e7eb2a62a72c49",
"wss://theforest.nostr1.com/",
"reply",
"75656740209960c74fe373e6943f8a21ab896889d8691276a60f86aadbc8f92a"
],
[
"p",
"036533caa872376946d4e4fdea4c1a0441eda38ca2d9d9417bb36006cbaabf58",
"wss://theforest.nostr1.com/"
],
[
"p",
"576d23dc3db2056d208849462fee358cf9f0f3310a2c63cb6c267a4b9f5848f9",
"wss://eden.nostr.land/"
],
[
"p",
"75656740209960c74fe373e6943f8a21ab896889d8691276a60f86aadbc8f92a"
]
],
"content": "Currently using a slightly modified version of ollama and a slightly custom build of llama-cpp stuffed into a container together. Your ollama and llama-cpp performance should be identical (it's both llama-cpp lol) \n\nDon't worry running all local is not great, all of my usable machines are dual processor which means no X8 or X16 slots that are usable in any compo share a single processor (root complex) so I have absolutely abhorrent interconnect speeds, like generally around 500MB/s transfer and it limits my decode to ~50 t/s aggregate (1 model, or 5) in any combination. NCCL can't find a path, and vgpu without a shared root makes SYS P2P routing impossible so im kinda stuck unless I pony up for a different host platform. \n\nIve been experimenting with embedding and decisions models using a gateway with AI routing to offload some stuff either local, or away from local depending on the task XD ",
"sig": "b5f0146a57d8dc8a3f0ab548be3cfec72343a73ba437bb40050c60bbca64fad46a327f4ce783c8f3a0c2257a30cff156a3030cb664a0572399dfe7978965f374"
}