Conversation

Notices

1. ✡️ ((fuggy)) :swastika: (fuggy@skippers-bin.com)'s status on Thursday, 14-Mar-2024 13:40:50 JST ✡️ ((fuggy)) :swastika:

   • mischievoustomato

   @mischievoustomato@rebased.taihou.website Yes it is a MoE model so it is super efficient on CPU, just be sure you have above 32 GBs RAM

   • ✡️ ((fuggy)) :swastika: (fuggy@skippers-bin.com)'s status on Thursday, 14-Mar-2024 13:44:14 JST ✡️ ((fuggy)) :swastika:

     in reply to

     • mischievoustomato

     @mischievoustomato@rebased.taihou.website If you have like exactly 32GBs of RAM you might want to unload it in the background, just make a dmenu script like I have and add something like this
     
     curl http://localhost:11434/api/generate -d "{\"model\": \"''${model}\", \"keep_alive\": 0}"
     
     Ignore the dumb escape sequence

   • ✡️ ((fuggy)) :swastika: (fuggy@skippers-bin.com)'s status on Thursday, 14-Mar-2024 13:45:36 JST ✡️ ((fuggy)) :swastika:

     in reply to

     • mischievoustomato

     @mischievoustomato@rebased.taihou.website I have 64 so I mostly just let it auto unload when it wants lol
     
     You can copy my script I should likely just put in it's own file instead of writing it inside Nix lol
     
     https://github.com/nonetrix/nixdots/blob/master/misc/shell-scripts/llm-actions.nix