Conversation

Notices

1. CrunkLord420 (crunklord420@clubcyberia.co)'s status on Friday, 27-Sep-2024 10:32:08 JST CrunkLord420
   The RTX 5090 rumors keep fucking me up. I wrote off the idea of getting when when it was supposed to have 28GB. Now the rumor says 32GB. What do you get with 32GB vs 24GB? No one is making models that fit that size. You're just adding context.
   
   Also that power requirements
   • Matt Hamilton (eriner@noauthority.social)'s status on Friday, 27-Sep-2024 10:45:37 JST Matt Hamilton

     in reply to

     @crunklord420 I have a 7900 XT (20GB VRAM) and I wish I had an extra ~5GB or so. Not only for context, but to be able to use vision-based models (LLAVA, Pixtral, etc) in addition to text LLMs without incurring the time penalty for loading/unloading. I can fit a 22B Minstral Q4 with 12k of context AND a full precision LLAVA model at the same time, but just barely. Sometimes I open a video and the GPU starts flipping out due to VRAM constraints. An extra few GB would really aid my use-case.

     CrunkLord420 likes this.
   • Iska (iska@catposter.club)'s status on Friday, 27-Sep-2024 10:45:47 JST Iska

     in reply to

     @crunklord420@clubcyberia.co What do you get with 32GB vs 24GB?Bandwidth. That's the real reason why higher vram versions perform better

     CrunkLord420 likes this.