Model Memory Requirements
#3
by
nvip1204
- opened
Can I run it on 4x3090?
number of parameter * size of each parameter in bytes = 72 * 10^9 * 2 bytes = 144 GB of VRAM if running in bf16, 72 GB for fp8, 36 for 4bit datatype so just add up the VRAM for your infrastructure and see if that falls under this
number of parameter * size of each parameter in bytes = 72 * 10^9 * 2 bytes = 144 GB of VRAM if running in bf16, 72 GB for fp8, 36 for 4bit datatype so just add up the VRAM for your infrastructure and see if that falls under this
And that's just the Vram to load the model itself onto, it also needs Vram to actually process the queries (plus whatever else your OS and other apps are using, assuming you don't use your PC for nothing but loading this model)..