Johannes Uusikuu

Small omlx models without vision & mtp for m1 m2 or low memory macs. Try the oQ63 quants they have 5 bit base but bump a lot of the model to 8 and 6 bit. My experimentation shows that giving a big bit budget bumps up 24 % of the tensors to 8 bit and around 15 percent to 6 bit. I do also experimentation with the last 4 layers in either 8 bit or bf16.