Ad: introducing Abacus Embeddings, a simple tweak to positional embeddings that enables LLMs to do addition, multiplication, sorting, and more. Our Abacus Embeddings trained only on 20- digit addition generalise near perfectly to 100+ digits.
https://media.mstdn.social/media_attachments/files/112/520/785/813/110/828/original/f060c9bb875acb13.jpg