Way back in 2017, Google’s “Attention Is All You Need” transformed the AI landscape. Now, nine years later, transformers are showing their age.
Their processing prowess is a double-edged sword: while they excel at handling long sequences of data, the sheer computational power needed for such tasks is sky-high. A 10,000-word document could mean 50 million multiplications – and that’s just scratching the surface.
New startups like Subquadratic in Miami are experimenting with sparse attention mechanisms to reduce this burden. Meanwhile, Manifest AI aims to replace attention entirely with a new ‘power retention’ approach, which summarises context more efficiently.
These innovations could make LLMs faster and more energy-efficient, but only time will tell if they can truly outsmart the transformers that dominate today’s market.
