The recent revolution in generative AI has been driven by the rapid growth in training and inference compute for Foundation Models (FMs). This scaling, however, brings unintended consequences, including high energy demand and associated carbon emissions. My research sets out to reverse this trend by targeting a fundamental inefficiency in dominant FM architectures such as Transformers. Currently, these map data into sequences of internal representations, whose length bottlenecks both prompt processing (compute-bound by typically quadratic attention) and output generation (memory-bandwidth bound by key–value cache reads). Yet this length is largely determined upfront by input segmentation (tokenisation) and typically remains fixed across layers; during decoding, it also accumulates unchanged in the key–value cache.
I have prototyped a new class of FM architectures that learn, end-to-end, to compress sequences of internal representations, effectively redefining the model’s “atomic units” for processing and memorising information. To accelerate adoption, I have repurposed existing state-of-the-art open-weight FMs (such as Qwen 3 in collaboration with NVIDIA and OLMo in collaboration with AI2) into adaptive variants and released them publicly.
This leads not only to substantial gains in efficiency (with 8× speedups without accuracy degradation) but also to the emergence of new capabilities: adaptive FMs can operate over broader effective horizons, as they can perceive longer inputs and generate longer outputs under a fixed budget. This enables (1) lifelong learning via a permanent, sub-linearly growing memory, (2) inference-time hyper-scaling for reasoning-intensive tasks (maths, science, coding), and (3) enhanced world modelling for multimodal planning and simulation. Adaptive FMs thus open a path towards greener, more capable generative AI.
Hugging Face checkpoints
State-of-the-art, fully open-source large language model with latent tokenization. Available in 1B and 7B sizes.
8x KV cache compression without quality degradation. Ideal for inference-time scaling.
Related papers
2 PhD studentships and 1 postdoctoral position
AToM is funded by an ERC Starting Grant and hosted at the Department of Computing of Imperial College London. I am recruiting two PhD students and one postdoctoral research associate. All positions are fully funded.
Each position will focus on one or more of the project’s 4 main foci:
Research Associate in Adaptive and Efficient LLM Architectures (job number ENG04037), Department of Computing, South Kensington campus.
Apply through the official Imperial job posting, attaching a full CV with publication list and a research statement (max 2 pages) on the research questions in adaptive and efficient architectures that you find most exciting and why your expertise is relevant. You are also welcome to fill in the expression-of-interest form so we can get in touch early.
PhD applications go through the Department of Computing’s central admissions. Please follow the PhD application guidelines, select ‘Computing Research’ as a degree, and name me as your proposed supervisor. In parallel, fill in the expression-of-interest form (attaching your CV, a link to your best piece of work, and a paragraph on which of the 4 foci above interest you the most). I review submissions on a rolling basis and reach out to promising candidates before the formal deadline.
If you have any specific questions, email me with [phd-atom] or [postdoc-atom] in the subject line. However, please note that due to the volume of request, despite my best effort, I am not always capable of replying to email enquiries.