RWKV explores recurrent language models
May 22, 2023
The RWKV paper describes a recurrent language-model architecture that supports parallel training and fixed-size recurrent state during generation.
An open research direction explores language models beyond standard Transformer attention. Its repository provides Apache-2.0 code.
Explore on Humanity Unfolds
People
- Bo Peng
- Eric Alcaide
- Quentin Anthony
- Alon Albalak
- Samuel Arcadinho
- Huanqi Cao
- Xin Cheng
- Michael Chung
- Matteo Grella
- Kranthi Kiran GV
- Xuzheng He
- Haowen Hou
- Przemyslaw Kazienko
- Jan Kocon
- Jiaming Kong
- Bartlomiej Koptyra
- Hayden Lau
- Krishna Sri Ipsit Mantri
- Ferdinand Mom
- Atsushi Saito
- Xiangru Tang
- Bolun Wang
- Johan S. Wind
- Stanislaw Wozniak
- Ruichong Zhang
- Zhenyuan Zhang
- Qihang Zhao
- Peng Zhou
- Jian Zhu
- Rui-Jie Zhu
Sources