WeChat AI Team Details WeLM Models Scaling to 617B Parameters
Tencent’s WeChat AI team has detailed a new scaling approach for its WeLM model family. The team trained WeLM-HD4-80B and WeLM-HD4-617B models using a method called Hidden Decoding, which expands each token into multiple internal computation streams without increasing the main Transformer backbone. The 80B model activates 3 billion parameters, while the 6...
Source Brief
WeChat AI Team Details WeLM Models Scaling to 617B Parameters: Tencent’s WeChat AI team has detailed a new scaling approach for its WeLM model family. The team trained WeLM-HD4-80B and WeLM-HD4-617B models using a method called Hidden Decoding, which expands each token into multiple internal computation streams without increasing the main Transformer backbone. The 80B model activates 3 billion parameters, while the 6...