NTT R&D Forum2023　Special session2：Natured Inspired Intelligence and a New Paradigm for LLM

The speaker expresses gratitude for being invited to the NTT R&D Forum and introduces Sakana AI, an AI R&D company co-founded with Lion Jones. The company's name, meaning 'fish' in Japanese, symbolizes the collective behavior of fish and the founders' desire to pursue independent and innovative ideas in AI, inspired by nature and evolution. The speaker outlines the talk's agenda, which includes discussing the company's team, vision, technical bets, and the rationale for starting the company in Japan. The speaker's background is highlighted, including his unconventional journey from studying neural networks during a period of limited interest to working in investment banking during the financial crisis, and eventually joining Google Brain to work on generative AI. His research has been nature-inspired, focusing on collective intelligence, evolution, and training AI systems to perform tasks akin to learning and adaptation.

The speaker discusses his skepticism about the current approach to training machine learning models and the pursuit of strong AI through ever-larger models. He advocates for a new foundation model inspired by complex adaptive systems and collective intelligence. The speaker's research background includes work on hyper networks, language models for generating sketches, and video generation models. He emphasizes the need for a more adaptive and energy-efficient approach to AI development, drawing parallels to natural systems that adapt and integrate into their environments. The speaker also touches on his belief that intelligence is about organization and collective behavior, not just the parameters within a neural network.

The speaker shares his perspective on the importance of establishing a strong AI R&D ecosystem in Japan, independent of Western or Chinese influence. He believes that Japan, with its supportive culture for AI and strong economy, is well-positioned to develop a global AI ecosystem. The speaker expresses his intention for Sakana AI to act as a catalyst in this development. He also addresses the challenges of attracting top AI talent in Japan but argues that the country's appeal makes it easier to draw talent, both domestically and internationally. The speaker reflects on his personal journey and the significance of collective intelligence in the pursuit of strong AI.

The speaker, known for his work on the Transformer model, discusses his earlier and subsequent work centered around character-level language modeling. He explains the concept of character-level modeling, where inputs are fed character by character rather than word by word, and its advantages, such as handling out-of-vocabulary words. The speaker details the computational and data challenges of character-level models and the development of techniques to make them effective. He also presents the benefits of character-level models for languages like Japanese, which have rich morphological variations, and encourages the adoption of character-level modeling in the AI community.

The speaker reiterates the benefits of character-level language models, especially for languages like Japanese, and urges the industry to adopt this approach. He contrasts the handling of English and Japanese by word-level and character-level models, highlighting the efficiency and accuracy of the latter for Japanese. The speaker also addresses the potential of character-level models for other character-based languages like Korean and Chinese, suggesting that understanding the deeper structures of language could facilitate learning and translation across various languages.

The speaker explores the capabilities and limitations of large language models, using examples of image generation and spelling accuracy. He points out that while large models can achieve impressive results, they still struggle with tasks that character-level models handle more naturally, such as spelling and reversing words. The speaker emphasizes the need to simplify AI approaches by adopting character-level modeling, which can solve many of the issues associated with word-level models. He concludes by encouraging the use of character-level models, particularly for the Japanese language, and hints at the broader implications for AI development.

In the concluding section, the speaker engages in a Q&A session, addressing questions about the influence of studying Japanese on one's perspective and the potential benefits of character-level modeling for developing multilingual AI systems. He discusses the cross-cultural understanding that multilingual training can provide and the unique challenges and opportunities presented by the Japanese-English language pair. The speaker also contemplates the idea of a universal language structure that AI models can learn, suggesting that training on multiple languages can lead to a deeper understanding of language that transcends specific linguistic pairs.