Paper deep dive: Evolutionary Optimization of Model Merging Recipes

The video delves into evolutionary optimization techniques for model merging, as explored by the Sakana lab in Japan. Focusing on swarm intelligence and biologically inspired algorithms, the lab, led by David Har, aims to revolutionize AI by steering clear of traditional approaches. The presenter expresses initial skepticism but is eager to dissect the paper and its contribution to the field of model merging, which includes a discussion on existing techniques and the paper's novel approach to automating the merging process across different domains.

The presenter discusses the paper's proposition of using evolutionary algorithms to automate the creation of advanced foundation models through model merging. The paper claims that current model merging relies heavily on human intuition and is somewhat of a 'black art'. The proposed method promises to be more systematic and less reliant on human intervention. The approach includes optimizations in both parameter space and data flow space, aiming to facilitate cross-domain model merging, as exemplified by combining a Japanese model with math reasoning capabilities to create a culturally aware model.

This section provides an overview of existing model merging techniques, including linear or spherical linear interpolation, task vector arithmetic, and recent works specific to language models. The presenter explains how these methods work, such as using task vectors to improve performance on specific tasks by combining them with base models. The discussion also touches on the challenges of interference when combining multiple models and how methods like Ties Merge and Dare address these issues by managing parameter updates more effectively.

The script introduces Franken merging, a technique that stacks layers from different models sequentially to create a deeper model, as opposed to averaging weights. This method is based on the observation that Transformer models' layers mostly pass data through with minimal updates. The presenter also discusses the optimization of data flow space, where the order and scaling of layers are adjusted to improve model performance, in contrast to parameter space optimization that focuses on weight combinations.

The presenter explains the concept of evolutionary computation, specifically the CMA-ES algorithm, which is used for optimizing parameters in model merging. This algorithm initializes a population of candidates, evaluates their performance, selects the best ones, and updates the distribution from which new candidates are sampled. The process is repeated to find the optimal combination of parameters, allowing for a search through a large, non-differentiable space without the need for gradient-based optimization.

The script addresses the challenge of the vast search space in data flow merging and introduces a method to reduce it by fixing the order of layers and only allowing the inclusion or exclusion of each layer. This approach simplifies the search space from an exponential to a linear problem, making it more manageable for evolutionary algorithms. The presenter also mentions the practical benefits of scaling inputs between layers as part of the optimization process.

The presenter shares the results of applying the merging techniques to create a model capable of answering Japanese math questions. The evolutionary algorithm successfully combines a Japanese language model and math models to achieve high accuracy on the task. The results demonstrate that both parameter space and data flow space merging, as well as their combination, outperform the individual models, indicating the effectiveness of the proposed techniques.

The script concludes with an exploration of cross-domain merging that includes image understanding models. The presenter describes how the techniques are applied to combine a vision model with a Japanese language model, resulting in a model that can answer visual questions in Japanese. The success of this approach on different datasets highlights the robustness and generalizability of the merging techniques, even when dealing with very different domains.

In the final section, the presenter discusses the implications of the paper's findings and the potential future of model merging. They contrast the paper's methodical approach with the hit-or-miss method of merging models based on leaderboard performance, emphasizing the importance of generalizability and avoiding overfitting to specific datasets. The presenter expresses excitement for the Sakana lab's vision of a swarm of specialized models that can be merged to form a versatile foundation model, capable of handling a wide range of tasks.