Group and Shuffle: Researchers at HSE University and AIRI Accelerate Neural Network Fine-Tuning
Researchers at HSE University and the AIRI Institute have proposed a method for quickly fine-tuning neural networks. Their approach involves processing data in groups and then optimally shuffling these groups to improve their interactions. The method outperforms alternatives in image generation and analysis, as well as in fine-tuning text models, all while requiring less memory and training time. The results have been presented at the NeurIPS 2024 Conference.
The larger the neural network, the more challenging it becomes to quickly adapt it to a new task. Retraining a model from scratch is a time-consuming and costly process. Therefore, developers seek cost-effective ways to adapt a model to a specific task while preserving the overall quality of the original.
One such approach is fine-tuning using orthogonal matrices, which, unlike other methods, preserve the essential features of the original model. Popular alternatives, such as block-diagonal or butterfly matrices, have drawbacks: they are either limited in scope or require extensive computations.
Researchers at the HSE Faculty of Computer Science and the AIRI Institute have proposed a new method of constructing matrices, which they call Group-and-Shuffle. Instead of working with all the data at once, they divide the parameters into small groups, process each group separately, and then shuffle them together. This structure is both flexible and efficient: it enables the model to adapt more precisely to the task while requiring fewer computations and less memory.
Building on GS matrices, the researchers developed GSOFT, a new method for orthogonal fine-tuning of neural networks. Unlike previous approaches, GSOFT uses fewer parameters while maintaining training stability and quality, even with limited data. The team also introduced a two-sided version of the method—Double GSOFT—which allows simultaneous adjustment of parameters from both sides, enhancing the model’s flexibility and accuracy.
'We discovered how to construct orthogonal matrices using only two special types of matrices, instead of five or six as required by previous methods. This saves computational resources and training time,' explains Nikolay Yudin, Research Assistant at the HSE Laboratory for Matrix and Tensor Methods in Machine Learning.
The researchers tested the approach on three types of tasks. When fine-tuning the RoBERTa language model, the method outperformed others while using a comparable number of parameters. In image generation, where the model needed to preserve the original features while adapting to the user’s request, GSOFT and Double GSOFT outperformed popular methods like LoRA and BOFT, all while using less memory and training time.
The authors also tested their approach on convolutional neural networks, which are commonly used for image and video analysis, such as in face recognition. The team adapted the GS matrices even for cases where the model required strong resistance to interference and distortion.
'We tested the method across various scenarios—from language and generative models to robust convolutional networks. In every case, it performed reliably while using fewer resources. This confirms that the method can be applied effectively to a variety of purposes,' comments Aibek Alanov, Senior Research Fellow at the Centre of Deep Learning and Bayesian Methods, AI and Digital Science Institute, HSE FCS, and leader of the Controllable Generative AI team at FusionBrain, AIRI.
See also:
HSE University Scholars Uncover E-Learning Preferences of Top Students
HSE University experts have analysed students’ digital footprints and shown for the first time that final grades depend on one’s personal approach to an online course. Balanced students have proven to be more successful than those who follow a more traditional and practical approach. The findings from this study will help create a more adaptive and personalised educational system. This research has been published in the journal The Internet and Higher Education.
HSE Scientists Develop Method to Stabilise Iodine in Solar Cells
Scientists at HSE MIEM, in collaboration with colleagues from China, have developed a method to improve the durability of perovskite solar cells by addressing iodine loss from the material. The researchers introduced quaternary ammonium molecules into the perovskite structure; these molecules form strong electrostatic pairs with iodine ions, effectively anchoring them within the crystal lattice. As a result, the solar cells retain more than 92% of their power after a thousand hours of operation at 85°C. The study has been published in Advanced Energy Materials.
HSE Researchers Create Genome-Wide Map of Quadruplexes
An international team, including researchers from HSE University, has created the first comprehensive map of quadruplexes—unstable DNA structures involved in gene regulation. For the first time, scientists have shown that these structures function in pairs: one is located in a DNA region that initiates gene transcription, while the other lies in a nearby region that enhances this process. In healthy tissues, quadruplexes regulate tissue-specific genes, whereas in cancerous tissues they influence genes responsible for cell growth and division. These findings may contribute to the development of new anticancer drugs that target quadruplexes. The study has been published in Nucleic Acids Research.
Mathematician from HSE University–Nizhny Novgorod Solves Equation Considered Unsolvable in Quadratures Since 19th Century
Mathematician Ivan Remizov from HSE University–Nizhny Novgorod and the Institute for Information Transmission Problems of the Russian Academy of Sciences has made a conceptual breakthrough in the theory of differential equations. He has derived a universal formula for solving problems that had been considered unsolvable in quadratures for more than 190 years. This result fundamentally reshapes one of the oldest areas of mathematics and has potential to have important implications for fundamental physics and economics. The paper has been published in Vladikavkaz Mathematical Journal.
Scientists Reveal How Language Supports Complex Cognitive Processing in the Brain
Valeria Vinogradova, a researcher at HSE University, together with British colleagues, studied how language proficiency affects cognitive processing in deaf adults. The study showed that higher language proficiency—regardless of whether the language is signed or spoken—is associated with higher activity and stronger functional connectivity within the brain network responsible for cognitive task performance. The findings have been published in Cerebral Cortex.
HSE AI Research Centre Simplifies Particle Physics Experiments
Scientists at the HSE AI Research Centre have developed a novel approach to determining robustness in deep learning models. Their method works eight times faster than an exhaustive model search and significantly reduces the need for manual verification. It can be applied to particle physics problems using neural networks of various architectures. The study has been published in IEEE Access.
Educational Programmes on Robotics and Neural Network Technologies Launch at HSE University’s Faculty of Computer Science
Every year, in response to IT industry demands, the Higher School of Economics Faculty of Computer Science launches new educational programmes while updating existing ones. In 2026, the faculty introduced Bachelor’s and Master’s degree programmes in robotics for the first time.
Scientists Show That Peer Influence Can Be as Effective as Expert Advice
Eating habits can be shaped not only by the authority of medical experts but also through ordinary conversations among friends. Researchers at HSE University have shown that advice from peers to reduce sugar consumption is just as effective as advice from experts. The study's findings have been published in Frontiers in Nutrition.
HSE University Develops Tool for Assessing Text Complexity in Low-Resource Languages
Researchers at the HSE Centre for Language and Brain have developed a tool for assessing text complexity in low-resource languages. The first version supports several of Russia’s minority languages, including Adyghe, Bashkir, Buryat, Tatar, Ossetian, and Udmurt. This is the first tool of its kind designed specifically for these languages, taking into account their unique morphological and lexical features.
HSE Scientists Uncover How Authoritativeness Shapes Trust
Researchers at the HSE Institute for Cognitive Neuroscience have studied how the brain responds to audio deepfakes—realistic fake speech recordings created using AI. The study shows that people tend to trust the current opinion of an authoritative speaker even when new statements contradict the speaker’s previous position. This effect also occurs when the statement conflicts with the listener’s internal attitudes. The research has been published in the journal NeuroImage.