LoRa in the IOT community are the low power long range wireless standards devices that are utilized to send a signal over large distances. However, in the AI field LoRa implies Low rank adaptation – which is relatively more efficient compute way to fine tune pretrained models (LLM, vision transformers, diffusion models).
With a pretrained model is general and to improve its performance on a particular task a process called fine-tuning needs to be performed. Now, it is easier, to fine tune a model for a task rather than develop a brand new model from scratch. Not only easier but also computationally less intense especially since restart with all the data and resources are not required to make a new model for a new task.
Large models by definition are large and fine tuning on the new dataset is computation expensive, needs data and is harder to share. To accommodate fine tuning other methods have been tried – such as bottleneck adaptors, prefix/prompt tuning which involve modifying all model weights which take resources and can lead to issues like forgetting wherein the model loses the capability that existed before in the model.
LoRa (low rank adaptation) focuses on a select set of parameters to modify. This is called low-rank decomposition to adapt the large model but freeze the pretrained model weights intact. Low rank matrices called matrix A and matrix B. Matrix A is for the changes that are needed and matrix B helps in projecting matrix A changes back to the original parameter space. Thus, knowledge in the core model is unchanged but now the new model can adapt to new tasks.
These matrices are placed in attention layers in transformers or in cross-attention and self attention in diffusion models.
The workflow is simple – freeze the base model → inject LoRa modules into specific layer, train LoRa parameters on yoru dataset. Then you deploy either as merged with base model or as separate modules for multi-task adaptability.
As example, it can be used for modifying a LLM for domain specific knowledge, transfer a new style in image generation.
