Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics
arXiv:2402.15415v2 Announce Type: replace-cross
Abstract: Low-Rank Adaptation (LoRA) is the dominant parameter-efficient fine-tuning method due to its favorable compute-performance trade-off, yet it suffers from catastrophic forgetting. We study forge…