The current PEFT LoRA modules map to the wrong target modules for Phi-2 since the architecture was recently updated.

I created a PR on the PEFT repo to update this to new target modules: huggingface/peft#1375

For now, we will fall back to target_modules for Phi-2 that lead to good model performance.