• Shih-Lun Wu, Yoon Kim, and Cheng-Zhi Anna Huang.
  "MIDI-LLM: Adapting Large Language Models for Text-to-MIDI Music Generation."
  NeurIPS AI4Music Workshop, 2025.

Built on Llama 3.2 (1B) with an extended vocabulary for MIDI tokens.

• Setup
• Inference (Generation) Usage
• Example Prompts
• Training Guidelines
• Citation

• A GPU with 16GB+ VRAM and CUDA 12.x is recommended

• Install Miniconda / Anaconda

• Create and activate Python 3.11 conda environment

conda create -n midi-llm python=3.11
conda activate midi-llm

• Install packages + download soundfont for MIDI-to-audio synthesis

# Conda pkgs for audio processing & synthesis
conda install conda-forge::ffmpeg
conda install conda-forge::fluidsynth

# Soundfont (credit -- '@Frank Wen' https://member.keymusician.com/Member/FluidR3_GM/README.html)
wget https://keymusician01.s3.amazonaws.com/FluidR3_GM.zip
mkdir -p soundfonts
unzip FluidR3_GM.zip -d ./soundfonts/FluidR3_GM
rm FluidR3_GM.zip

• Install PyTorch with pip

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

(Note: this is an example for CUDA 12.6, check PyTorch website if you're on other CUDA versions)

• Check if PyTorch works correctly on CUDA GPU

python -c "import torch; x = torch.randn(30, 30).cuda(); y = x.clone(); z = torch.mm(x, y); print(f'GPU works correctly, output shape: {z.shape}')"

• Install other dependencies

pip install -r requirements.txt

• Verify all installation

python -c "import torch; from vllm import LLM; from anticipation.convert import events_to_midi; print('Setup successful')"

IMPORTANT: We provide two inference backends with different trade-offs:

• vLLM (generate_vllm.py): Faster token generation but more complex setup and longer initialization. Recommended for batch inference (multiple prompts) or interactive sessions.
• Transformers (generate_transformers.py): Simpler setup and faster initialization, but slower generation. Recommended for quick single-prompt testing.

Both scripts share the same arguments (except for --fp8 quantization, which only works in vLLM) and output format.

python generate_transformers.py \
    --prompt "A cheerful rock song with bright electric guitars"

Outputs 4 MIDIs (and synthesized MP3s) conditioned on the same prompt by default.

python generate_vllm.py \
    --prompts_file assets/example_prompts.txt \
    --fp8 \
    --no-synthesize

• assets/example_prompts.txt contains 4 example prompts (one per line).
• --fp8 performs FP8 quantization for faster inference.
• --no-synthesize skips audio synthesis (outputs MIDI only).

python generate_vllm.py \
    --interactive \
    --output_root generations_interactive/ \
    --n_outputs 1

Loads the model once, then lets you enter prompts continuously. Press Enter with an empty prompt to exit.

• Outputs will be stored under generations_interactive/
• --n_outputs 1 generates only 1 output for each prompt

See full options for either script with:

python generate_transformers.py --help # or
python generate_vllm.py --help

[output_root]/
└── 2025-10-30_143022/           # Session timestamp
    ├── 20251030_143022_prompt_1/
    │   ├── prompt.txt
    │   ├── gen_1.mid
    │   ├── gen_1.mp3
    │   └── ...
    └── generation_stats.json

Here are some example prompts to get you started. The model can work with both detailed descriptions similar to what's seen at training, and creative free-form prompts.

A melodic and energetic rock song with a touch of pop influence, featuring synth 
strings, piano, distortion guitar, synth voice, and drums, all contributing to a 
blend of happy and dark moods. Set in the key of A minor with a 4/4 time signature, 
this fast-paced track showcases a chord progression of Bm, Cmaj7, and Gmaj7.

A slow and relaxing classical piece featuring a church organ and French horn, likely 
to be used as a soundtrack in a dramatic or emotional film. Written in A minor and 4/4 
time. The chord progression of E7, Am, and E contributes to the piece's sentimental 
atmosphere.

An energetic and motivating pop song you love to hear on a long road trip.

Upbeat and playful jazz music with lively saxophones, like you're going out on a 
Sunday picnic.

We provide high-level guidance for researchers interested in training their own models. If there is sufficient interest from the community, we will consider releasing the full data processing and training pipeline.

If you find our repo and model useful, please cite our research as

@inproceedings{wu2025midillm,
  title={{MIDI-LLM}: Adapting large language models for text-to-{MIDI} music generation},
  author={Wu, Shih-Lun and Kim, Yoon and Huang, Cheng-Zhi Anna},
  booktitle={Proc. NeurIPS AI4Music Workshop},
  year={2025}
}