IJCAI2024: First work to propose a solution to select optimal prompts by IAA. 首篇利用IAA来筛选生成模型最佳提示词的工作.
This repo contains the official implementation of AK4Prompts of the IJCAI 2024 paper.
Create a conda environment with the following command:
conda create -n ak4prompts python=3.10
conda activate ak4prompts
pip install -r requirements.txtRun the demo.ipynb
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.utils import make_image_grid
from AK4Prompts import AK4Prompts
from AK4Prompts_pipeline import AK4PromptsPipeline
device = 'cuda:0'
pipeline= AutoPipelineForText2Image.from_pretrained("stabilityai/sdxl-turbo", torch_dtype=torch.float16).to(device)
ak4prompts = AK4Prompts().to(device)
ak4prompts_path = "/checkpoints/SD1.5_LCMLoRA_S4_aes1_clip2.25_hps2.25/pytorch_model.bin"
ak4prompts.load_state_dict(torch.load(ak4prompts_path))
ak4prompts_pipeline = AK4PromptsPipeline(pipeline=pipeline,ak4prompts=ak4prompts,keywords_filename="keywords_list.txt")
prompt = "vase of mixed flowers"
scores_weights = {'aesthetic':1,'clip':5,'hps':3}
prompt_with_keywords = ak4prompts_pipeline.keywords_ranking(prompt=prompt,scores_weights=scores_weights,topk=10)
image = pipeline(prompt=prompt_with_keywords, num_inference_steps=4, guidance_scale=1.0).images[0]
imageThe demo.ipynb script also demonstrates the evolution of images generated by setting different degrees of specific preferences.
Evaluates the model checkpoint and baselines. Evaluation includes calculating the rewards and storing the images to local.
python evaluation.py --topk 10 --aes 1.0 --clip 5.0 --hps 3.0 --guidance_scale 1.0topk: choose topk keywords
aes,clip,hps: weights for customized keywords-ranking
guidance_scale: classifier-free guidance scale for TIS model
Accelerate will automatically handle multi-GPU setting. The code can work on a single GPU, as we automatically handle gradient accumulation as per the available GPUs in the CUDA_VISIBLE_DEVICES environment variable. If you are using a GPU with a small or big RAM, please edit the per_gpu_capacity variable accordingly.
accelerate launch train.py --config config/train_config.py:aesthetic_clip_hps| 🎁 Projects | 📚 Publication | 🌈 Content | ⭐ Stars |
| Attacker Against IAA Model【美学模型的攻击和安全评估框架】 | TIP 2025 | Code, Dataset | |
| Personalized Aesthetics Assessment【个性化美学评估新范式】 | CVPR 2025 | Code, Dataset | |
| Pixel-level image exposure assessment【首个像素级曝光评估】 | NIPS 2024 | Code, Dataset | |
| Long-tail solution for image aesthetics assessment【美学评估数据不平衡解决方案】 | ICML 2024 | Code | |
| CLIP-based image aesthetics assessment【基于CLIP多因素色彩美学评估】 | Information Fusion 2024 | Code, Dataset | |
| Compare-based image aesthetics assessment【基于对比学习的多因素美学评估】 | ACMMM 2024 | Code | |
| Image color aesthetics assessment【首个色彩美学评估】 | ICCV 2023 | Code, Dataset | |
| Image aesthetics assessment【通用美学评估】 | ACMMM 2023 | Code | |
| Theme-oriented image aesthetics assessment【首个多主题美学评估】 | IJCAI 2022 | Code, Dataset | |
| Select prompt based on image aesthetics assessment【基于美学评估的提示词筛选】 | IJCAI 2024 | Code | |
| Motion rhythm synchronization with beats【动作与韵律对齐】 | IJCAI 2024 | Code, Dataset | |
| Champion Solution for AIGC Image Quality Assessment【NTIRE AIGC图像质量评估赛道冠军】 | CVPRW NTIRE 2024 | Code |