fix(inference): prevent touching gpu_ids in config when do inference

fix(inference): prevent touching gpu_ids in config when do inference #2538