Thanks to visit codestin.com
Credit goes to github.com

Skip to content

How to directly use DNA-Rendering data in 3DGS? #12

New issue
New issue
Closed
Closed
How to directly use DNA-Rendering data in 3DGS?#12
@DNA-Rendering

Description

@DNA-Rendering
DNA-Rendering
opened on May 13, 2024

For users' convenience, we provide data reader functions based on 3DGS to load data from SMC files directly, rather than to preprocess our data with COLMAP. Please check "scripts/3DGS" for your reference.

Let me briefly introduce the changes we made. It loads one pose's multi-view images to train, and save testing views for evaluation.

"scripts/3DGS/SMCReaders.py" is necessary to load SMC files.

In readDNARenderingInfo(), it calls readCamerasDNARendering() to load necessary data and uses the SMPLx vertices as initial points for 3DGS. Views for testing and training are hard-coded, please modify them for your convenience.

DNA-Rendering/scripts/3DGS/dataset_readers.py

Lines 352 to 388 in 1e41b62

def readDNARenderingInfo(path, white_background, eval):
test_view_arr = [3, 5, 7, 9, 11, 15, 17, 19, 21, 23, 27, 29, 31, 33, 35, 39, 43, 45]
train_view_arr = [x for x in range(48) if x not in test_view_arr]
train_info_dict = {
"views": train_view_arr,
"frame_idx": 1,
}
test_info_dict = {
"views": test_view_arr,
"frame_idx": 1,
}
print("Reading Training Transforms", flush=True)
train_cam_infos, smplx_vertices = readCamerasDNARendering(path, train_info_dict, white_background, return_smplx=True)
print("Reading Test Transforms", flush=True)
test_cam_infos, _ = readCamerasDNARendering(path, test_info_dict, white_background, return_smplx=False)
if not eval:
train_cam_infos.extend(test_cam_infos)
test_cam_infos = []
nerf_normalization = getNerfppNorm(train_cam_infos)
parent_dir = os.path.dirname(os.path.dirname(path))
ply_path = os.path.join(parent_dir, "points3d.ply")
print("Using SMPLX vertices to initiate", flush=True)
num_pts, _ = smplx_vertices.shape
shs = np.random.random((num_pts, 3)) / 255.0
pcd = BasicPointCloud(points=smplx_vertices, colors=SH2RGB(shs), normals=np.zeros((num_pts, 3)))
storePly(ply_path, smplx_vertices, SH2RGB(shs) * 255)
scene_info = SceneInfo(point_cloud=pcd,
train_cameras=train_cam_infos,
test_cameras=test_cam_infos,
nerf_normalization=nerf_normalization,
ply_path=ply_path)
return scene_info

In readCamerasDNARendering(), it loads SMPLx parameters and the RGB images, masks RGB images without background, converts the OpenCV cameras to the usable version of 3D gaussian splatting.

DNA-Rendering/scripts/3DGS/dataset_readers.py

Lines 265 to 350 in 1e41b62

def readCamerasDNARendering(path, info_dict, white_background, image_scaling=0.5, return_smplx=False):
output_view = info_dict["views"]
frame_idx = info_dict["frame_idx"]
ratio = image_scaling
cam_infos = []
main_file = path
annot_file = path.replace('main', 'annotations').split('.')[0] + '_annots.smc'
main_reader = SMCReader(main_file)
annot_reader = SMCReader(annot_file)
smplx_vertices = None
if return_smplx:
gender = main_reader.actor_info['gender']
model = SMPLX(
'assets/body_models/smplx/', smpl_type='smplx',
gender=gender, use_face_contour=True, flat_hand_mean=False, use_pca=False,
num_betas=10, num_expression_coeffs=10, ext='npz'
)
smplx_dict = annot_reader.get_SMPLx(Frame_id=frame_idx)
betas = torch.from_numpy(smplx_dict["betas"]).unsqueeze(0).float()
expression = torch.from_numpy(smplx_dict["expression"]).unsqueeze(0).float()
fullpose = torch.from_numpy(smplx_dict["fullpose"]).unsqueeze(0).float()
translation = torch.from_numpy(smplx_dict['transl']).unsqueeze(0).float()
output = model(
betas=betas,
expression=expression,
global_orient = fullpose[:, 0].clone(),
body_pose = fullpose[:, 1:22].clone(),
jaw_pose = fullpose[:, 22].clone(),
leye_pose = fullpose[:, 23].clone(),
reye_pose = fullpose[:, 24].clone(),
left_hand_pose = fullpose[:, 25:40].clone(),
right_hand_pose = fullpose[:, 40:55].clone(),
transl = translation,
return_verts=True)
smplx_vertices = output.vertices.detach().cpu().numpy().squeeze()
parent_dir = os.path.dirname(os.path.dirname(path))
out_img_dir = os.path.join(parent_dir, "images")
# os.makedirs(out_img_dir, exist_ok=True)
bg = np.array([255, 255, 255]) if white_background else np.array([0, 0, 0])
idx = 0
for view_index in output_view:
# Load K, R, T
cam_params = annot_reader.get_Calibration(view_index)
K = cam_params['K']
D = cam_params['D'] # k1, k2, p1, p2, k3
RT = cam_params['RT']
# Load image, mask
image = main_reader.get_img('Camera_5mp', view_index, Image_type='color', Frame_id=frame_idx)
image = cv2.undistort(image, K, D)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
mask = annot_reader.get_mask(view_index, Frame_id=frame_idx)
mask = cv2.undistort(mask, K, D)
mask = mask[..., np.newaxis].astype(np.float32) / 255.0
image = image * mask + bg * (1.0 - mask)
c2w = np.array(RT, dtype=np.float32)
w2c = np.linalg.inv(c2w)
R = np.transpose(w2c[:3,:3]) # R is stored transposed due to 'glm' in CUDA code
T = w2c[:3, 3].copy()
if ratio != 1.0:
H, W = int(image.shape[0] * ratio), int(image.shape[1] * ratio)
image = cv2.resize(image, (W, H), interpolation=cv2.INTER_AREA)
mask = cv2.resize(mask, (W, H), interpolation=cv2.INTER_NEAREST)
K[:2] = K[:2] * ratio
H, W, _ = image.shape
focalX = K[0,0]
focalY = K[1,1]
FovX = focal2fov(focalX, W)
FovY = focal2fov(focalY, H)
image = Image.fromarray(np.array(image, dtype=np.byte), "RGB")
image_name = "%04d" % view_index
image_path = os.path.join(out_img_dir, "%s.png" % image_name)
cam_infos.append(CameraInfo(uid=idx, R=R, T=T, FovY=FovY, FovX=FovX, image=image,
image_path=image_path, image_name=image_name, width=W, height=H))
idx += 1
return cam_infos, smplx_vertices

Finally, we add the necessary entries to integrate into the original 3DGS code base.

DNA-Rendering/scripts/3DGS/dataset_readers.py

Lines 390 to 394 in 1e41b62

sceneLoadTypeCallbacks = {
"Colmap": readColmapSceneInfo,
"Blender" : readNerfSyntheticInfo,
"DNA-Rendeing": readDNARenderingInfo,
}

DNA-Rendering/scripts/3DGS/__init__.py

Lines 48 to 50 in 1e41b62

elif args.source_path.endswith(".smc"):
print("Found smc file, assuming DNA-Rendering data set!")
scene_info = sceneLoadTypeCallbacks["DNA-Rendeing"](args.source_path, args.white_background, args.eval)

Basically you could replace "dataset_readers.py" and "__init__.py" in 3DGS/scene with our scripts.

To use 3DGS with our smc file, please put the smc files as following structure:

|-- root_folder
    |-- main
         |-- xxxx_xx.smc     ### main smc file 
     |-- annotations
         |-- xxxx_xx_annots.smc    ### annotation smc file

Training command: python train.py -s <path to main smc file>

Hope it helps!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions