@kushalkolar
Two things:

1. Input on how I am updating the position of the selector. Right now, I am just using a distance measure to find the nearest point on the line. There might be a better alternative if you have any thoughts.
2. When I try to add a ball marker to a 3D line, adding seems to be fine, but the movement is weird. I think maybe it has to do with mapping from screen to world space. The z is always going to be zero by default. Then, when I try to move along the 3d line, I get nans in the position of x and y (which seems strange to me). The 2D case seems to work fine, and I think it is because the z-column is always zero.

1. Input on how I am updating the position of the selector. Right now, I am just using a distance measure to find the nearest point on the line. There might be a better alternative if you have any thoughts.

I think it LGTM, technically distance requires sqrt but we just need argmin and sqrt is monotonic so it doesn't matter.

2. When I try to add a ball marker to a 3D line, adding seems to be fine, but the movement is weird. I think maybe it has to do with mapping from screen to world space. The z is always going to be zero by default. Then, when I try to move along the 3d line, I get `nans` in the position of x and y (which seems strange to me). The 2D case seems to work fine, and I think it is because the z-column is always zero.

I think you could get a guess of the z position of the closest line point by raycasting, I think you should be able to use the Camera to figure out the ray vector that corresponds to the given screen pixel @almarklein ?

Are you interested in the vector, in world space, that points down the screen (in z)?

In that case you could do something like:

ndc = la.vec_transform(world_pos, camera.camera_matrix)
ndc1 = np.array([*ndc[:3], ndc[3]+0.01])
ndc2 = np.array([*ndc[:3], ndc[3]-0.01])

inverse_projection = la.mat_inverse(camera.camera_matrix)
world1 = la.vec_transform(ndc1, inverse_projection)
world2 = la.vec_transform(ndc2, inverse_projection)
vec = world2 - world1