Description

In some instances, libcudf and/or rapidsmpf may call Thrust algorithms with an execution policy with a custom allocator that allocates device-accessible host memory. This was observed to fail on some systems, because trivial_copy_from_device uses cudaMemcpyDeviceToHost which becomes a cudaErrorInvalidValue. If we instead use cudaMemcpyDefault, the CUDA runtime correctly infers the source/destination locations and no error is observed.

Sample error output from @nirandaperera:

[13:01:35.917][140065985130496][CUDA][E] Returning 1 (CUDA_ERROR_INVALID_VALUE) from cuMemcpyDtoHAsync_v2
terminate called after throwing an instance of 'thrust::system::system_error'
  what():  __copy:: D->H: failed: cudaErrorInvalidValue: invalid argument

I added a basic test that calls trivial_copy_(from|to)_device with host pointers. This should cover the failure case, though it is actually triggered by some more complex Thrust algorithms in cudf::pack that I did not think were worth replicating as a unit test. Being able to use (pinned) host memory more easily is becoming important for performance, so we'd like to support this behavior.

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.