New onnx custom allocator API#827
Conversation
…rked onnx allocator api
…emory consumption between jemalloc and libc allocators) + remove bert model that served for benchmark only
…disAI into onnx_custom_allocator
Codecov Report
@@ Coverage Diff @@
## master #827 +/- ##
==========================================
+ Coverage 79.97% 81.32% +1.35%
==========================================
Files 53 56 +3
Lines 8009 8182 +173
==========================================
+ Hits 6405 6654 +249
+ Misses 1604 1528 -76
Continue to review full report at Codecov.
|
| * array_new instead */ | ||
| static array_t array_new_sz(uint32_t elem_sz, uint32_t cap, uint32_t len) { | ||
| array_hdr_t *hdr = array_alloc_fn(sizeof(array_hdr_t) + cap * elem_sz); | ||
| array_hdr_t *hdr = (array_hdr_t *)array_alloc_fn(sizeof(array_hdr_t) + cap * elem_sz); |
There was a problem hiding this comment.
C++ compiler raising an error if we don't have this conversion explicitly (it is used form onnx_allocator.cpp)
| OrtAllocator::Alloc = [](OrtAllocator* this_, size_t size) { return static_cast<RAIOrtAllocator*>(this_)->Alloc(size); }; | ||
| OrtAllocator::Free = [](OrtAllocator* this_, void* p) { static_cast<RAIOrtAllocator*>(this_)->Free(p); }; | ||
| OrtAllocator::Info = [](const OrtAllocator* this_) { return static_cast<const RAIOrtAllocator*>(this_)->Info(); }; | ||
| Ort::ThrowOnError(Ort::GetApi().CreateCpuMemoryInfo(OrtDeviceAllocator, OrtMemTypeDefault, &cpu_memory_info)); |
There was a problem hiding this comment.
why this is wrapped with ThrowOnError?
There was a problem hiding this comment.
I copied it from ONNX test. Even thought there is no obvious reason for CreateCpuMemoryInfo to fail, I guess that every API call should be wrapped in case that the return status is not NULL (which means invalid).
| 'AI.MODELEXECUTE', 'mnist_0{1}', 'INPUTS', 1, 'a{1}', 'OUTPUTS', 1, 'b{1}', | ||
| error_msg_is_substr=True) | ||
|
|
||
| def run_parallel_onnx_sessions(con): |
There was a problem hiding this comment.
you can avoid running in parallel if all you need is one session
There was a problem hiding this comment.
This test is here to see that our allocator is "thread-safe" (we have multiple RedisAI workers executing sessions and access the allocator).
Use the new ONNX
RegisterAllocatorAPI in 1.9.0 version so that onnx sessions will use Redis allocator (in CPU models only) for memory management.This include a new load time configuration that sets the memory limit for ONNX (default is to have no limit)
We return an appropriate error and exit gracefully if the memory limit has exceeded.