关于ProcessNewMessage中RDMA路径和TCP路径差异的疑问 #3205
Description
rdma_performance用例，不设置FLAGS_READ_PROGRESSIVELY，rdma_use_polling=true
1、brpc 1.15.0 版本中，TCP路径会将n个消息全部解析到之后，在最后通过bthread_flush统一进行通知，我理解这是为了避免在解析数据过程中调用signal_task，提升解析性能？但为什么rdma的polling模式是解析一个消息就通知一个呢？
#if BRPC_WITH_RDMA
if (!m->is_read_progressive() && !rdma::FLAGS_rdma_use_polling)
#else
if (!m->is_read_progressive())
#endif
{
// Transfer ownership to last_msg
last_msg.reset(msg.release());
} else {
QueueMessage(msg.release(), &num_bthread_created,
m->_keytable_pool);
bthread_flush();
num_bthread_created = 0;
}
}
if (num_bthread_created) {
bthread_flush();
}
2、brpc 1.15.0 版本中，signal_task限制num_task，那也就是说一次ProcessNewMessage最多驱动的bthread的数量为3？2个是通过signal_task让其他tg去steal执行，1个留在本bthread执行。如果一次ProcessNewMessage识别到的消息比较多的话（n个），其他n-3个消息还是会被留在本tg上等待被处理吗？
void TaskControl::signal_task(int num_task, bthread_tag_t tag) {
if (num_task <= 0) {
return;
}
// TODO(gejun): Current algorithm does not guarantee enough threads will
// be created to match caller's requests. But in another side, there's also
// many useless signalings according to current impl. Capping the concurrency
// is a good balance between performance and timeliness of scheduling.
if (num_task > 2) {
num_task = 2;
}
auto& pl = tag_pl(tag);
size_t start_index = butil::fmix64(pthread_numeric_id()) % _pl_num_of_each_tag;
for (size_t i = 0; i < _pl_num_of_each_tag && num_task > 0; ++i) {
num_task -= pl[start_index].signal(1);
if (++start_index >= _pl_num_of_each_tag) {
start_index = 0;
}
}
if (num_task > 0 &&
FLAGS_bthread_min_concurrency > 0 && // test min_concurrency for performance
_concurrency.load(butil::memory_order_relaxed) < FLAGS_bthread_concurrency) {
// TODO: Reduce this lock
BAIDU_SCOPED_LOCK(g_task_control_mutex);
if (_concurrency.load(butil::memory_order_acquire) < FLAGS_bthread_concurrency) {
add_workers(1, tag);
}
}
}
3、最新的brpc 1.16.0 版本中，rdma polling模式的路径做了修改，改成了和TCP一样，也是在最后通过bthread_flush统一进行通知，这个修改是为了什么呢？
if (!m->is_read_progressive()) {
// Transfer ownership to last_msg
last_msg.reset(msg.release());
} else {
QueueMessage(msg.release(), &num_bthread_created,
m->_keytable_pool);
bthread_flush();
num_bthread_created = 0;
}
}
#if BRPC_WITH_RDMA
// In RDMA polling mode, all messages must be executed in a new bthread and
// not in the bthread where the polling bthread is located, because the
// method for processing messages may call synchronization primitives,
// causing the polling bthread to be scheduled out.
if (rdma::FLAGS_rdma_use_polling) {
QueueMessage(last_msg.release(), &num_bthread_created,
m->_keytable_pool);
}
#endif
if (num_bthread_created) {
bthread_flush();
}
return 0;
期待大佬们的回复！！！