Add nexthop strategy plugins #7023

rob05c · 2020-07-21T17:56:56Z

Adds a remap plugin hook to add a nexthop strategy object, and moves and adds other necessary API symbols necessary to implement strategies in plugins.

Also adds an example nexthop strategy plugin.

This adds the following to the API:

enum TSHostStatus
enum TSHostStatusReason
enum TSParentResultType
class TSNextHopSelectionStrategy
struct TSParentResult
const char *ParentResultStr
bool TSHostnameIsSelf(const char* hostname)
bool TSHostStatusGet(const char *hostname, TSHostStatus* status, unsigned int *reason)
void TSHostStatusSet(const char *hostname, TSHostStatus status, const unsigned int down_time, const unsigned int reason)
TSParentResult* TSHttpTxnParentResultGet(TSHttpTxn txnp)
void TSHttpTxnParentResultSet(TSHttpTxn txnp, struct TSParentResult* result)
TSReturnCode TSRemapInitStrategy(TSNextHopSelectionStrategy *&strategy, void *ih, char *errbuf, int errbuf_size)

I tried to minimize API changes, but I didn't see a way to do it any smaller.
The example plugin is also a lot of duplication from the core ConsistentHash strategy. I didn't see a reasonable way to reduce that either, while keeping the ConsistentHash strategy in core for now.
I'm definitely open to suggestions.

include/ts/ts.h

SolidWallOfCode · 2020-08-12T13:18:42Z

include/ts/nexthop.h

-                           const time_t now = 0)                                                           = 0;
-  virtual ~NHHealthStatus() {}
-};
+class TSNextHopSelectionStrategy


Does this make the plugin API require C++?

I can move this to include/tscpp. Is that ok?

Sort of. TSNextHopSelectionStrategy is used by the Remap hook TSRemapInitStrategy, not in InkApi.cc. A Remap plugin can still be C and just not implement that function.

Likewise, TSHttpTxnParentResult(G|S)et in InkApi.cc uses a TSParentResult, which has C++ stuff in it. I just exposed that struct to the API, I didn't create it. If someone doesn't need to call that function, if I move parentresult.h to include/tscpp, they can write a C plugin and just not call the functions that use it.

Is that ok? Or is there a better way?

include/ts/parentresult.h

SolidWallOfCode · 2020-08-12T13:22:13Z

include/ts/parentresult.h

+  int port;
+  bool retry;
+  TSParentResultType result;
+  bool chash_init[TS_MAX_GROUP_RINGS] = {false};


Hmmm, is this C++? If so, std::bitset is a better choice here, otherwise I'd look at making this a bit mask object.

Yeah, the object is C++, unfortunately.

This is an API version of a core struct that already exists. If it's changed here, moving the data between the API and core will have to translate between them, and will probably be less efficient.

Alternatively, I could change it in core. It looks like that wouldn't be too invasive.

I haven't benchmarked, but it was my understanding that std::bitset is optimized for space, and is often less performant than bool[] or bitmasking. Notwithstanding, I'd still prefer to do a bitmask, because the struct only has one C++ member, and I'd like to move it in the direction of C for the sake of the API.

include/ts/ts.h

include/tscpp/api/parentresult.h

SolidWallOfCode · 2020-10-02T22:30:11Z

plugins/experimental/nexthop_strategy_consistenthash/consistenthash.cc

+constexpr std::string_view hash_key_cache         = "cache_key";
+
+static HostRecord *
+chash_lookup(std::shared_ptr<ATSConsistentHash> ring, uint64_t hash_key, ATSConsistentHashIter *iter, bool *wrapped,


This is more than enough state to justify an iterator.

This was moved/copied from core. Not sure what you mean? It takes a ATSConsistentHashIter, that's what that is. You mean taking all these args and wrapping them up in an iterator object, so things would call chIter->begin(), ++chIter, etc?

plugins/experimental/nexthop_strategy_consistenthash/consistenthash.cc

SolidWallOfCode · 2020-10-02T23:42:23Z

plugins/experimental/nexthop_strategy_consistenthash/consistenthash.cc

+      HostRecord *p = host_groups[i][j].get();
+      // need to copy the 'hash_string' or 'hostname' cstring to 'name' for insertion into ATSConsistentHash.
+      if (!p->hash_string.empty()) {
+        p->name = const_cast<char *>(p->hash_string.c_str());


Is const_cast really required here? It's a bit risky.

Also copied from core. Yeah, because HostRecord.name is not const. I was told it couldn't be for reasons, and that it would never be changed. Likewise, I was assured hash_string would always outlive it, making returning the c_str safe.

I can try to verify those things, but it would take some time. FWIW I certainly don't like it either, but didn't see an obvious alternative.

plugins/experimental/nexthop_strategy_consistenthash/consistenthash.cc

SolidWallOfCode · 2020-10-06T14:09:53Z

proxy/ParentSelection.cc

+}
+
+void
+ParentResult::copyTo(TSParentResult *r)


Any reason these aren't assignment operators?

Eh, I lean against obfuscating nontrivial copies, and these funcs are just complex enough to be nontrivial. I can change it if you want / if overriding assignment is the idiom in ATS

src/traffic_server/InkAPI.cc

SolidWallOfCode · 2020-10-06T14:46:49Z

proxy/http/remap/RemapPluginInfo.cc

+  char tmpbuf[2048];
+  ink_zero(tmpbuf);
+  TSNextHopSelectionStrategy *strategy_raw = nullptr;
+  if (init_strategy_cb(strategy_raw, ih, tmpbuf, sizeof(tmpbuf) - 1) != TS_SUCCESS) {


How does strategy_raw become not nullptr?

init_strategy_cb is TSRemapInitStrategy, which takes a TSNextHopSelectionStrategy *&. A plugin sets that pointer-ref.

trafficserver/plugins/experimental/nexthop_strategy_consistenthash/plugin_consistenthash.cc

Line 74 in 00e236d

strategy = static_cast<TSNextHopSelectionStrategy *>(ih);

proxy/http/remap/PluginFactory.h

a-canary · 2020-10-08T20:15:02Z

-0, AFAIK the general plugin design is to use event handlers. Which I realized will take a lot more work but also simplify the API and the HttpSM quite a bit in the end. I hope we can move towards that once Susan gets the ConnectionSM refactored.

SolidWallOfCode · 2020-10-13T12:22:59Z

include/tscpp/api/nexthop.h

-  virtual ~NHPluginStrategy() {}
+  virtual void findNextHop(TSHttpTxn txnp, time_t now = 0)                                                                 = 0;
+  virtual void markNextHop(TSHttpTxn txnp, const char *hostname, const int port, const NHCmd status, const time_t now = 0) = 0;
+  virtual bool nextHopExists(TSHttpTxn txnp)                                                                               = 0;


Is this really useful? Why not have findNextHop return bool?

findNextHop modifies the state to be the next parent; nextHopExists returns without modifying state.
I'd be a big +1 on making findNextHop return and be a pure function. But that'd require refactoring how Strategies work and are called in core. Which I'm guessing itself is a reflection of how ParentSelection always did it.

SolidWallOfCode · 2020-10-13T12:44:21Z

include/tscpp/api/parentresult.h

+  bool wrap_around;
+  bool mapWrapped[2];
+  int last_lookup;
+  ATSConsistentHashIter chashIter[TS_MAX_GROUP_RINGS];


Are the hash values and iterators needed in this result structure? Is this intended as a state object and not just a result?

Yeah, it's a state object. Used by both ParentSelection and Strategies in core (or rather, copied to an identical structure in core).

I.e.
https://github.com/apache/trafficserver/blob/786463/proxy/ParentConsistentHash.cc#L110
https://github.com/apache/trafficserver/blob/786463/proxy/http/remap/NextHopConsistentHash.cc#L335

proxy/http/remap/PluginFactory.cc

brbzull0 · 2020-11-05T15:23:26Z

plugins/experimental/nexthop_strategy_consistenthash/consistenthash_config.cc

+    }
+
+    for (unsigned int i = 0; i < strategies.size(); ++i) {
+      YAML::Node strategy = strategies[i];


Have you considered following this for custom Data Types like TSNextHopSelectionStrategy?

Adds a remap plugin hook to add a strategy object, and moves and adds other necessary API symbols necessary to implement strategies in plugins. Also adds an example strategy plugin.

Also changes the HostStatus and what it exposes, and changes char* to also take the len to make string_view construction faster/easier.

rob05c · 2021-01-28T22:29:16Z

Changing back to a draft: I have an idea to get the best of both worlds, small core changes like this PR, but also powerful and with real continuations, hooks, etc.

I'm not 100% sure the idea will work. If it does, I'll close those. If not, I'll un-draft it.

rob05c · 2021-02-15T16:31:03Z

Closing in favor of #7467

rob05c force-pushed the nexthop-rework-retry-plus-pugins2 branch from 64733d7 to 8f02d13 Compare July 21, 2020 19:05

SolidWallOfCode reviewed Aug 12, 2020

View reviewed changes

include/ts/ts.h Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Aug 12, 2020

View reviewed changes

include/ts/parentresult.h Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Aug 12, 2020

View reviewed changes

randall added the Parent Proxy label Aug 13, 2020

bryancall assigned rob05c Sep 11, 2020

bryancall modified the milestone: 10.0.0 Sep 11, 2020

SolidWallOfCode reviewed Oct 2, 2020

View reviewed changes

include/ts/ts.h Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Oct 2, 2020

View reviewed changes

include/tscpp/api/parentresult.h Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Oct 2, 2020

View reviewed changes

plugins/experimental/nexthop_strategy_consistenthash/consistenthash.cc Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Oct 2, 2020

View reviewed changes

plugins/experimental/nexthop_strategy_consistenthash/consistenthash.cc Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Oct 6, 2020

View reviewed changes

src/traffic_server/InkAPI.cc Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Oct 6, 2020

View reviewed changes

src/traffic_server/InkAPI.cc Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Oct 6, 2020

View reviewed changes

proxy/http/remap/PluginFactory.h Outdated Show resolved Hide resolved

SolidWallOfCode reviewed Oct 13, 2020

View reviewed changes

proxy/http/remap/PluginFactory.cc Outdated Show resolved Hide resolved

rob05c force-pushed the nexthop-rework-retry-plus-pugins2 branch from 3f9cd6e to e75a4a5 Compare October 22, 2020 15:00

rob05c force-pushed the nexthop-rework-retry-plus-pugins2 branch 2 times, most recently from 4c579f3 to e102d13 Compare November 4, 2020 18:50

rob05c marked this pull request as ready for review November 4, 2020 18:50

brbzull0 reviewed Nov 5, 2020

View reviewed changes

Add nexthop strategy plugins

b46f8a9

Adds a remap plugin hook to add a strategy object, and moves and adds other necessary API symbols necessary to implement strategies in plugins. Also adds an example strategy plugin.

rob05c added 14 commits December 14, 2020 12:37

Change NextHop Plugin API to copy result

0f1b33a

clang-format

8441e6b

Change enums to TS prefix, typedef

a247d1b

Move API C++ files to include/tscpp

830be98

Fix NextHop unit tests

387f2d2

Fix comment for TSHostStatusGet+Set

ece3d57

Change TSParentResult port int to in_port_t

6cf3985

Remove plugin nexthop ?true:false

ba46135

Change assignment+if to ?:

bcc3f7f

Change NextHop Strategy funcs to string_view

a9e2a14

Also changes the HostStatus and what it exposes, and changes char* to also take the len to make string_view construction faster/easier.

Change RemapPluginInst.name to string_view

357544f

Change NextHop plugin to not move shared_ptr

b78a74a

Rename PluginInst.getStrategy to initStrategy

a89b8b7

clang-format

9412a14

rob05c force-pushed the nexthop-rework-retry-plus-pugins2 branch from e102d13 to 9412a14 Compare December 14, 2020 19:37

bryancall marked this pull request as draft January 28, 2021 22:28

rob05c closed this Feb 15, 2021

Add nexthop strategy plugins #7023

Add nexthop strategy plugins #7023

Uh oh!

Conversation

rob05c commented Jul 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rob05c Aug 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

a-canary commented Oct 8, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

brbzull0 Nov 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rob05c commented Jan 28, 2021

Uh oh!

rob05c commented Feb 15, 2021

Uh oh!

Uh oh!

rob05c commented Jul 21, 2020 •

edited

Loading

rob05c Aug 12, 2020 •

edited

Loading

brbzull0 Nov 5, 2020 •

edited

Loading