Add multithread support #353
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi, we also have a logic in setup_http function to update threading option to process. |
arctan5x
reviewed
Dec 3, 2019
you mean the logic here? |
GordonWang
approved these changes
Dec 3, 2019
|
@li-wu Yes. That is correct. |
|
@arctan5x I walked through the function |
|
@arctan5x could you help confirm my previous comment? Thanks. |
|
What do you mean by that there is no way to change it to "thread"? |
|
Seems like that there is no dynamic way to change threading mode while the server is running. Since users don't control how server starts (only able to control threading through args/evn vars), how are users using this change? |
I mean there is no logic to change the mode to thread mode in the function |
If users like perf team are using the docker, that might be a problem since they are using /start REST endpoint to start the service. Any good suggestion? |
|
It's ok for our perf team. Actually, we only need to make sure the eventgen server mode using multithread by default. I think we can change the |
|
If perf team is going to change entrypoint.sh, I would say this PR would do its job. One thing I would note is that /setup changes threading to multiprocess. So please note that when you use. |
arctan5x
approved these changes
Dec 16, 2019
jmeixensperger
added a commit
that referenced
this pull request
Apr 6, 2020
* Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Fix dateutil package bug * cleanup memory (#326) * cleanup memory * addressing comment * Fixed standalone healthcheck (#328) * Add python2 libs required by perf test case (#331) * make splunk_eventgen as a module (#332) * Fix oom caused by ujson (#336) * Fix oom caused by ujson * Fix test env * add release automation script (#335) * add release automation script * Fix process leak and start/stop 500 issue (#344) * Fix process leak and start/stop 500 issue * Fix test case fail * Fix high system load issue (#345) * Adding an option (#340) * Adding an option * Adding ujson2 for python3 as well * minor improvements * Scp plugin (#343) * intermediate code for scp_plugin * code clean up * Added doc * revert windbag * Cap * added test * revert (#347) * changing name (#349) * Change (#350) * change token type (#351) * Add multithread support (#353) * Add multithread support * Fix typo * Revert coverage version to 4.5.4 (#355) * Fix CI failure cased by jinja new version and update the log config to reduce log in app (#358) * Add missing documentation for negative floats, as introduced in PR#127 (#360) * Bump nokogiri from 1.10.4 to 1.10.8 in /docs (#359) Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.4 to 1.10.8. - [Release notes](https://github.com/sparklemotion/nokogiri/releases) - [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md) - [Commits](sparklemotion/nokogiri@v1.10.4...v1.10.8) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Li Wu <[email protected]> * Fix jinja plugin bug and test addon sample (#361) * regex stanza bugfix (#365) * prelim changes * fix regex length matching * fix wildcard sample + csv cases * small csv fix, add test case * revert accidental change * Upgraded test instance * add httpevent collector * forgot to update stanza name * add escape for csv matching + test case * add sample, remove stanza * push not working Co-authored-by: tonyl <[email protected]> * Improvement/release script (#368) * add PR to master * update title in sent PR Co-authored-by: Li Wu <[email protected]> * Correct python2/3 issue (#363) * msg must be a bytes like object * msg must be a bytes like object Co-authored-by: Li Wu <[email protected]> * update eventgen version to 7.1.0 Co-authored-by: Li Wu <[email protected]> Co-authored-by: Tony Lee <[email protected]> Co-authored-by: Patrik Nordlén <[email protected]> Co-authored-by: Guodong Wang <[email protected]> Co-authored-by: David Wang <[email protected]> Co-authored-by: Guodong Wang <[email protected]> Co-authored-by: Erwin Vrolijk <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ryan Faircloth <[email protected]>
jmeixensperger
added a commit
that referenced
this pull request
Apr 15, 2020
* Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Fix dateutil package bug * cleanup memory (#326) * cleanup memory * addressing comment * Fixed standalone healthcheck (#328) * Add python2 libs required by perf test case (#331) * make splunk_eventgen as a module (#332) * Fix oom caused by ujson (#336) * Fix oom caused by ujson * Fix test env * add release automation script (#335) * add release automation script * Fix process leak and start/stop 500 issue (#344) * Fix process leak and start/stop 500 issue * Fix test case fail * Fix high system load issue (#345) * Adding an option (#340) * Adding an option * Adding ujson2 for python3 as well * minor improvements * Scp plugin (#343) * intermediate code for scp_plugin * code clean up * Added doc * revert windbag * Cap * added test * revert (#347) * changing name (#349) * Change (#350) * change token type (#351) * Add multithread support (#353) * Add multithread support * Fix typo * Revert coverage version to 4.5.4 (#355) * Fix CI failure cased by jinja new version and update the log config to reduce log in app (#358) * Add missing documentation for negative floats, as introduced in PR#127 (#360) * Bump nokogiri from 1.10.4 to 1.10.8 in /docs (#359) Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.4 to 1.10.8. - [Release notes](https://github.com/sparklemotion/nokogiri/releases) - [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md) - [Commits](sparklemotion/nokogiri@v1.10.4...v1.10.8) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Li Wu <[email protected]> * Fix jinja plugin bug and test addon sample (#361) * regex stanza bugfix (#365) * prelim changes * fix regex length matching * fix wildcard sample + csv cases * small csv fix, add test case * revert accidental change * Upgraded test instance * add httpevent collector * forgot to update stanza name * add escape for csv matching + test case * add sample, remove stanza * push not working Co-authored-by: tonyl <[email protected]> * Improvement/release script (#368) * add PR to master * update title in sent PR Co-authored-by: Li Wu <[email protected]> * Correct python2/3 issue (#363) * msg must be a bytes like object * msg must be a bytes like object Co-authored-by: Li Wu <[email protected]> * Release eventgen 7.1.0. Merge to develop branch. (#372) * Updating Master (#312) * Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Update Master (#314) * Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Release 7.0.0 (#324) * Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Release 7.0.0 (#325) * Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Fix dateutil package bug * update eventgen version to 7.1.0 Co-authored-by: Tony Lee <[email protected]> Co-authored-by: Li Wu <[email protected]> Co-authored-by: Li Wu <[email protected]> Co-authored-by: Tony Lee <[email protected]> Co-authored-by: Jack Meixensperger <[email protected]> Co-authored-by: Patrik Nordlén <[email protected]> Co-authored-by: David Wang <[email protected]> Co-authored-by: Erwin Vrolijk <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ryan Faircloth <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.