re-implement dynamic worker coproc spawning #61
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fixed division rounding in pSpawn
Reviewer's Guide by SourceryThis pull request re-implements dynamic worker coproc spawning in Sequence diagram for dynamic worker coproc spawningsequenceDiagram
participant MainThread as Main Thread
participant pSpawnCoproc as pSpawn Coproc
participant WorkerCoproc as Worker Coproc
MainThread->>pSpawnCoproc: Spawns pSpawn coproc
activate pSpawnCoproc
pSpawnCoproc->>MainThread: Waits for fd_nSpawn0 signal
MainThread->>pSpawnCoproc: Sends fd_nSpawn0 signal
loop until quit or max procs
pSpawnCoproc->>MainThread: Reads runLines and runTime from fd_nSpawn
alt System load < pLOAD_max and processing slower than input
pSpawnCoproc->>pSpawnCoproc: Calculates pAdd (number of workers to add)
loop pAdd times
pSpawnCoproc->>MainThread: source coprocSrcCode
MainThread->>WorkerCoproc: Spawns Worker Coproc
end
pSpawnCoproc->>MainThread: Updates worker count
else System load > pLOAD_max
pSpawnCoproc->>pSpawnCoproc: Continues to next iteration
end
end
deactivate pSpawnCoproc
Updated class diagram for CPU load calculation functionsclassDiagram
class _forkrun_get_load {
-loadMaxVal: int
-cpu_user: int
-cpu_nice: int
-cpu_system: int
-cpu_idle: int
-cpu_IOwait: int
-cpu_irq: int
-cpu_softirq: int
-cpu_steal: int
-cpu_guest: int
-cpu_guestnice: int
-tLOAD: int
-tALL: int
-tALL0: int
-cpu_ALL: int
-cpu_ALL0: int
-cpu_LOAD: int
-cpu_LOAD0: int
-pLOAD: int
-pLOAD0: int
-argCount: int
-initFlag: bool
-echoFlag: bool
+():
}
note for _forkrun_get_load "Calculates average CPU load"
class _forkrun_get_load_pid {
-loadMaxVal: int
-tLOAD: int
-tALL0: int
-tALL: int
-cpu_ALL: int
-cpu_ALL0: int
-cpu_LOAD: int
-cpu_LOAD0: int
-pLOAD: int
-pLOAD0: int
-argCount: int
-u0: int
-s0: int
-u1: int
-s1: int
-initFlag: bool
-echoFlag: bool
-grep_str: string
-pidA: array
-cpu_ALLA: array
-cpu_LOADA: array
+():
}
note for _forkrun_get_load_pid "Calculates CPU load for specific PIDs"
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey @jkool702 - I've reviewed your changes - here's some feedback:
Overall Comments:
- It looks like you're trying to optimize coproc spawning based on system load and input data rate, which is a great idea, but the logic is very complex and hard to follow.
- Consider breaking down the
_forkrun_get_load_pidfunction into smaller, more manageable pieces to improve readability.
Here's what I looked at during the review
- 🟡 General issues: 1 issue found
- 🟢 Security: all looks good
- 🟢 Testing: all looks good
- 🟡 Complexity: 1 issue found
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+47
to
+50
| [[ -d /mnt/ramdisk/usr ]] && [[ -z "${findDir}" ]] && { | ||
| findDir='/mnt/ramdisk/usr' | ||
| ramdiskTransferFlag=false | ||
| } |
There was a problem hiding this comment.
question (bug_risk): Review conditional logic for setting findDir.
The new conditional sets findDir based on the existence of /mnt/ramdisk/usr. Ensure that this logic correctly covers all intended cases, especially when findDir is already set or when directory availability changes, to avoid unexpected behavior.
| } | ||
|
|
||
| # get average system load since the last time a new worker coproc was spawned | ||
| mapfile -t pLOADA < <(_forkrun_get_load "${pLOADA0[@]}") |
There was a problem hiding this comment.
issue (complexity): Consider extracting the worker-spawn calculation and CPU load update logic into separate helper functions to improve readability and maintainability.
Consider extracting the nested arithmetic and conditional blocks into separate helper functions. For example, you could pull the worker‐spawn calculation logic into its own function and similarly the CPU load estimation into another. This not only clarifies the intent but also reduces the “inlined” complexity.
For instance, you could refactor the worker count calculation like this:
calculate_worker_addition() {
local pLoadA="$1"
local pLoadMax="$2"
local pLoad1="$3"
local nProcsMax="$4"
local kkProcs="$5"
local nCPU="$6"
local pAddMax=$(( nProcsMax - kkProcs ))
(( pAddMax > ( 1 + ( 2 * nCPU ) ) / 3 )) && pAddMax=$(( ( 1 + ( 2 * nCPU ) ) / 3 ))
local pAdd=$(( ( pLoadMax - pLoadA ) / pLoad1 ))
(( pAdd < 1 )) && pAdd=0
(( pAdd > pAddMax )) && pAdd=$pAddMax
# Additional harmonizing of estimates can go here.
echo "$pAdd"
}Then in your main loop, simply call:
new_workers=$(calculate_worker_addition "$pLOADA" "$pLOAD_max" "$pLOAD1" "$nProcsMax" "$kkProcs" "$nCPU")
(( new_workers < 1 )) && continueSimilarly, consider extracting the CPU load update logic into a helper function. This will make each piece self-contained and easier to manage while keeping functionality intact.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Re-implement dynamic worker coproc spawning in forkrun.bash. This change dynamically adjusts the number of worker coprocs based on system load and data processing rate, optimizing resource utilization.