Add GPU Memory Polling Function #911
Conversation
Add get_gpu_memory function for polling GPUs on a machine and their available vRAM.
Add newline to end of file
Add my name to Authors markdown per SLEAP outside contributor guidelines.
Codecov Report
@@ Coverage Diff @@
## develop #911 +/- ##
===========================================
- Coverage 67.63% 67.58% -0.05%
===========================================
Files 130 130
Lines 22209 22226 +17
===========================================
+ Hits 15020 15022 +2
- Misses 7189 7204 +15
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
TODO to make this usable: Add a flag to Lines 1932 to 1941 in 2688d56 Something like: if args.first_gpu:
sleap.nn.system.use_first_gpu()
logger.info("Using the first GPU for acceleration.")
elif args.last_gpu:
sleap.nn.system.use_last_gpu()
logger.info("Using the last GPU for acceleration.")
else:
if args.gpu == "auto":
gpu_ind = np.argmin(sleap.nn.system.get_gpu_memory().values())
else:
gpu_ind = int(args.gpu)
sleap.nn.system.use_gpu(gpu_ind)
logger.info(f"Using GPU {args.gpu} for acceleration.")And then a similar setup for inference here: Lines 4270 to 4279 in 2688d56 |
sleap/nn/system.py
Outdated
| # Append percent of GPU available to GPU ID | ||
| memory_dict[gpu_id] = round(int(available_memory) / int(total_memory), 4) |
There was a problem hiding this comment.
Do we want this to be a percentage?
EDIT: Do we want this to be a fraction instead of just the available_memory?
There was a problem hiding this comment.
I don't know! I wasn't sure if having it as a percentage would be helpful/more readable/understandable so I just left it that way.
There was a problem hiding this comment.
I also realize now that the comment says percentage but it's not a percent value that's given, so we can change the comment or the value. I have no preference and also don't know what best practice is lol
There was a problem hiding this comment.
I think as it is currently written it is a fraction no? Ratio of:
Available Memory
Total Memory
|
Although we cannot test gpu features through github actions yet, I tested the new additions locally - they passed. |
|
Neato! |
|
This is so much cleaner! I don't know why I didn't think of using the list indices as the index of the card. That makes a lot more sense. Nice one Liezl! |
Description
Imports
subprocessfor invokingnvidia-smiquery that asks for GPU indexes, unused/available memory, and total memory on the card. Returns a dictionary that has GPU index as the key with fraction of GPU available as the value.Types of changes
Does this address any currently open issues?
Nope!
Outside contributors checklist
Are tests required for this function and, if so, what should that look like?
Much redacted documentation in the function, down to write docs elsewhere if you'd like!
Thank you for contributing to SLEAP!
❤️