fixed /template_list too long to load #4017
Conversation
WalkthroughThis pull request updates the Changes
Sequence Diagram(s)sequenceDiagram
participant User as User
participant TL as TemplateList
participant GTF as get_template_files
participant P as Paginator
participant TPE as ThreadPoolExecutor
participant PTB as process_template_batch
User->>TL: Request template list
TL->>GTF: Retrieve template files from directory
GTF-->>TL: Return sorted and cached templates
TL->>P: Create paginator with templates (per_page=20)
P-->>TL: Return current page templates
TL->>TPE: Process current page templates concurrently
TPE->>PTB: Execute template extraction
PTB-->>TPE: Return processed template info
TPE-->>TL: Aggregate processed templates
TL->>User: Render view with templates and pagination info
Assessment against linked issues
Suggested reviewers
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
website/views/core.py (2)
1983-1987: Refine docstring for clarity.
Currently, the docstring briefly mentions optimized pagination. Consider describing the concurrency usage, caching details, and how filters (e.g., “with_sidenav”) work to give maintainers a clearer understanding of the function’s overall behavior.
2148-2212: Extend pagination & directory traversal checks.
- The code scans only top-level subdirectories within the main template directory. If deeper nesting is needed, consider a recursive approach.
- The fallback to page 1 when an invalid page is detected is appropriate, but validating negative or zero page numbers would further harden pagination.
- The final context building is clear, nicely grouping templates by directory and aggregating their metadata.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
website/views/core.py(3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: Run Tests
- GitHub Check: docker-test
🔇 Additional comments (2)
website/views/core.py (2)
2001-2096: Validate file reads & error handling in helper methods.
The helper functionsextract_template_infoandget_template_filesprovide caching and file scanning, which is great for performance. However, repeatedly reading files withopen(..., "r")might pose a risk when dealing with large or malformed files. Consider adding:
- A size check early for each file if extremely large files may appear.
- A broader exception handling instead of
IOErroronly, to guard against unexpected file content or encoding errors.
2097-2146: Validate ThreadPoolExecutor size & concurrency.
Using a fixedmax_workers=4ensures parallel processing doesn’t overwhelm system resources. This is fine for modest concurrency. If the template directory grows large, you may consider a more dynamic pool size or a queue-based approach. Overall, the concurrency usage here is clear and reasonably safe.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
website/views/core.py (1)
2075-2100: Careful with variables from outer scope in nested function.The
get_template_filesfunction uses variables from the outer scope (search_query,filter_by,sort,direction) which creates tight coupling. If this function needs to be reused or refactored, these dependencies might cause issues.Consider passing these variables as parameters:
- def get_template_files(directory): + def get_template_files(directory, search_query, filter_by, sort, direction): """Get only template files without processing content""" cache_key = f"template_files_{directory}_{search_query}_{filter_by}" files = cache.get(cache_key)Then update the calls to this function:
- main_templates = get_template_files(main_template_dir) + main_templates = get_template_files(main_template_dir, search_query, filter_by, sort, direction) - subdir_templates = get_template_files(subdir_path) + subdir_templates = get_template_files(subdir_path, search_query, filter_by, sort, direction)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
website/views/core.py(3 hunks)
🧰 Additional context used
🧬 Code Definitions (1)
website/views/core.py (1)
website/models.py (1)
IP(887-901)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: Run Tests
- GitHub Check: docker-test
- GitHub Check: Analyze (python)
🔇 Additional comments (6)
website/views/core.py (6)
1987-1991: Great addition of docstring and concurrent processing!The added docstring improves the function's documentation, and importing ThreadPoolExecutor enables processing multiple templates in parallel, which should help with performance.
2002-2003: Handle potential non-integer 'page' query parameter.Casting the page directly with
int()can throw a ValueError if the query parameter is not a valid integer. The subsequenttry: ... except (PageNotAnInteger, EmptyPage): ...block won't catch that error, since it's raised before the paginator check.- page = int(request.GET.get("page", 1)) + try: + page = int(request.GET.get("page", 1)) + except ValueError: + page = 1
2005-2025: Well-implemented template metadata extraction with caching.Good implementation of template metadata extraction with proper caching to avoid repeated file reads. The cache duration of 24 hours is appropriate since template content rarely changes in production. The try-except block properly handles potential IOError when reading files.
2101-2150: Excellent optimization with concurrent processing.Using ThreadPoolExecutor to process templates concurrently is a great performance optimization. The caching strategy is well-implemented for both view counts and template URLs, reducing database queries and processing time.
Consider making the max_workers value configurable based on the server's resources:
- with ThreadPoolExecutor(max_workers=4) as executor: + # Use min(32, os.cpu_count() + 4) as recommended by Python docs for IO-bound tasks + max_workers = min(32, (os.cpu_count() or 2) + 4) + with ThreadPoolExecutor(max_workers=max_workers) as executor:
2169-2195: Good pagination implementation with on-demand processing.This is the key improvement that solves the performance issue mentioned in the PR. By creating a paginator with all templates but only processing the ones for the current page, the load time is significantly reduced.
There's duplicated code in the try-except block. Consider refactoring to avoid repetition:
# Create paginator with ALL templates (but don't process them yet) paginator = Paginator(all_templates, per_page) try: # Get the page object for the current page page_obj = paginator.page(page) except (PageNotAnInteger, EmptyPage): page_obj = paginator.page(1) # Now process only the templates for the current page processed_templates = [] for dir_name, template in page_obj.object_list: template_dir = {"name": dir_name, "templates": process_template_batch([template])} processed_templates.extend([(dir_name, t) for t in template_dir["templates"]]) # Replace the object_list with processed templates page_obj.object_list = processed_templates
2196-2216: Clean context preparation and rendering.The context preparation is well-structured and includes all necessary information for the template. Using a page object for pagination is the correct approach.
It resolves #3997
Problem: While loading the /template_list page, previously it was loading all the templates first and then rendering it in the frontend using paginator. So, the function was collecting data of all the templates including the templates that are not going to be displayed at first to the frontend. This was taking time.
Approach: I just optimized the function to collect data of those templates only which are going to be displayed on the frontend. Like if you are on the first page only the data of the first 20 templates will be collected, then on the second page only 20 after the first 20, and so on. This reduces the page loading time, previously it was collecting data of all 206 templates, now only 20 templates.
Summary by CodeRabbit
New Features
Refactor