-
Notifications
You must be signed in to change notification settings - Fork 2
Major update – Watches saved to project file, adds status icons, improved multi-file/multi-layer handling, etc. #11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…watches on project load. Updated version number strings.
…ayer. Many additional improvements. This version includes copious debug statements.
…he logging. No functional changes to code.
For ease of reference to 3rd-party documentation, the [currently unused] first_start variable was left untouched.
Renamed variables to match the objects type (node vs. layer).
…LayersAddedCallback and legend_layers_added_callback, respectively. (This matches the naming convention used in the rest of the code,)
|
Thanks for the huge update, I hope to find some time later today to review it. Watching files instead of layers and project support are very good improvements. ❤️ |
|
Enjoy! I've been running this for the past week with frequent layer updates on a large number of watched layers (34) and haven't run into any issues not fixed in the PR's commits. Also, if you're using the Layer Color Plugin then the status icons won't show up with the current version of that plugin and of QGIS. PRs on both are pending that fix the issue (LayerColorPlugin and QGIS). For now if you're running that plugin you need to disable it and then restart QGIS before status icons will show up (or use my modified versions: LayerColorPlugin and QGIS; both are needed to fix the issue) |
evetion
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A quick nitpick, the styling is a bit all over the place. Maybe that's something for me to clean up after this PR (changing it now might only make the review harder). I will probably do so with https://docs.astral.sh/ruff/, and just run it automatically on save in the editor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First of all, thanks for the effort, there are some major improvements here! 👍🏻 That said this is a rather large change (and might've been good to split it up into multiple PRs per issue/feature), and increases the complexity of the codebase.
Given that, it might be good to move some stuff to a separate file. I also see a lot of nested functions that don't (need to) capture variables, so they can be moved to outside the function (to either inside the class or to utils.py).
It seems the most complex addition (given the administration needed everywhere) is multi-layer-single-file watch. I'm not sure what to do about it, but I think it could in its own class/administration, so you don't need the logic (and separate dicts).
Some overall styling patterns that are not optimal, and I might change later:
- Mixed camelCasing and snake_casing (the latter is preferred for all variables and functions, only classes have PascalCasing).
- # END comments (introduced because of the nesting)
- Many comments that only describe what, not why. These are unnecessary.
| # Our code will automatically add watches to added layers that are | ||
| # backed by local files that are already being watched. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, might be good to document that though (that layers that were not manually watched will still refresh).
| # Attempt to reconnect all watchers set in the current project | ||
| if self.iface.activeLayer(): | ||
| self.reconnectWatches() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this required (and the same as the project load event?)
| """ | ||
| if QgsLayerTree.isLayer(node): | ||
| layer = node.layer() | ||
| if hasattr(node, "customProperty"): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
customProperty always exists I believe. You probably want to check for the correct key with https://qgis.org/pyqgis/3.40/core/QgsMapLayer.html#qgis.core.QgsMapLayer.customPropertyKeys. Or get it directly with https://qgis.org/pyqgis/3.40/core/QgsMapLayer.html#qgis.core.QgsMapLayer.customProperty as you do below, but set a default value to False.
| Does not remove the layers' reloader/watchLayer properties. | ||
| """ | ||
|
|
||
| def remove_node_status_icons(node): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This nesting does is not required (no capturing of variables), so it's better to place it outside.
| if layer.id() in self.pathForLayerID: | ||
| return True | ||
| else: | ||
| return False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| if layer.id() in self.pathForLayerID: | |
| return True | |
| else: | |
| return False | |
| return layer.id() in self.pathForLayerID |
| # Sanity check | ||
| if watcher is None: | ||
| # Shouldn't happen | ||
| self.warning("Can't stop watching the removed layer because we never started watching it!") | ||
| else: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This check is not needed, as we can still del None?
| # Determine which layers are currently selected | ||
| # layers is List[QgsMapLayer] (QgsMapLayer is e.g. QgsVectorLayer) | ||
| # *** FIXME - Add support for selectedLayersRecursive (since 3.4) *** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be best to remove these repeated comments and just make an issue to use the recursive option in the future?
| if not 'path' in components: | ||
| # Layer's data source does not appear to be a local file | ||
| return None | ||
|
|
||
| # A "path" value is present, get its value | ||
| # (This is the name of the local data file containing the layer's data) | ||
| path = components['path'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can be simplified to components.get('path'). If not found, you'll get None back.
| # Verify that the file containing the layer's data actually exists | ||
| if not isfile(path): | ||
| # Path doesn't specify an extant local file | ||
| def getLayerPath(self, layer): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be moved to utils.py.
| # Remove any no-longer-extant layer from watcher structures | ||
| self.unwatchLayer(layer, quiet=True) | ||
| self.updateStatusIcons() | ||
| return 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this needed?
| # backed by local files that are already being watched. | ||
| project = QgsProject.instance() | ||
| if project is not None: | ||
| project.legendLayersAdded.connect( self.getLegendLayersAddedCallback() ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| if path in self.layerIDsForPath: | ||
| for layer_id in self.layerIDsForPath[path]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could simplify as for layer_id in self.layerIDsForPath.get(path, []):
Can you be a bit more specific? [ I know I added indentation to blank lines (the original code doesn't have this); this was done to make the structure clearer (I have space/tab highlighting turned on in my editor). ] |
De nada. A lot of it was just stuff that I wanted for my own use (saving watch state to the project, status icons) and much of the rest was covering corner cases (e.g. data source changed). Generally if I make changes that I reckon others might find useful I'll share them with the original project.
I originally had these split into different branches (see the commit history). I found that the code was getting very messy, hard to understand, and hard to keep in sync so I merged the three branches into one and refactored the code. [Refactoring each branch separately (and keeping the changes to each in sync) would have been tricky and error-prone.]
Makes sense if it's feasible. One I think you called out specifically was the logging methods. These could be made standalone functions (no longer
Some of the rationale for this:
The dicts are also needed for e.g. the case where a watch is being removed due to the layer's data source having changed; in this instance there's no way to determine through the QGIS API what the former source was and thus, absent a mapping hash, no easy/efficient way to determine which file watch should be removed. The watches could be moved into their own class but I think the complexity would be higher in that case.
As for changes to my PR, this would effect only the
This is a stylistic thing on my part. Nesting is a large part of why they're present (and consistency; the end of all functions and methods are thus marked). Some of the nested code could be moved into stand-along methods but I generally favour keeping code that does a single task (and doesn't duplicate code elsewhere) encapsulated in single function or method, e.g. a function used only in a given method would be defined within that method [it's clearer that way IMO]. That said, there are cases where moving code into [top-level] subfunctions makes sense and can increase clarity. I saw you flagged some cases where code could be moved into a new top-level function/method; I'll comment more there.
I'll keep an eye out for this when I go through your code comments. |
Yes, warning and info don't need self. warnAndLog could be moved, and take
I agree you probably needs several dict like objects, but I meant that you have a separate class(es) for them, so you can isolate the logic, and ensure the dicts are in sync (and always set the custom property). This also makes it testable. You could define methods like
I completely understand, and I don't want to make this process more difficult. Let's postpone the stylistic changes (easy to disagree on, and it makes merging/syncing with the fork painful, so I will just default to ruff after we got everything merged). Let me know if you want me to make some of these other suggested changes later on. |
Or move them into a class that takes log = LoggingClass(iface);
…
log.info(message);
log.warning(message);
log.warning(message, notify=True); # né warnAndLogNote that
I'll take a look and see what would be involved in splitting the watcher-related code into its own class (including the mapping hashes) with access strictly though methods.
Sounds good to me. I'll leave all the stylistic stuff as-is for for now. |
Overview
This is a major update to the code. Most of the core logic has been completely rewritten. The main changes are:
If a watch is added to a layer then all other layers in the project that are backed by the same file automatically have watches applied. A high-level description of the data structures and associated logic is given below.
camelCase, functions and local variables are named usingsnake_case. All variables holdingQgsMapLayerobjects are namedlayerand all those holdingQgsLayerTreeNodeobjects (QgsLayerTreeGroup,QgsLayerTreeLayer, etc.) are namednode(or variations on those names).This code has been fairly well tested by manually checking various corner cases and no issues have been observed.
Persistence
All layers (
QgsMapLayerobjects) whose underlying files are being watched have the custom propertyreloader/watchLayerset toTrue. These custom properties are stored in the project files and are used at project load time to determine which layers to install file change watches on.Data structures and core logic
This version of the code uses a single watcher per watched file. All watchers share a single callback function. On file change the watcher callback loops through the list of layers associated with the passed file path (via
layerIDsForPath[path]) and triggers a reload/repaint for each layer in the list. The data structures used are:watcherswatchers[path_string]→ watcher_objectDictionary (hash/associative array) with file path strings as the keys and file change watcher objects as the values.
There is a separate watcher object in this dictionary for each watched file path, however all file watcher objects share a single callback function (see next section).
pathForLayerIDpathForLayerID[layer_id]→ path_stringDictionary with layers tree layer ID strings as the keys and file path strings as the values.
Multiple layer IDs can map to the same file path.
The keys comprise the IDs of all of the layers tree layers that have watches applied.
layerIDsForPathlayerIDsForPath[path_string]→[layer_id ( (,layer_id ) … )]Dictionary with file path strings as the keys and lists of layer tree layer ID strings as the values.
The keys comprise all of the file paths that are being watched.
There is a one-to-one mapping between the keys in
layerIDsForPathand the keys inwatchers.Callbacks
The following callbacks are installed (in order of declaration):
fileChangedCallbackFunction(class variable)The callback singleton for handling all file changed signals.
This is called by watchers when a watched file changes.
All watchers share this one callback function.
See
getFileChangedCallbackFunction(), below.legend_layers_added_callback(added_layers)(was
layersAddedToLayerTree(added_layers))Called when one or more layers are added to the project's layer tree.
Added layers that are backed by one of the watched files will have a watch added.
This callback is applied once to the entire project.
rows_inserted_callback(parent, first, last)Called when layers are moved, grouped, etc.
This is used to restore icons on the watched layers.
This callback is applied once to the entire project.
reloadCallback()Reloads selected layer(s).
Called when user selects "Reload selected layer(s)"
reopenCallback()Reopens selected layer(s) and also updates the layer's extent.
This is in contrast to
reloadCallbackwhich keeps does not update the layer's extent.Called when user selects "Reopen selected layer(s)"
watchCallback()Starts watching selected layer(s) for changes.
Called when user selects "Start watching layer(s) for changes".
All layers backed by the same file as one of the selected layers will also be watched.
data_source_changed_callback(layer)via
addDataSourceChangedCallback(self, layer)Called when the data set definition (e.g. its path) is changed for a given layer.
Note that this is not the same as a change in the file pointed to by the data source definition.
This callback is added to every layer tree layer that has an installed watcher.
file_changed_callback(path)via
getFileChangedCallbackFunction()The latter method returns the file change callback singleton (stored in
fileChangedCallbackFunctionclass variable). This method creates the callback singleton when needed.There is a single file change callback function for all watched files/layers. The callback function triggers reloads/redraws as appropriate based on the passed file path string and the
layerIDsForPathstructure described in the previous section.This callback is applied to every watcher object.
will_be_deleted_callback(layer)Called immediately before a layer tree layer object is deleted.
This callback removes the layer's file change watch and "reloader/watchLayer" custom property.
This callback is added to every layer tree layer that has an installed watcher.
unwatchCallback()Stops watching selected layer(s) for changes.
Called when user selects "Stop watching layer(s) for changes"
All layers backed by the same file as one of the selected layers will also cease to be watched.