Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@Alex-Kent
Copy link
Contributor

@Alex-Kent Alex-Kent commented Mar 18, 2025

Overview

This is a major update to the code. Most of the core logic has been completely rewritten. The main changes are:

  • Watches are stored to the project file and restored when the project is loaded.
  • Added status icons that are displayed in the layers tree for each watched layer.
  • Watches are now per file, not per layer.
    If a watch is added to a layer then all other layers in the project that are backed by the same file automatically have watches applied. A high-level description of the data structures and associated logic is given below.
  • Adding a layer to a project that has the same backing file as a watched file will automatically have a watch applied.
  • Changing a layer's data source to the same backing file as a watched file will automatically have a watch applied.
  • Streamlined logging by adding methods to log warnings and informational messages.
  • Refactored code to make it more readable. Methods and all instance variables (bar one) are named using camelCase, functions and local variables are named using snake_case. All variables holding QgsMapLayer objects are named layer and all those holding QgsLayerTreeNode objects (QgsLayerTreeGroup, QgsLayerTreeLayer, etc.) are named node (or variations on those names).

This code has been fairly well tested by manually checking various corner cases and no issues have been observed.

Persistence

All layers (QgsMapLayer objects) whose underlying files are being watched have the custom property reloader/watchLayer set to True. These custom properties are stored in the project files and are used at project load time to determine which layers to install file change watches on.

Data structures and core logic

This version of the code uses a single watcher per watched file. All watchers share a single callback function. On file change the watcher callback loops through the list of layers associated with the passed file path (via layerIDsForPath[path]) and triggers a reload/repaint for each layer in the list. The data structures used are:

watchers

watchers[path_string]watcher_object
Dictionary (hash/associative array) with file path strings as the keys and file change watcher objects as the values.
There is a separate watcher object in this dictionary for each watched file path, however all file watcher objects share a single callback function (see next section).

pathForLayerID

pathForLayerID[layer_id]path_string
Dictionary with layers tree layer ID strings as the keys and file path strings as the values.
Multiple layer IDs can map to the same file path.
The keys comprise the IDs of all of the layers tree layers that have watches applied.

layerIDsForPath

layerIDsForPath[path_string][ layer_id ( ( , layer_id ) … ) ]
Dictionary with file path strings as the keys and lists of layer tree layer ID strings as the values.
The keys comprise all of the file paths that are being watched.
There is a one-to-one mapping between the keys in layerIDsForPath and the keys in watchers.

Callbacks

The following callbacks are installed (in order of declaration):

  • fileChangedCallbackFunction (class variable)
    The callback singleton for handling all file changed signals.
    This is called by watchers when a watched file changes.
    All watchers share this one callback function.
    See getFileChangedCallbackFunction(), below.

  • legend_layers_added_callback(added_layers)
    (was layersAddedToLayerTree(added_layers))
    Called when one or more layers are added to the project's layer tree.
    Added layers that are backed by one of the watched files will have a watch added.
    This callback is applied once to the entire project.

  • rows_inserted_callback(parent, first, last)
    Called when layers are moved, grouped, etc.
    This is used to restore icons on the watched layers.
    This callback is applied once to the entire project.

  • reloadCallback()
    Reloads selected layer(s).
    Called when user selects "Reload selected layer(s)"

  • reopenCallback()
    Reopens selected layer(s) and also updates the layer's extent.
    This is in contrast to reloadCallback which keeps does not update the layer's extent.
    Called when user selects "Reopen selected layer(s)"

  • watchCallback()
    Starts watching selected layer(s) for changes.
    Called when user selects "Start watching layer(s) for changes".
    All layers backed by the same file as one of the selected layers will also be watched.

  • data_source_changed_callback(layer)
    via addDataSourceChangedCallback(self, layer)
    Called when the data set definition (e.g. its path) is changed for a given layer.
    Note that this is not the same as a change in the file pointed to by the data source definition.
    This callback is added to every layer tree layer that has an installed watcher.

  • file_changed_callback(path)
    via getFileChangedCallbackFunction()
    The latter method returns the file change callback singleton (stored in fileChangedCallbackFunction class variable). This method creates the callback singleton when needed.

    There is a single file change callback function for all watched files/layers. The callback function triggers reloads/redraws as appropriate based on the passed file path string and the layerIDsForPath structure described in the previous section.

    This callback is applied to every watcher object.

  • will_be_deleted_callback(layer)
    Called immediately before a layer tree layer object is deleted.
    This callback removes the layer's file change watch and "reloader/watchLayer" custom property.
    This callback is added to every layer tree layer that has an installed watcher.

  • unwatchCallback()
    Stops watching selected layer(s) for changes.
    Called when user selects "Stop watching layer(s) for changes"
    All layers backed by the same file as one of the selected layers will also cease to be watched.

…watches on project load.

Updated version number strings.
…ayer. Many additional improvements.

This version includes copious debug statements.
For ease of reference to 3rd-party documentation, the [currently unused] first_start variable was left untouched.
Renamed variables to match the objects type (node vs. layer).
…LayersAddedCallback and legend_layers_added_callback, respectively. (This matches the naming convention used in the rest of the code,)
@evetion
Copy link
Owner

evetion commented Mar 23, 2025

Thanks for the huge update, I hope to find some time later today to review it.

Watching files instead of layers and project support are very good improvements. ❤️

@Alex-Kent
Copy link
Contributor Author

Enjoy!

I've been running this for the past week with frequent layer updates on a large number of watched layers (34) and haven't run into any issues not fixed in the PR's commits.

Also, if you're using the Layer Color Plugin then the status icons won't show up with the current version of that plugin and of QGIS. PRs on both are pending that fix the issue (LayerColorPlugin and QGIS). For now if you're running that plugin you need to disable it and then restart QGIS before status icons will show up (or use my modified versions: LayerColorPlugin and QGIS; both are needed to fix the issue)

Copy link
Owner

@evetion evetion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A quick nitpick, the styling is a bit all over the place. Maybe that's something for me to clean up after this PR (changing it now might only make the review harder). I will probably do so with https://docs.astral.sh/ruff/, and just run it automatically on save in the editor.

Copy link
Owner

@evetion evetion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thanks for the effort, there are some major improvements here! 👍🏻 That said this is a rather large change (and might've been good to split it up into multiple PRs per issue/feature), and increases the complexity of the codebase.

Given that, it might be good to move some stuff to a separate file. I also see a lot of nested functions that don't (need to) capture variables, so they can be moved to outside the function (to either inside the class or to utils.py).

It seems the most complex addition (given the administration needed everywhere) is multi-layer-single-file watch. I'm not sure what to do about it, but I think it could in its own class/administration, so you don't need the logic (and separate dicts).

Some overall styling patterns that are not optimal, and I might change later:

  • Mixed camelCasing and snake_casing (the latter is preferred for all variables and functions, only classes have PascalCasing).
  • # END comments (introduced because of the nesting)
  • Many comments that only describe what, not why. These are unnecessary.

Comment on lines +303 to +304
# Our code will automatically add watches to added layers that are
# backed by local files that are already being watched.
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, might be good to document that though (that layers that were not manually watched will still refresh).

Comment on lines +309 to +311
# Attempt to reconnect all watchers set in the current project
if self.iface.activeLayer():
self.reconnectWatches()
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this required (and the same as the project load event?)

"""
if QgsLayerTree.isLayer(node):
layer = node.layer()
if hasattr(node, "customProperty"):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

customProperty always exists I believe. You probably want to check for the correct key with https://qgis.org/pyqgis/3.40/core/QgsMapLayer.html#qgis.core.QgsMapLayer.customPropertyKeys. Or get it directly with https://qgis.org/pyqgis/3.40/core/QgsMapLayer.html#qgis.core.QgsMapLayer.customProperty as you do below, but set a default value to False.

Does not remove the layers' reloader/watchLayer properties.
"""

def remove_node_status_icons(node):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This nesting does is not required (no capturing of variables), so it's better to place it outside.

Comment on lines +1020 to +1023
if layer.id() in self.pathForLayerID:
return True
else:
return False
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if layer.id() in self.pathForLayerID:
return True
else:
return False
return layer.id() in self.pathForLayerID

Comment on lines +536 to +540
# Sanity check
if watcher is None:
# Shouldn't happen
self.warning("Can't stop watching the removed layer because we never started watching it!")
else:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check is not needed, as we can still del None?

Comment on lines +612 to +614
# Determine which layers are currently selected
# layers is List[QgsMapLayer] (QgsMapLayer is e.g. QgsVectorLayer)
# *** FIXME - Add support for selectedLayersRecursive (since 3.4) ***
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be best to remove these repeated comments and just make an issue to use the recursive option in the future?

Comment on lines +660 to +666
if not 'path' in components:
# Layer's data source does not appear to be a local file
return None

# A "path" value is present, get its value
# (This is the name of the local data file containing the layer's data)
path = components['path']
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be simplified to components.get('path'). If not found, you'll get None back.

# Verify that the file containing the layer's data actually exists
if not isfile(path):
# Path doesn't specify an extant local file
def getLayerPath(self, layer):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be moved to utils.py.

Comment on lines +798 to +801
# Remove any no-longer-extant layer from watcher structures
self.unwatchLayer(layer, quiet=True)
self.updateStatusIcons()
return 0
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

# backed by local files that are already being watched.
project = QgsProject.instance()
if project is not None:
project.legendLayersAdded.connect( self.getLegendLayersAddedCallback() )
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +737 to +738
if path in self.layerIDsForPath:
for layer_id in self.layerIDsForPath[path]:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could simplify as for layer_id in self.layerIDsForPath.get(path, []):

@Alex-Kent
Copy link
Contributor Author

the styling is a bit all over the place

Can you be a bit more specific?

[ I know I added indentation to blank lines (the original code doesn't have this); this was done to make the structure clearer (I have space/tab highlighting turned on in my editor). ]

@Alex-Kent
Copy link
Contributor Author

First of all, thanks for the effort, there are some major improvements here!

De nada. A lot of it was just stuff that I wanted for my own use (saving watch state to the project, status icons) and much of the rest was covering corner cases (e.g. data source changed). Generally if I make changes that I reckon others might find useful I'll share them with the original project.

this is a rather large change (and might've been good to split it up into multiple PRs per issue/feature)

I originally had these split into different branches (see the commit history). I found that the code was getting very messy, hard to understand, and hard to keep in sync so I merged the three branches into one and refactored the code. [Refactoring each branch separately (and keeping the changes to each in sync) would have been tricky and error-prone.]

Given that, it might be good to move some stuff to a separate file. I also see a lot of nested functions that don't (need to) capture variables, so they can be moved to outside the function (to either inside the class or to utils.py).

Makes sense if it's feasible. One I think you called out specifically was the logging methods. These could be made standalone functions (no longer Reloader class methods) but IIRC would still need this to be be passed (the QGIS logging method needs it IIRC). Will comment more in response to your comment on that code.

It seems the most complex addition (given the administration needed everywhere) is multi-layer-single-file watch. I'm not sure what to do about it, but I think it could in its own class/administration, so you don't need the logic (and separate dicts).

Some of the rationale for this:

  • Clearly divide the roles of the various parts of code.
  • Code efficiency and code clarity: e.g. avoiding the need to check a layer's custom property when a simple hash/dict lookup would suffice.
  • Fewer systems calls by having one watcher per file, even if watched multiple times. Having duplicate watchers could in particular be an issue on Windows systems which are (in my experience) extremely inefficient when it comes to things like [their equivalent of] stating a file.
  • Avoid potential maintenance issues with keeping the various parts of the code in sync.

The dicts are also needed for e.g. the case where a watch is being removed due to the layer's data source having changed; in this instance there's no way to determine through the QGIS API what the former source was and thus, absent a mapping hash, no easy/efficient way to determine which file watch should be removed.

The watches could be moved into their own class but I think the complexity would be higher in that case.

Some overall styling patterns that are not optimal, and I might change later:

  • Mixed camelCasing and snake_casing (the latter is preferred for all variables and functions, only classes have PascalCasing).

As for changes to my PR, this would effect only the pathForLayerID, layerIDsForPath, and fileChangedCallbackFunction class instance variables (which would change to path_for_layer_id, layer_ids_for_path, and perhaps file_changed_callback_singleton), correct? All methods (everything with a self parameter) get camelCase and all functions (those without an explicit self parameter) get snake_case.

  • # END comments (introduced because of the nesting)

This is a stylistic thing on my part. Nesting is a large part of why they're present (and consistency; the end of all functions and methods are thus marked). Some of the nested code could be moved into stand-along methods but I generally favour keeping code that does a single task (and doesn't duplicate code elsewhere) encapsulated in single function or method, e.g. a function used only in a given method would be defined within that method [it's clearer that way IMO]. That said, there are cases where moving code into [top-level] subfunctions makes sense and can increase clarity. I saw you flagged some cases where code could be moved into a new top-level function/method; I'll comment more there.

  • Many comments that only describe what, not why. These are unnecessary.

I'll keep an eye out for this when I go through your code comments.

@evetion
Copy link
Owner

evetion commented Mar 24, 2025

Makes sense if it's feasible. One I think you called out specifically was the logging methods. These could be made standalone functions (no longer Reloader class methods) but IIRC would still need this to be be passed (the QGIS logging method needs it IIRC).

Yes, warning and info don't need self. warnAndLog could be moved, and take iface as a first argument (if moved out, it's best not to pass self, but pass the (most) specific object that's needed).

The dicts are also needed for e.g. the case where a watch is being removed due to the layer's data source having changed; in this instance there's no way to determine through the QGIS API what the former source was and thus, absent a mapping hash, no easy/efficient way to determine which file watch should be removed.

I agree you probably needs several dict like objects, but I meant that you have a separate class(es) for them, so you can isolate the logic, and ensure the dicts are in sync (and always set the custom property). This also makes it testable. You could define methods like is_watched(filepath) or is_watched(layer_id), which makes the code in the reloader class more readable.

I originally had these split into different branches (see the commit history). I found that the code was getting very messy, hard to understand, and hard to keep in sync so I merged the three branches into one and refactored the code. [Refactoring each branch separately (and keeping the changes to each in sync) would have been tricky and error-prone.]

I completely understand, and I don't want to make this process more difficult. Let's postpone the stylistic changes (easy to disagree on, and it makes merging/syncing with the fork painful, so I will just default to ruff after we got everything merged). Let me know if you want me to make some of these other suggested changes later on.

@Alex-Kent
Copy link
Contributor Author

Makes sense if it's feasible. One I think you called out specifically was the logging methods. These could be made standalone functions (no longer Reloader class methods) but IIRC would still need this to be be passed (the QGIS logging method needs it IIRC).

Yes, warning and info don't need self. warnAndLog could be moved, and take iface as a first argument (if moved out, it's best not to pass self, but pass the (most) specific object that's needed).

Or move them into a class that takes iface as an argument to its constructor. Then one wouldn't have to pass it at all in the rest of the code, e.g.:

log = LoggingClass(iface);
…
log.info(message);
log.warning(message);
log.warning(message, notify=True);  # né warnAndLog

Note that warnAndLog is only used once in the code (when the user requests a watch on something that isn't backed by a local file).

The dicts are also needed for e.g. the case where a watch is being removed due to the layer's data source having changed; in this instance there's no way to determine through the QGIS API what the former source was and thus, absent a mapping hash, no easy/efficient way to determine which file watch should be removed.

I agree you probably needs several dict like objects, but I meant that you have a separate class(es) for them, so you can isolate the logic, and ensure the dicts are in sync (and always set the custom property). This also makes it testable. You could define methods like is_watched(filepath) or is_watched(layer_id), which makes the code in the reloader class more readable.

I'll take a look and see what would be involved in splitting the watcher-related code into its own class (including the mapping hashes) with access strictly though methods.

I completely understand, and I don't want to make this process more difficult. Let's postpone the stylistic changes (easy to disagree on, and it makes merging/syncing with the fork painful, so I will just default to ruff after we got everything merged). Let me know if you want me to make some of these other suggested changes later on.

Sounds good to me. I'll leave all the stylistic stuff as-is for for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants