Update: Added support for Filetree sharing between directory resolver - this feature is needed so that catalogers running concurrently can work on the same references.

Following issues are targeted

• Share Filetree between resolver via source struct.
• Thread save directory resolver indexing - Adding a lock around the indexing logic
• Share indexed file information - copied shared tree references before indexing.
• Fix file metadata GID,UID sample (Linux based only).
• indexAllRoots skip duplication root indexing.

Test

• Added shared directory unit test - TestDirectoryResolver_SharedTreeMultipleFilesByPath

@houdini91 very nice addition!

I see the performance reason for introducing the shared trees between directoryResolver instances, and it's a good one. However, I think there is an opportunity to simplify that approach while still keeping the same performance. Instead of creating multiple instances of the directoryResolver and injecting a shared file tree as a dependency, it could be simpler to memoize the Source. FileResolver() method such that only one directoryResolver instance is created and cached for future calls. Additional benefit is that the FileTree remains an internal concern of the directoryResolver and there would no longer be a need to keep a mutex on Source.

What do you think about this as an alternative approach to the shared filetree?

@wagoodman I totally agree, sharing a directoryResolver is probably simpler than sharing a FileTree pointer.
Originally thought about sharing the directoryResolver pointer but i was not sure how it will effect parts i have not yet explored.

Can you please clarify a bit on the Mutex comment?
It seems to me that even if source.FileResolver() creates only one directoryResolver (Stores pointer in struct) the initialization process still can create a race (Two threads initialization the first directoryResolver)
Eg:

// Called by catalogers child threads concurrently 
func (s Source) FileResolver(scope Scope) (FileResolver, error) {
	switch s.Metadata.Scheme {
	case DirectoryScheme:
                if s.dirResolver == nil {
                          // Not thread safe
		          resolver, err := newDirectoryResolver(s.Tree, s.Metadata.Path)
                          if err == nil {
                                 s.dirResolver = resolver
                          }
                }
                return s.dirResolver, err
...
...

• Maybe we can initialize the directory resolver in the source New() constructor function? (Called by main thread)?
• Maybe leave the Mutex and only replace the FileTree pointer with a directoryResolver pointer?

Maybe we can initialize the directory resolver in the source New() constructor function? (Called by main thread)?

This wouldn't be ideal since any FileResolver calls may be passed different scopes. I realize that for the case of the directory resolve this doesn't technically matter (since a directory resolver is like having an image with a single layer, so different scope options have no effect), but doing resolver initialization in two places where it could be focused to one is where the concern is.

Maybe leave the Mutex and only replace the FileTree pointer with a directoryResolver pointer?

You're right, I overlooked this --the mutex for the resolver technically should already be there today. Let's leave the mutex 👍

@wagoodman

it could be simpler to memoize the Source. FileResolver() method such that only one directoryResolver instance

Updated code to reflect change.

• One issue i had was that the source struct was not passed around by value causing any pointer initialization not be reflected outside of the instance scope.
  I have updated code to use a pointer of the source struct instead.

@houdini91 this looks good, thanks for the updates! Only one last change to get past validations: https://github.com/anchore/syft/pull/467/checks?check_run_id=3508971289 a linter failure for a variable name. Once passing we'll merge this in.

@wagoodman Fixed lint error,
your welcome It was my pleasure.