This repository was archived by the owner on Jan 3, 2023. It is now read-only.
On-demand namespace fetch support #2045
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
Please consider multiple dirs in fetch list and what if there is dependency among these dirs. |
PHILO-HE
reviewed
Jan 28, 2019
smart-common/src/main/java/org/smartdata/conf/SmartConfKeys.java
Outdated
Show resolved
Hide resolved
| Collection<String> fetchDirs = | ||
| conf.getTrimmedStringCollection(SmartConfKeys.SMART_NAMESPACE_FETCHER_DIRS_KEY); | ||
| fetchList = new ArrayList<>(fetchDirs.size()); | ||
| for (String dir : fetchDirs) { |
Member
There was a problem hiding this comment.
Can we access the item in fetchDirs by for each?
Member
|
Please update the doc accordingly with the explanation for "ignore" property in deployment guide. |
Member
Member
|
Inotify issue for the ignored or fetched dirs. |
PHILO-HE
reviewed
Jan 29, 2019
smart-common/src/main/java/org/smartdata/conf/SmartConfKeys.java
Outdated
Show resolved
Hide resolved
| deque.add(ROOT); | ||
| } else { | ||
| for (String dir : fetchList) { | ||
| for (String dir : fetchDirs) { |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| pendingParent = parent; | ||
| } | ||
| } else { | ||
| LOG.warn("Configured directory does not exist"); |
Member
There was a problem hiding this comment.
Should not tackle unexisted dir in this section..
It seems that startAfter == null doesn't mean that the configured directory does not exist.
Member
|
Please cover some cases in unit test. |
8e9e69e to
0c78a2f
Compare
PHILO-HE
reviewed
Feb 19, 2019
...upport/smart-hadoop/src/main/java/org/smartdata/hdfs/metric/fetcher/InotifyEventApplier.java
Show resolved
Hide resolved
9cc232d to
496b328
Compare
43af9e4 to
30f4815
Compare
5950a84 to
1908815
Compare
18e7b78 to
af5d1aa
Compare
PHILO-HE
reviewed
Mar 8, 2019
...p-support/smart-hadoop/src/main/java/org/smartdata/hdfs/metric/fetcher/NamespaceFetcher.java
Outdated
Show resolved
Hide resolved
| HdfsFetchTask.init(); | ||
| } | ||
|
|
||
| public void startFetch(String dir) { |
PHILO-HE
reviewed
Mar 8, 2019
PHILO-HE
reviewed
Mar 18, 2019
| * Namespace, access info and other info only related to files under these dirs will be tackled. | ||
| * For more than one directories, they should be separated by ",". | ||
| */ | ||
| public static final String SMART_NAMESPACE_FETCHER_DIRS_KEY = "smart.fetch.dirs"; |
| public void apply(List<Event> events) throws IOException, MetaStoreException { | ||
| private void initialize(){ | ||
| SmartConf conf = new SmartConf(); | ||
| Collection<String> ignoreDirs = conf.getTrimmedStringCollection( |
Member
There was a problem hiding this comment.
move this piece of code to SmartConf
ece8432 to
058aea0
Compare
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.