Filebeat in memory states registry improvements #6346

urso · 2018-02-09T15:49:33Z

make States type easier to find by moving into separate states.go file
Ensure provides States constructor is actually used
Add ID->array index, index for faster lookups on update and find
operations. When updating states in a big registry, the Updates
converged to quadratic complexity. The index helps in keeping the
complexity about linear in number of state updates.
Debug will print number of states subject to future cleanups (if state TTL > 0)
Add title to states unit tests

- make States type easier to find by moving into separate states.go file - Ensure provides States constructor is actually used - Add ID->array index, index for faster lookups on update and find operations. When updating states in a big registry, the Updates converged to quadratic complexity. The index helps in keeping the complexity about linear in number of state updates. - Debug will print number of states subject to future cleanups (if state TTL > 0) - Add title to states unit tests

houndci-bot · 2018-02-09T15:49:39Z

filebeat/input/file/states.go

+	return len(s.states)
+}
+
+// Returns a copy of the file states


comment on exported method States.GetStates should be of the form "GetStates ..."

houndci-bot · 2018-02-09T15:49:39Z

filebeat/input/file/states.go

+	}
+}
+
+func (s *States) FindPrevious(newState State) State {


exported method States.FindPrevious should have comment or be unexported

houndci-bot · 2018-02-09T15:49:40Z

filebeat/input/file/states.go

+	return len(s.states)
+}
+
+// Returns a copy of the file states


comment on exported method States.GetStates should be of the form "GetStates ..."

houndci-bot · 2018-02-09T15:49:40Z

filebeat/input/file/states.go

+	}
+}
+
+func (s *States) FindPrevious(newState State) State {


exported method States.FindPrevious should have comment or be unexported

houndci-bot · 2018-02-09T15:49:40Z

filebeat/input/file/states.go

+	return len(s.states)
+}
+
+// Returns a copy of the file states


comment on exported method States.GetStates should be of the form "GetStates ..."

houndci-bot · 2018-02-09T15:49:40Z

filebeat/input/file/states.go

+	}
+}
+
+func (s *States) FindPrevious(newState State) State {


exported method States.FindPrevious should have comment or be unexported

urso · 2018-02-09T15:51:09Z

In-memory state registry optimizations only (original ticket: #6323).

houndci-bot · 2018-02-09T19:24:53Z

filebeat/input/file/states.go

+// The number of states that were cleaned up and number of states that can be
+// cleaned up in the future is returned.
+
+func (s *States) Cleanup() (int, int) {


exported method States.Cleanup should have comment or be unexported

ph · 2018-02-09T21:18:45Z

One test is still failing for filebeat? ERROR: Checks that states are properly removed after clean_inactive

ph

Code look good one test is failling.

ph · 2018-02-09T21:21:35Z

filebeat/input/file/states.go

+	return s.states[i]
+}
+
+// findPrevious returns the previous state fo the file


ph · 2018-02-09T21:22:17Z

filebeat/input/file/states.go

+// Cleanup cleans up the state array. All states which are older then `older` are removed
+// The number of states that were cleaned up and number of states that can be
+// cleaned up in the future is returned.
+


ph · 2018-02-09T21:31:29Z

filebeat/input/file/states.go

+			// remove entry by relocating last entry + shrink states array
+			last := len(s.states) - 1
+			s.states[i] = s.states[last]
+			s.states = s.states[:last]


thinking out loud here, related to the case of a lot of files.
Could this generate a lot of eviction/resizing in the real world? I guess we are just moving pointers.

You mean CPU cache? No idea without testing/profiling. But we basically narrow down the array from beginning and end at the same time.

Also depends on how many entries need to be removed.

ruflin

Definitively huge +1 on using an index for the lookup, be it as part of the state map string or separate index. Left some minor questions.

ruflin · 2018-02-11T23:37:58Z

filebeat/input/file/states.go

+	states []State
+
+	// idx maps state IDs to state indexes for fast lookup and modifications.
+	idx map[string]int


What is the reason you changed your mind to the separate map string for keeping track of the ids?

Too many copies :)

The states mapping has these very frequent access patterns:

access single entry by ID

loop/copy all states

Performance wise, both access patterns are equally important, as they are run very very often (basically per event).

Optimizing the access by ID case (using map[string]state) I found the profiling being dominated by extra copies, even if only one field is required. Introducing an index on array indexes, we still have the well working 'all-access loop' + we optimize single entry access to be about O(1) complexity.

Kind of best of both worlds with a little constant overhead when actually removing entries from the registry. As state update/add + iteration + copy are the most often used operations, the added overhead on removal is pretty much worth it.

ruflin · 2018-02-11T23:39:45Z

filebeat/input/file/states.go

+	statesBefore := len(s.states)
+	numCanExpire := 0
+
+	L := len(s.states)


upper case on purpose?

yep. L for "length" :)

ruflin · 2018-02-11T23:41:17Z

filebeat/registrar/registrar.go

 	logp.Debug("registrar",
-		"Registrar states cleaned up. Before: %d, After: %d",
-		beforeCount, beforeCount-cleanedStates)
+		"Registrar states cleaned up. Before: %d, After: %d, Pending: %d",


I already see questions coming on what pending cleanup means :-)

We can call it 'active'. The number of 'pending' states is all states with TTL > 0.

Following the state changes and TTLs, I found TTL basically has 4 meanings:

-2: zombie state (no prospector/input processing the file)

-1: alive (prospector/input owning the file, but no TTL)

0: done (entry is marked as removable)

>0: subject to timeout

Any better naming?

Currently The 'pending' count is the number of states potentially subject to being timed out. But states -1 and >0 indicate the state to be 'active'.

ruflin · 2018-02-11T23:44:00Z

filebeat/input/file/states.go

+			}
+		} else {
+			i++
+			if canExpire {


What is the reason you added canExpire? If there is clean_inactive configured it applies to all states.

See my TTL comment. The canExpire marks states with TTL > 0 -> states subject to timeout. If Cleanup returns and number of events with TTL>0 is 0, no cleanup needs to be run until a new state with TTL>=0 is added.

I also make use of this in my followup PR (#6347) by deferring/suspending states Cleanup in the registrar.

ruflin · 2018-02-11T23:44:49Z

filebeat/input/file/states.go

+			delete(s.idx, state.ID())
+			logp.Debug("state", "State removed for %v because of older: %v", state.Source, state.TTL)
+
+			L--


Looking at the code with i++ and L-- I'm thinking if the option with combining index and registry would actually bring less complexity?

Maybe it's not that clear as some extra work is done within the loop, but the loop basically sorts the list into alive and dead states. The L is the current length. Removal/sorting is in-place. That is, we're forced to check the current index again upon removal.
The i++ and L-- act as delimiters of processed and alive/dead states => both need to be updated, depending if state is alive or not. Very basic in-place removal loop without extra copies.

would this be more clear for you:

func (s *States) Cleanup() (int, int) { s.Lock() defer s.Unlock() L := len(s.states) currentTime := time.Now() statesBefore := L numCanExpire := 0 // number of alive states with TTL > 0 for i := 0; i < L; { state := &s.states[i] canExpire := state.TTL > 0 expired := (canExpire && currentTime.Sub(state.Timestamp) > state.TTL) if (state.TTL == 0 || expired) && state.Finished { // remove by replacing entry at i with last entry and shrink 'alive' states length L-- if L != i { // swap and update index s.states[i], s.states[L] = s.states[L], s.states[i] s.idx[s.states[i].ID()] = i } } else { i++ if canExpire { numCanExpire++ } } } alive := s.states[:L] dead := s.states[L:] for i := range dead { delete(s.idx, dead[i].ID()) } s.states = alive return statesBefore - len(s.states), numCanExpire }

When I've looked at the code initially I had to use pseudo code here to make sure the in place was working correctly.

I prefer the first version.

urso · 2018-02-12T18:10:58Z

Failing ci test is on auditbeat. Unrelated to this PR.

ph

LGTM

ph · 2018-02-13T20:33:17Z

@urso for me I think changes are well worth it, lets have the green light from @ruflin too.

ruflin · 2018-02-15T10:14:29Z

@ph Please go forward and merge if Jenkins failure is not related. The most important part is the functionality and there we agree. We can still do minor refactoring later.

- Defer registrar state gc until the registry needs to be written. - when updating regitrar states from batch, ensure all updated states will have same timestamp. Depends on #6346

* Filebeat in memory states registry improvements - make States type easier to find by moving into separate states.go file - Ensure provides States constructor is actually used - Add ID->array index, index for faster lookups on update and find operations. When updating states in a big registry, the Updates converged to quadratic complexity. The index helps in keeping the complexity about linear in number of state updates. - Debug will print number of states subject to future cleanups (if state TTL > 0) - Add title to states unit tests * Fix godocs * update state index in Cleanup * Refine index handling + reintroduce debug message * review (cherry picked from commit 490cbcd)

- Defer registrar state gc until the registry needs to be written. - when updating regitrar states from batch, ensure all updated states will have same timestamp. Depends on elastic#6346 (cherry picked from commit 2d2400b)

- Defer registrar state gc until the registry needs to be written. - when updating regitrar states from batch, ensure all updated states will have same timestamp. Depends on #6346 (cherry picked from commit 2d2400b)

Backport changes from elastic#6346 and elastic#6347, to improve state update performance on large registry files. - make States type easier to find by moving into separate states.go file - Add ID->array index, index for faster lookups on update and find operations. When updating states in a big registry, the Updates converged to quadratic complexity. The index helps in keeping the complexity about linear in number of state updates. - Do not gc state if there are no 'pending' entries

…ic#6804) * Filebeat in memory states registry improvements - make States type easier to find by moving into separate states.go file - Ensure provides States constructor is actually used - Add ID->array index, index for faster lookups on update and find operations. When updating states in a big registry, the Updates converged to quadratic complexity. The index helps in keeping the complexity about linear in number of state updates. - Debug will print number of states subject to future cleanups (if state TTL > 0) - Add title to states unit tests * Fix godocs * update state index in Cleanup * Refine index handling + reintroduce debug message * review (cherry picked from commit 042b284)

- Defer registrar state gc until the registry needs to be written. - when updating regitrar states from batch, ensure all updated states will have same timestamp. Depends on elastic#6346 (cherry picked from commit 0ae953c)

houndci-bot reviewed Feb 9, 2018

View reviewed changes

urso requested review from ph, ruflin and tsg February 9, 2018 15:49

Fix godocs

8b928cd

This was referenced Feb 9, 2018

Defer registry state cleanup #6347

Merged

Filebeat registrar update optimizations #6323

Closed

update state index in Cleanup

9583063

houndci-bot reviewed Feb 9, 2018

View reviewed changes

ph reviewed Feb 9, 2018

View reviewed changes

urso added 2 commits February 10, 2018 00:33

Refine index handling + reintroduce debug message

d6c97fd

review

4f787a7

ruflin reviewed Feb 11, 2018

View reviewed changes

ruflin added review Filebeat Filebeat labels Feb 11, 2018

ph approved these changes Feb 13, 2018

View reviewed changes

kvch merged commit 490cbcd into elastic:master Feb 15, 2018

urso deleted the opt/filebeat-state-registry branch April 9, 2018 14:35

urso mentioned this pull request Apr 9, 2018

Cherry-pick #6346 to 6.2: Filebeat in memory states registry improvements #6804

Merged

urso mentioned this pull request Apr 9, 2018

Cherry-pick #6347 to 6.2: Defer registry state cleanup #6812

Merged

urso mentioned this pull request May 28, 2018

Backport filebeat registry state update improvements #7188

Closed

Filebeat in memory states registry improvements #6346

Filebeat in memory states registry improvements #6346

Uh oh!

Conversation

urso commented Feb 9, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

urso commented Feb 9, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ph commented Feb 9, 2018

Uh oh!

ph left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

urso commented Feb 12, 2018

Uh oh!

ph left a comment

Choose a reason for hiding this comment

Uh oh!

ph commented Feb 13, 2018

Uh oh!

ruflin commented Feb 15, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants