Improve handling of different path variants on Windows (elastic#3781) (…

…elastic#3786)

* Improve handling of different path variants on Windows

If forward slashes were used on Windows the glob was not matching on startup which lead to the issue that data was resent. This is solved in 5.3 by using path.Abs for all paths which also includes Clean. To make sure Path and Glob are always clean, Cleanup was added to the MatchFile part. This makes sure in case old data / incorrect data is loaded, the comparison will still work. More tests were added for windows to verify change.

Before this change, option 1 below did not work.

```
"F:/wwwLogs/flights-wsapi/WebServicesWebApi.log"
"F:\\wwwLogs\\flights-wsapi\\WebServicesWebApi.log"
'F:\wwwLogs\flights-wsapi\WebServicesWebApi.log'
```

Based on https://discuss.elastic.co/t/duplicate-events-with-filebeat-on-windows-on-service-restart/78743/10

* Move clean path outside for loop

(cherry picked from commit 4527162)

Minor edits

Update testing environment for 5.2.1 (elastic#3564)

Update testing dependencies in 5.2 to build a9f63967 (elastic#3448)

Fix bug when migrating old states, 2 restarts are required (elastic#3322

) (elastic#3326)

The state of migrated states was not properly updated in the registry file. This lead to the issue that after the first restart, the states were migrated but the prospector assumed the states were not finished and didn't start harvesting. A second restart resolved the problem.

Discussion started here: https://discuss.elastic.co/t/filebeat-upgrade-requiring-multiple-restarts/70414/8
(cherry picked from commit fb47507)

remove changelog entry (change was included in 5.1.1) (elastic#3123)

Backport: doc changes in elastic#2953, elastic#3002, elastic#3010, el…

…astic#3036, elastic#3041 (elastic#3043)

* Add consulbeat (elastic#2953)

* Add redditbeat to community beats list

* Improve "pipelines" documentation

Mention the similarity to the "indices" array and add an example.

* Default docs update for filbeat

* Default docs for close_eof and close_timeout added
* Remove note for logstash-forward based code
* Updated output docs as elasticsearch can now also be used for parsing (ingest)

* Show download link to dashboards

Docs: Fix path for beats-dir

Backport doc changes into 5.0 (elastic#2847)

* Clarify instructions for running configtest

* Add to FAQ the solution for having nested JSON objects under Discovery (elastic#2726)

* docs: Fix broken Kibana doc link in faq (elastic#2837)

The Kibana docs overhaul for 5.0+ changed the link to the index pattern
field refresh instructions.

* Use variable for filepath plus some light edits (elastic#2838)

Merge pull request elastic#2724 from tsg/backport_2631

Backport 2631: Breaking changes guide + fixes