Add docs about configuring LS for use with Filebeat modules #6832

dedemorton · 2017-03-22T23:29:02Z

@suyograo Here's what I'm planning to say about Filebeat modules in the 5.3 Logstash docs. I have the LS configs commented out, but will add them after you merge your PR. Let me know if the messaging here is in line with what you expect for 5.3.

IMO the manual configuration experience would be a lot easier if Filebeat allowed Logstash as an output. Instead of generating an error and closing, Filebeat should generate a warning when Logstash is configured as the output. Of course, if -setup is specified and ES is not configured as the output, Filebeat should generate an error.

Right now, FBM users who want to use Logstash instead of ingest node miss out on the benefit that FBMs provide for setting up the Filebeat config automatically. The only real benefit of using Filebeat modules at all for Logstash users (AFAIK) is that they can run the -setup command to get the dashboards.

Note that I've added some questions to reviewers within the text of the document.

The Logstash configs that will be included in this doc are in this PR: #6791

suyograo · 2017-03-23T01:22:45Z

docs/static/filebeat-modules.asciidoc

+contain ingest node pipelines, Elasticsearch templates, Filebeat prospector
+configurations, and Kibana dashboards.
+
+Filebeat modules do not currently provide Logstash pipeline configurations.


Can we add a sub-section called "Migrating from ingest pipelines". In this section, we can add context on why a user would want to graduate to using LS instead of the ingest pipelines. The following reasons are all valid:

Use multiple outputs. Ingest was designed to only support ES as an output. For example, users may want to archive their incoming data to S3 as well as indexing it in ES.

Use the persistent queue feature to handle spikes when ingesting data (from beats and other sources).

Take advantage of the richer transformation capabilities in Logstash.

+1, I actually prefer the term "Graduating" in this context.

For the last bullet, we can extend to say "like external lookups."

suyograo · 2017-03-23T01:23:01Z

docs/static/filebeat-modules.asciidoc

+build Logstash pipeline configurations that are equivalent to the ingest
+node pipelines available with the Filebeat modules.
+
+Then you'll be able to use the sample Kibana dashboards available with Filebeat


How about changing sample Kibana -> same Kibana?

By providing the LS configs, we are telling users they can continue to use the dashboards set up by the filebeat modules

+1, the point is its a "seamless" transition.

suyograo · 2017-03-23T01:58:44Z

@dedemorton @acchen97 I'm a bit confused on how to message this to our users.

IMO, if someone is moving from FBM -> ES to FBM -> LS -> ES, I would consider this as an advanced user. I think at this point, we should assume that modules are already setup, dashboards are already available in ES, but the user now wants to use LS as the transform engine instead of ingest. If we think about it this way, we could just ask the user to start ./filebeat with LS output and provide documentation on how to use LS config equivalents of ingest.

suyograo · 2017-03-23T01:26:00Z

docs/static/filebeat-modules.asciidoc

+Filebeat needs to create the index pattern and load the sample dashboards into the
+Kibana index. 
+
+After the dashboards are loaded, you'll see the message


Hmm this error is weird.. Does this happen only with LS output enabled?

Can we maybe have Logstash just define this index mapping and pattern on LS startup?

@suyograo You can't use the -setup step with logstash configured as output because the dashboards need to be loaded into an Elasticsearch instance.

My mistake. This error happens only when you have Elasticsearch and Logstash configured as outputs. I'll removed that sentence from the topic.

suyograo · 2017-03-23T01:28:10Z

docs/static/filebeat-modules.asciidoc

+
+. Configure Filebeat to send log lines to Logstash.
+
+In version 5.3, Filebeat modules won't work when Logstash is configured as


I think this line is confusing. We should just remove it.

I guess from this point on -- once you've run setup from FBM -- you are using FB as a regular shipper to LS. We should say, "remove the ES output from filebeat.yml and add LS as an output".

output.logstash: hosts: ["localhost:5044"]

That should just work then.

See earlier comment. The problem here is the configuration options get set dynamically when you run the Filebeat modules (I don't think the config for the module is persisted anywhere). So you can just run a module once to get Filebeat configured. If you want to use Logstash as the output in 5.3, my understanding is that you need to configure Filebeat, too. @tsg Can you confirm that this is true and that I'm not misunderstanding something fundamental here?

suyograo · 2017-03-23T01:33:14Z

docs/static/filebeat-modules.asciidoc

+* <<parsing-nginx>>
+* <<parsing-system>>
+
+//REVIEWERS: Do we want to add an example that shows how to conditionally select the grok pattern? If not, what guidance should we provide to help users understand how to build a config that works with more than one type of log file?


I am not sure why we need to talk about conditionals here. Can you explain more? Each FBM handles a different log type (mysql, apache2, ..) , so we would do the same in LS as well, no?

@suyograo For example, let's say that a user wants to build a Logstash pipeline that parses both apache2 error and access logs. Wouldn't they need conditional logic or something similar to do that? Right now, we show a separate Logstash config for each fileset.

See Tudor's comment here: #6542 (comment)

suyograo · 2017-03-23T01:36:21Z

docs/static/filebeat-modules.asciidoc

+
+[source,shell]
+----------------------------------------------------------------------
+./filebeat -e -setup -E "output.elasticsearch.hosts=["http://localhost:9200"]"


Is this missing -modules=nginx flag?

@suyograo That's how things should work, but that's not how they work right now. In 5.3, if you run a Filebeat module with the logstash output enabled, you see the error CRIT Exiting: Filebeat modules configured but the Elasticsearch output is not configured/enabled:

Rhodas-MacBook-Pro:filebeat-5.3.0-darwin-x86_64 dedemorton$ sudo ./filebeat -e -modules=system 2017/03/23 17:03:16.622482 beat.go:285: INFO Home path: [/Users/dedemorton/BuildTesting/5.3.0_e5a8c674/filebeat-5.3.0-darwin-x86_64] Config path: [/Users/dedemorton/BuildTesting/5.3.0_e5a8c674/filebeat-5.3.0-darwin-x86_64] Data path: [/Users/dedemorton/BuildTesting/5.3.0_e5a8c674/filebeat-5.3.0-darwin-x86_64/data] Logs path: [/Users/dedemorton/BuildTesting/5.3.0_e5a8c674/filebeat-5.3.0-darwin-x86_64/logs] 2017/03/23 17:03:16.622515 beat.go:186: INFO Setup Beat: filebeat; Version: 5.3.0 2017/03/23 17:03:16.622554 metrics.go:23: INFO Metrics logging every 30s 2017/03/23 17:03:16.622572 logstash.go:90: INFO Max Retries set to: 3 2017/03/23 17:03:16.622625 outputs.go:108: INFO Activated logstash as output plugin. 2017/03/23 17:03:16.622682 publish.go:295: INFO Publisher name: Rhodas-MacBook-Pro.local 2017/03/23 17:03:16.622771 async.go:63: INFO Flush Interval set to: 1s 2017/03/23 17:03:16.622778 async.go:64: INFO Max Bulk Size set to: 2048 2017/03/23 17:03:16.623475 beat.go:221: INFO filebeat start running. 2017/03/23 17:03:16.623535 metrics.go:51: INFO Total non-zero values: 2017/03/23 17:03:16.623542 metrics.go:52: INFO Uptime: 4.588856ms 2017/03/23 17:03:16.623546 beat.go:225: INFO filebeat stopped. 2017/03/23 17:03:16.623551 beat.go:339: CRIT Exiting: Filebeat modules configured but the Elasticsearch output is not configured/enabled Exiting: Filebeat modules configured but the Elasticsearch output is not configured/enabled

That's why I've outlined the steps the way I have. TBH, I think the workflow is kind of jenky overall.

I think @suyograo is right here, the -modules=nginx flag is needed and works because this is specifically meant to use the ES output (-E "output.elasticsearch.hosts=["http://localhost:9200"]").

whoops...you're right. I was looking at this out of context. :-)

suyograo · 2017-03-23T01:53:56Z

docs/static/filebeat-modules.asciidoc

+
+[source,yml]
+----------------------------------------------------------------------
+filebeat.prospectors:


Do we need to specify the filebeat.yml here? If a user has already done ./filebeat --module=nginx they should have the relevant filebeat.yml file, right?

I would think so, if they already set the LS output explicitly.

@suyograo I don't think it works that way. The filebeat.yml isn't updated when you run the module. The settings are determined at runtime based on what's in the manifest file (and what the user passes as variables on the command line). @tsg Can you confirm? It's possible that I've missed something big here. :-)

Yeah, this is needed for 5.3, but hopefully we'll be able to simplify it with 5.4. I'd say a single filebeat.yml example per module is enough.

I had missed this in my understanding of how FBM works. Agree with what Dede has here, and making it better for 5.4

acchen97 · 2017-03-23T05:55:56Z

docs/static/filebeat-modules.asciidoc

+== Working with Filebeat Modules
+
+Starting with version 5.3, Filebeat comes packaged with pre-built 
+{filebeat}filebeat-modules.html[modules] that contain the configuration needed


Myabe "configurations needed to collect, parse, enrich, and visualize data from various log file formats out-of-the-box"?

Sure, but I'll remove "out-of-the-box" because it's grammatically ambiguous (could be misread--and mistranslated--as modifying "log file formats"

acchen97 · 2017-03-23T06:06:28Z

@dedemorton thanks for this, left a few comments, but looks good overall. We should link off to these docs from the FBM documentation as well.

tsg · 2017-03-23T18:08:20Z

docs/static/filebeat-modules.asciidoc

+  paths:
+    - /var/log/mysql/mysql-slow.log*
+    - /var/lib/mysql/hostname-slow.log
+  exclude_files: [".gz$"]


Here the multiline configuration is missing. You can take it from: https://github.com/elastic/beats/blob/master/filebeat/module/mysql/slowlog/config/slowlog.yml#L7

It would be good if these samples would resemble what FBM do as much as possible. I can help with that if want.

@tsg I think the multiline is defined in the Logstash config. You're at a bit of a disadvantage because you can't see the Logstash configs in this review (Suyog is still working on them). They are in this PR: #6791.

dedemorton · 2017-04-08T03:19:21Z

@suyograo I've updated the Filbeat configs, but I still don't think this is 100% there.

Here's a list of the problems that I'm seeing:

The apache access log darwin-2.4.23.log still returns "_grokparsefailure" tag (works ok on ubuntu-2.2.22.log, tho)
The nginx error log seems to parse OK, but in the dashboard I under "Nginx Errors over time," I don't see any data, just the message: "Area charts require more than one data point. Try adding an X-Axis Aggregation" (Not sure this has anything to do with LS parsing, tho. I might have seen this with the FBMs.)
The system auth log auth-ubuntu1204.log returns "_grokparsefailure" and _geoip_lookup_failure tags. I don't have time right now to figure out if I've done something wrong in my test files....
UPDATE 4/17: Tested against secure-rhel7.log, and works.
Didn't have time to test the syslog because I got Kibana into some weird state where it wasn't showing any data.
UPDATE 4/17: I tested the syslog and I got grok parse failures on every log that I tried to test. Looks like the multiline I used in Filebeat might be wrong? I used the one from the FBM config, so I'm not sure what I did wrong.

dedemorton · 2017-04-17T21:52:32Z

@suyograo I did a bit more testing of the LS configs today. I didn't want to merge this while you were away, but I'd like to get it merged soon. For the problems I listed in the previous comment:

For 1 and 3, I think the LS config is OK because it does work on some of the logs. I assume I'm getting geoIP lookup failures because I'm using localhost, so it's probably OK to ignore those failures.
For 2, I think the problem is the dashboard, not the LS config. But what do I know? WDYT?
For 4, not sure what to do. Maybe you or @tsg can take a look and see if the multiline config in Filebeat is wrong. I'm pretty sure I copied the config that's used in the Filebeat module.

One other thing: I think the setup step is a little hacky. This step may actually result in a filebeat index getting created for system events...because the default Filebeat config harvests the system log by default. I'm not sure if that matters...or whether I should mention it. But I think some people might wonder why the system info is being captured. If I comment out that section in the filebeat.yml, I think the setup still happens, but Filebeat stops with a critical error. That's not necessarily bad because we want users to stop Filebeat anyhow, but it still seems kind of hacky. Wondering what you and Tudor think about this.

suyograo · 2017-04-24T14:52:36Z

Apologies for the delay; I'll re-run this today and update here.

suyograo · 2017-04-25T18:26:19Z

I tested the syslog config and it does not work as you mentioned. I don't know what the issue is here since the multiline config looks ok. I am thinking we should merge the PR as is which will give users some "template" to use for migration and we can iterate on it. I have a feeling this is one of the differences between the grok implementation in ES and LS

suyograo · 2017-04-25T18:26:23Z

LGTM

Cherrypick #6832 5.3

Cherrypick #6832 into 5.4

Cherrypick #6832 5.x

dedemorton added docs work in progress labels Mar 22, 2017

suyograo reviewed Mar 23, 2017

View reviewed changes

suyograo suggested changes Mar 23, 2017

View reviewed changes

acchen97 reviewed Mar 23, 2017

View reviewed changes

acchen97 closed this Mar 23, 2017

acchen97 reopened this Mar 23, 2017

tsg reviewed Mar 23, 2017

View reviewed changes

dedemorton force-pushed the fbm_docs_issue#6542 branch from 7de5e34 to 4f61ce2 Compare March 24, 2017 22:04

Add docs about configuring LS for use with Filebeat modules

9128754

dedemorton force-pushed the fbm_docs_issue#6542 branch 2 times, most recently from cd4f3d4 to 6f26395 Compare April 8, 2017 03:10

Add fixes from the review

4fedae8

dedemorton force-pushed the fbm_docs_issue#6542 branch from cd418f5 to 4fedae8 Compare April 17, 2017 21:31

dedemorton added needs review and removed work in progress labels Apr 18, 2017

suyograo approved these changes Apr 25, 2017

View reviewed changes

dedemorton merged commit 163d27d into elastic:master Apr 27, 2017

dedemorton mentioned this pull request Apr 27, 2017

Cherrypick #6832 5.3 #6992

Merged

dedemorton added a commit that referenced this pull request Apr 27, 2017

Merge pull request #6992 from dedemorton/cherrypick_#6832_5.3

edd1b40

Cherrypick #6832 5.3

dedemorton mentioned this pull request Apr 27, 2017

Cherrypick #6832 into 5.4 #6993

Merged

dedemorton added a commit that referenced this pull request Apr 27, 2017

Merge pull request #6993 from dedemorton/cherrypick_#6832_5.4

0037bc8

Cherrypick #6832 into 5.4

dedemorton added a commit that referenced this pull request Apr 27, 2017

Merge pull request #6994 from dedemorton/cherrypick_#6832_5.x

50bca34

Cherrypick #6832 5.x

Add docs about configuring LS for use with Filebeat modules #6832

Add docs about configuring LS for use with Filebeat modules #6832

Uh oh!

Conversation

dedemorton commented Mar 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suyograo commented Mar 23, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

acchen97 commented Mar 23, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dedemorton commented Apr 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dedemorton commented Apr 17, 2017

Uh oh!

suyograo commented Apr 24, 2017

Uh oh!

suyograo commented Apr 25, 2017

Uh oh!

suyograo commented Apr 25, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

dedemorton commented Mar 22, 2017 •

edited

Loading

dedemorton commented Apr 8, 2017 •

edited

Loading