Cisco Meraki metrics package [WIP] #11069

tommyers-elastic · 2024-09-10T21:51:59Z

Proposed commit message

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

packages/cisco_meraki_metrics/manifest.yml

packages/cisco_meraki_metrics/docs/README.md

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml

packages/cisco_meraki_metrics/changelog.yml

packages/cisco_meraki_metrics/data_stream/device_health/manifest.yml

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml

packages/cisco_meraki_metrics/data_stream/device_health/manifest.yml

Co-authored-by: Ishleen Kaur <[email protected]>

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml

gpop63 · 2024-09-13T11:32:37Z

Here's the full mappings, hopefully I didn't miss any. Tried to add unit and metric_type as well where it makes sense. Also renamed some metrics that were misspelled like latancy and left a review in beats as well.

Details

- name: meraki
  type: group
  fields:
    - name: organization_id
      type: keyword
      dimension: true
    - name: device
      type: group
      fields:      
        - name: serial
          type: keyword
          dimension: true
        - name: address
          type: keyword
        - name: firmware
          type: keyword
        - name: imei
          type: keyword
        - name: lan_ip
          type: ip
        - name: location
          type: keyword
        - name: mac
          type: keyword
          dimension: true
        - name: model
          type: keyword
        - name: name
          type: keyword
        - name: network_id
          type: keyword
          dimension: true
        - name: notes
          type: keyword
        - name: product_type
          type: keyword
        - name: tags
          type: keyword
        - name: details
          type: object
        - name: status
          type: group
          fields:
            - name: gateway
              type: ip
            - name: ip_type
              type: keyword
            - name: last_reported_at
              type: date
            - name: primary_dns
              type: keyword
            - name: public_ip
              type: ip
            - name: secondary_dns
              type: keyword
            - name: value
              type: keyword
    - name: uplink
      type: group
      fields:
        - name: high_availablity
          type: group
          fields:
            - name: enabled
              type: boolean
            - name: role
              type: keyword
        - name: last_reported_at
          type: date
        - name: interface
          type: keyword
          dimension: true
        - name: status
          type: keyword
        - name: ip
          type: ip
        - name: gateway
          type: ip
        - name: public_ip
          type: ip
        - name: primary_dns
          type: keyword
        - name: secondary_dns
          type: keyword
        - name: ip_assigned_by
          type: keyword
        - name: loss_latency
          type: group
          fields:
            - name: ip
              type: ip
            - name: loss_percent
              type: float
              metric_type: gauge
              unit: percent
            - name: latency_ms
              type: float
              metric_type: gauge
              unit: ms
    - name: cellular
      type: group
      fields:
        - name: gateway
          type: group
          fields:
            - name: uplink
              type: group
              fields:
                - name: network_id
                  type: keyword
                  dimension: true
                - name: last_reported_at
                  type: date
                - name: apn
                  type: keyword
                - name: connection_type
                  type: keyword
                - name: dns1
                  type: keyword
                - name: dns2
                  type: keyword
                - name: gateway
                  type: ip
                - name: iccid
                  type: keyword
                  dimension: true
                - name: interface
                  type: keyword
                - name: ip
                  type: ip
                - name: model
                  type: keyword
                - name: provider
                  type: keyword
                - name: public_ip
                  type: ip
                - name: signal_stat
                  type: group
                  fields:
                    - name: rsrp
                      type: float
                      metric_type: gauge
                    - name: rsrq
                      type: float
                      metric_type: gauge
                - name: signal_type
                  type: keyword
                - name: status
                  type: keyword
    - name: network
      type: group
      fields:
        - name: appliance
          type: group
          fields:
            - name: vpn
              type: group
              fields:
                - name: site_to_site
                  type: group
                  fields:
                    - name: network_id
                      type: keyword
                      dimension: true
                    - name: mode
                      type: keyword
                    - name: hub
                      type: group
                      fields:
                        - name: hub_id
                          type: keyword
                        - name: use_default_route
                          type: boolean
                    - name: subnet
                      type: group
                      fields:
                        - name: local_subnet
                          type: ip_range
                        - name: use_vpn
                          type: boolean
        - name: health
          type: group
          fields:
            - name: channel
              type: group
              fields:
                - name: radio
                  type: group
                  fields:
                    - name: serial
                      type: keyword
                      dimension: true
                    - name: model
                      type: keyword
                    - name: tags
                      type: keyword
                    - name: wifi0
                      type: group
                      fields:
                        - name: start_time
                          type: date
                        - name: end_time
                          type: date
                        - name: utilization80211
                          type: float
                          metric_type: gauge
                          unit: percent
                        - name: utilizationNon80211
                          type: float
                          metric_type: gauge
                          unit: percent
                        - name: utilizationTotal
                          type: float
                          metric_type: gauge
                          unit: percent
                    - name: wifi1
                      type: group
                      fields:
                        - name: start_time
                          type: date
                        - name: end_time
                          type: date
                        - name: utilization80211
                          type: float
                          metric_type: gauge
                          unit: percent
                        - name: utilizationNon80211
                          type: float
                          metric_type: gauge
                          unit: percent
                        - name: utilizationTotal
                          type: float
                          metric_type: gauge
                          unit: percent
    - name: switch
      type: group
      fields:
        - name: port
          type: group
          fields:
            - name: port_id
              type: keyword
              dimension: true
            - name: name
              type: keyword
            - name: tags
              type: keyword
            - name: enabled
              type: boolean
            - name: poe_enabled
              type: boolean
            - name: vlan
              type: integer
            - name: voice_vlan
              type: integer
            - name: allowed_vlans
              type: keyword
            - name: rstp_enabled
              type: boolean
            - name: stp_guard
              type: keyword
            - name: link_negotiation
              type: keyword
            - name: access_policy_type
              type: keyword
            - name: sticky_mac_allow_list
              type: keyword
            - name: sticky_mac_allow_list_limit
              type: integer
            - name: status
              type: group
              fields:
                - name: enabled
                  type: boolean
                - name: status
                  type: keyword
                - name: spanning_tree
                  type: group
                  fields:
                    - name: statuses
                      type: keyword
                - name: is_uplink
                  type: boolean
                - name: errors
                  type: keyword
                - name: warnings
                  type: keyword
                - name: speed
                  type: keyword
                - name: duplex
                  type: keyword
                - name: usage_in_kb
                  type: group
                  fields:
                    - name: sent
                      type: long
                      unit: byte
                    - name: recv
                      type: long
                      unit: byte
                    - name: total
                      type: long
                      unit: byte
                - name: client_count
                  type: integer
                  metric_type: gauge
                - name: power_usage_in_wh
                  type: float
                  metric_type: gauge
                - name: traffic_in_kbps
                  type: group
                  fields:
                    - name: total
                      type: float
                      metric_type: gauge
                      unit: byte
                    - name: sent
                      type: float
                      metric_type: gauge
                      unit: byte
                    - name: recv
                      type: float
                      metric_type: gauge
                      unit: byte
                - name: secure_port
                  type: group
                  fields:
                    - name: enabled
                      type: boolean
                    - name: active
                      type: boolean
                    - name: authentication_status
                      type: keyword
                    - name: config_overrides
                      type: group
                      fields:
                        - name: type
                          type: keyword
                        - name: vlan
                          type: integer
                        - name: voice_vlan
                          type: integer
                        - name: allowed_vlans
                          type: keyword
                - name: cdp
                  type: group
                  fields:
                    - name: system_name
                      type: keyword
                    - name: platform
                      type: keyword
                    - name: device_id
                      type: keyword
                    - name: port_id
                      type: keyword
                    - name: native_vlan
                      type: integer
                    - name: address
                      type: ip
                    - name: management_address
                      type: ip
                    - name: version
                      type: keyword
                    - name: vtp_management_domain
                      type: keyword
                    - name: capabilities
                      type: keyword
                - name: lldp
                  type: group
                  fields:
                    - name: system_name
                      type: keyword
                    - name: system_description
                      type: keyword
                    - name: chassis_id
                      type: keyword
                    - name: port_id
                      type: keyword
                    - name: management_vlan
                      type: integer
                    - name: port_vlan
                      type: integer
                    - name: management_address
                      type: ip
                    - name: port_description
                      type: keyword
                    - name: system_capabilities
                      type: keyword
    - name: wireless
      type: group
      fields:
        - name: device
          type: group
          fields:
            - name: channel
              type: group
              fields:
                - name: utilization
                  type: object
    - name: appliance
      type: group
      fields:
        - name: network
          type: group
          fields:
            - name: port
              type: group
              fields:
                - name: number
                  type: integer
                - name: enabled
                  type: boolean
                - name: type
                  type: keyword
                - name: drop_untagged_traffic
                  type: boolean
                - name: vlan
                  type: integer
                - name: allowed_vlans
                  type: keyword
                - name: access_policy
                  type: keyword
    - name: license
      type: group
      fields:
        - name: device_model
          type: keyword
        - name: expiration_date
          type: date
        - name: status
          type: keyword
        - name: count
          type: long
          metric_type: gauge
        - name: state
          type: keyword
        - name: expiration_state
          type: keyword
        - name: expiration_threshold_days
          type: long
        - name: soonest_activation_date
          type: date
        - name: soonest_activation_count
          type: long
          metric_type: gauge
        - name: oldest_activation_date
          type: date
        - name: oldest_activation_count
          type: long
          metric_type: gauge
        - name: type
          type: keyword
        - name: systems_manager
          type: group
          fields:
            - name: active_seats
              type: long
              metric_type: gauge
            - name: org_wide_enrolled_devices
              type: long
              metric_type: gauge
            - name: total_seats
              type: long
              metric_type: gauge
            - name: unassigned_seats
              type: long
              metric_type: gauge

add CODEOWNERS entry

gpop63 · 2024-09-30T18:05:47Z

I added agent fields mappings (048665e) because we need mappings for host.mac and host.ip. Right now it might contain fields that we don't need for this integration but I'll leave it for now and come back to it in the final review. More on this issue can be read here.

elastic-package test static -v

2024/09/30 20:45:55  WARN CommitHash is undefined, in both /home/gab/.elastic-package/version and the compiled binary, config may be out of date.
2024/09/30 20:45:55 DEBUG Enable verbose logging
2024/09/30 20:45:55 DEBUG latest version (cached): v0.104.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.104.0 (Timestamp 2024-09-30 20:22:06.351051917 +0300 EEST)
2024/09/30 20:45:55  INFO New version is available - v0.104.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.104.0
Run static tests for the package
2024/09/30 20:45:55 DEBUG Imported ECS fields definition from external schema for validation (embedded in package: false, stack uses ecs@mappings template: true)
--- Test results for package: cisco_meraki_metrics - START ---
FAILURE DETAILS:
cisco_meraki_metrics/device_health Verify sample_event.json:
[0] field "host.ip" is not normalized as expected: expected array, found "172.21.0.4" (string)
[1] field "host.mac" is not normalized as expected: expected array, found "02-42-AC-15-00-04" (string)


╭──────────────────────┬───────────────┬───────────┬──────────────────────────┬────────────────────────────────────────────┬──────────────╮
│ PACKAGE              │ DATA STREAM   │ TEST TYPE │ TEST NAME                │ RESULT                                     │ TIME ELAPSED │
├──────────────────────┼───────────────┼───────────┼──────────────────────────┼────────────────────────────────────────────┼──────────────┤
│ cisco_meraki_metrics │ device_health │ static    │ Verify sample_event.json │ FAIL: one or more errors found in document │  55.100145ms │
╰──────────────────────┴───────────────┴───────────┴──────────────────────────┴────────────────────────────────────────────┴──────────────╯
--- Test results for package: cisco_meraki_metrics - END   ---
Done
Error: one or more test cases failed

tommyers-elastic · 2024-10-01T08:28:52Z

@gpop63 do we need all the extra mappings, or can we get away with just host.ip and host.mac?

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml

packages/cisco_meraki_metrics/data_stream/device_health/manifest.yml

packages/cisco_meraki_metrics/manifest.yml

packages/cisco_meraki_metrics/data_stream/device_health/fields/agent.yml

packages/cisco_meraki_metrics/img/cisco-logo.svg

shmsr · 2024-10-03T09:42:21Z

I've added some comments. Everything else looks good!

Co-authored-by: subham sarkar <[email protected]>

tommyers-elastic · 2024-10-03T19:03:34Z

packages/cisco_meraki_metrics/manifest.yml

+  - security
+conditions:
+  kibana:
+    version: "^8.15.2"


FYI - we need to leave this at 8.15.2 for this to work with the 8.15.2-SNAPSHOT build, but we should be aware that this will NOT work with the public 8.15.2 version. because this is a beta package this is ok, but as soon as 8.15.3 is released we need to bump this version to 8.15.3. please open a ticket on this so we don't forget. thanks!!

packages/cisco_meraki_metrics/manifest.yml

shmsr

I've put one comment. Rest LGTM. Approving!

elastic-sonarqube · 2024-10-03T21:08:39Z

Quality Gate failed

Failed conditions
76.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

elasticmachine · 2024-10-03T21:08:56Z

💚 Build Succeeded

Buildkite Build
Commit: 0b0d9fc

History

💔 Build #16764 failed 23af007
💚 Build #16754 succeeded 53df8a1
💚 Build #16725 succeeded 53bd0ec
💔 Build #16724 failed 7c8a4c1
💔 Build #16717 failed 7c8a4c1

agithomas · 2024-10-04T04:22:02Z

Can we consider re-arranging the order of the input control selector to a more hierarchical ordering
Organisation ID -> Device Model -> Device Name -> Device Serial Number?

agithomas · 2024-10-04T04:33:53Z

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml

+            - name: voice_vlan
+              type: long
+              description: The voice VLAN of the switch port. Only applicable to access ports.
+            - name: status


In the dashboard, can we maintain a table to display all the statuses (client count, duplex, errors, is_uplink, power_usage_in_wh, speed, status, warning, stp_statuuses) for the specific device ?

agithomas · 2024-10-04T04:39:03Z

...raki_metrics/kibana/dashboard/cisco_meraki_metrics-d6b9863a-88e2-4e3d-a2a7-36ca1ee525b1.json

+                                                "stop": 3.33
+                                            },
+                                            {
+                                                "color": "#d6bf57",


Will it be good to show the list of devices that are not in Green status?

agithomas · 2024-10-04T04:39:37Z

...raki_metrics/kibana/dashboard/cisco_meraki_metrics-d6b9863a-88e2-4e3d-a2a7-36ca1ee525b1.json

+                                "palette": {
+                                    "name": "custom",
+                                    "params": {
+                                        "colorStops": [


Will it be good to show the list of devices having unlink latency outside the green status?

agithomas · 2024-10-04T04:42:26Z

Please consider adding the unit of the metric, either in the Y-axis or in the panel title label.

Example: Switch port data throughput, Channel utilisation, Switch Port power usage

agithomas · 2024-10-04T04:45:46Z

...raki_metrics/kibana/dashboard/cisco_meraki_metrics-d6b9863a-88e2-4e3d-a2a7-36ca1ee525b1.json

+            }
+        ],
+        "timeRestore": false,
+        "title": "[Metrics Cisco Meraki] Device Health Overview",


In the screenshot file, the name is appearing as Device Health Overview(1). Looks like the screenshot was taken from a cloned dashboard.

agithomas · 2024-10-04T04:46:59Z

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml

+                - name: enabled
+                  type: boolean
+                  description: Whether the port is configured to be enabled.
+                - name: errors


Kindly include a metric visualisation to capture this information.

If there is a possibility of recurring errors, showing the trend of errors using a line visualisation can be considered.

agithomas · 2024-10-04T04:49:22Z

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml

+                - name: status
+                  type: keyword
+                  description: The current connection status of the port.
+                - name: warnings


Similar to the errors (please see the above comment), showing the number of warnings or trend of the warnings will be helpful

agithomas · 2024-10-04T04:50:18Z

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml

+                        - name: voice_vlan
+                          type: long
+                          description: The voice VLAN of the . Only applicable to access ports.
+                - name: throughput


Can we consider including throughput values in the dashboard?

agithomas · 2024-10-04T05:35:02Z

Please revisit the logo used. I guess, there exist a more relevant one.

agithomas · 2024-10-04T06:31:41Z

Please revisit the logo used. I guess, there exist a more relevant one.

I understand that the current logo is selected to be consistent with other cisco integrations. Marking it as resolved.

elastic-vault-github-plugin-prod · 2024-10-04T08:30:26Z

Package cisco_meraki_metrics - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=cisco_meraki_metrics

* meraki metrics package initial commit * fix metricset name * fix indentation of 'uplink' field mappings * Update packages/cisco_meraki_metrics/changelog.yml Co-authored-by: Ishleen Kaur <[email protected]> * update mappings * fix validation errors in mappings * add store:true to text mappings to allow integration install * add fields descriptions * add sample event * add ecs version * update readme docs * add missing unit mappings * improve titles and descriptions * fix owner add CODEOWNERS entry * automatic lint * add organization_id field * add agent fields * add ecs field reference in docs * add ingest pipeline transforms uplink loss pct field converts rsrp and rsrq to floats * fix field types fix descriptions * remove unused agent fields * add tags and processors * add configuration steps in docs * make organizations an array in agent config * add dashboard * use average instead of sum for uplink latency * add device map lens * update dashboard add dashboard image in integration remove old dashboard remove sample screenshot * capitalize 'metrics' in integration title * fix dashboard title * delete wrongly added dashboard * address reviews * Update version to 0.1.0 Co-authored-by: subham sarkar <[email protected]> * fix manifest version --------- Co-authored-by: Ishleen Kaur <[email protected]> Co-authored-by: Gabriel Pop <[email protected]> Co-authored-by: subham sarkar <[email protected]>

tommyers-elastic added 2 commits September 10, 2024 22:51

meraki metrics package initial commit

ee2e36c

fix metricset name

0263a77

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Sep 10, 2024

fix indentation of 'uplink' field mappings

c49dd24

tommyers-elastic requested review from agithomas and ishleenk17 September 10, 2024 22:02

ishleenk17 reviewed Sep 11, 2024

View reviewed changes

Update packages/cisco_meraki_metrics/changelog.yml

69992f9

Co-authored-by: Ishleen Kaur <[email protected]>

ishleenk17 reviewed Sep 11, 2024

View reviewed changes

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml Show resolved Hide resolved

tommyers-elastic and others added 14 commits September 23, 2024 14:23

update mappings

7db487d

fix validation errors in mappings

2118c52

add store:true to text mappings to allow integration install

5159265

add fields descriptions

8fb0af4

add sample event

4466ebe

add ecs version

d140957

update readme docs

bca9b2f

add missing unit mappings

b82ddc8

improve titles and descriptions

9c9045d

fix owner

79e3f1e

add CODEOWNERS entry

automatic lint

522d2d7

add organization_id field

e9779a7

add agent fields

048665e

add ecs field reference in docs

9f3d5ab

tommyers-elastic commented Oct 1, 2024

View reviewed changes

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml Outdated Show resolved Hide resolved

tommyers-elastic commented Oct 1, 2024

View reviewed changes

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml Show resolved Hide resolved

tommyers-elastic commented Oct 1, 2024

View reviewed changes

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml Outdated Show resolved Hide resolved

tommyers-elastic commented Oct 1, 2024

View reviewed changes

packages/cisco_meraki_metrics/data_stream/device_health/fields/fields.yml Show resolved Hide resolved

shmsr reviewed Oct 3, 2024

View reviewed changes

packages/cisco_meraki_metrics/data_stream/device_health/manifest.yml Show resolved Hide resolved

packages/cisco_meraki_metrics/manifest.yml Outdated Show resolved Hide resolved

shmsr reviewed Oct 3, 2024

View reviewed changes

packages/cisco_meraki_metrics/data_stream/device_health/fields/agent.yml Show resolved Hide resolved

shmsr reviewed Oct 3, 2024

View reviewed changes

packages/cisco_meraki_metrics/img/cisco-logo.svg Show resolved Hide resolved

Update version to 0.1.0

23af007

Co-authored-by: subham sarkar <[email protected]>

tommyers-elastic requested review from shmsr, lucian-ioan and ishleenk17 October 3, 2024 18:56

tommyers-elastic commented Oct 3, 2024

View reviewed changes

shmsr reviewed Oct 3, 2024

View reviewed changes

packages/cisco_meraki_metrics/manifest.yml Outdated Show resolved Hide resolved

shmsr approved these changes Oct 3, 2024

View reviewed changes

fix manifest version

0b0d9fc

agithomas reviewed Oct 4, 2024

View reviewed changes

gpop63 merged commit 51d4eb9 into elastic:main Oct 4, 2024
4 of 5 checks passed

andrewkroh added the Integration:cisco_meraki_metrics Cisco Meraki Metrics label Oct 4, 2024

Cisco Meraki metrics package [WIP] #11069

Cisco Meraki metrics package [WIP] #11069

Uh oh!

Conversation

tommyers-elastic commented Sep 10, 2024 • edited by gpop63 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gpop63 commented Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpop63 commented Sep 30, 2024

Uh oh!

tommyers-elastic commented Oct 1, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shmsr commented Oct 3, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shmsr left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Oct 3, 2024

Quality Gate failed

Uh oh!

elasticmachine commented Oct 3, 2024

💚 Build Succeeded

History

Uh oh!

agithomas commented Oct 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agithomas commented Oct 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agithomas commented Oct 4, 2024

Uh oh!

agithomas commented Oct 4, 2024

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 4, 2024

Uh oh!

tommyers-elastic commented Sep 10, 2024 •

edited by gpop63

Loading

gpop63 commented Sep 13, 2024 •

edited

Loading