Expedition Ver 1.1.0 QuickStart
Expedition Ver 1.1.0 QuickStart
Hands-on Workshop
Expedition:
Migration and Security
Assessment
http://www.paloaltonetworks.com
© 2019 Palo Alto Networks. Proprietary and Confidential Last Update: 20190501
Table of Contents
Introduction ............................................................................................................... 4
Activity 0 – Initial Setup ............................................................................................ 5
Task 1 – Configure Machine Learning Module (HTTPS) ............................................................................. 5
Task 2 – Review and fix Internal Checks (SSH) ........................................................................................... 6
Dashboard.................................................................................................................. 8
Activity 1 – PanOS Traffic Logs (where the Magic begins) .................................. 9
Task 1 – VM-Series. Configure Scheduled Log Export ................................................................................ 9
Task 2 – Import a Device into Expedition ................................................................................................ 11
Task 3 – Process Log Files ....................................................................................................................... 12
Task 4 – Creating a Project ...................................................................................................................... 14
Activity 2 – Rule Enrichment .................................................................................. 16
Task 1 – The Log Connector .................................................................................................................... 16
Task 2 – Set Rules for Enrichment ........................................................................................................... 18
Task 3 – Enrich the missing Zones ........................................................................................................... 21
Task 4 – Enrich Source IP Address ........................................................................................................... 22
Task 5 – Enrich App-ID ............................................................................................................................ 23
Task 6 – Export the Discovery to Excel .................................................................................................... 26
Activity 3 – Rule Suggestions by M. Learning ...................................................... 27
Task 1 – Set Rules for M. Learning .......................................................................................................... 27
Task 2 – Review the learned Servers ....................................................................................................... 30
Task 3 – Import Suggested Rules............................................................................................................. 31
Activity 4 – Best Practices Adoption ..................................................................... 35
Task 1 – Run the Best Practices Assessment Tool (BPA) .......................................................................... 35
Task 2 – Export Report to Excel ............................................................................................................... 37
Task 3 – Apply remediation to the failed Checks ..................................................................................... 39
Task 4 – Reviewing the Security Policies Best Practices ........................................................................... 42
Activity 5 – Importing Iron-Skillet .................................................................................. 44
Task 1 – Import a new Iron-Skillet configuration ..................................................................................... 44
Task 2 – Move Custom Reports and Security Profiles to your Configuration ........................................... 45
Task 3 – Apply the iron-skillet profiles to your Rules ............................................................................... 47
Expedition is the fourth evolution of the Palo Alto Networks Migration Tool. The main purpose of this tool
is to help to reduce the time and effort to migrate a configuration from one of the supported vendors to
Palo Alto Networks.
By using the Migration Tool, we were able to convert a configuration from Checkpoint, Cisco or any other
vendor to PanOS and gave you more time to improve the results. Migration Tool 3 added some functionality
to allow our customers to enforce security policies based on App-ID and User-ID.
With Expedition we have gone one step further. Not only because we want to continue helping to facilitate
the transition of a security policy from other vendors to PanOS but we want to ensure the outcome is the
best as possible. This is why we added a Machine Learning module who can help you to generate new
security policies based on real traffic logs and the introduction of the Best Practices Assessment Tool to
check that the configuration complies with the Best Practices recommended by our security experts.
In 2019 we introduced support for project Iron-Skillet too. Iron-Skillet provides a day1 configuration for a
PanOS device with some of the configuration best practices already configured. With this you can create a
base configuration to be used in your migrations with almost everything configured without any effort and
allowing you to customize some of the parameters like hostname or Management IP address before
generate it.
Expeditions is the glue between many initiatives born from different internal projects here at Palo Alto
Networks to be easier to consume them.
Step 1: First, make sure your laptop is installed with a modern browser that supports HTML 5.0. We
recommend using the latest version of Chrome. We recommend using the Private Browsing
mode in your browser for this lab so any extensions do not interfere.
Validate the Expedition ML Address IS NOT 127.0.0.1 and set the TEMPORARY DATA STRUCTURE
FOLDER Path to:
/datastore
Note: Expedition can read CSV logs generated by PanOS devices but to analyze the data requires those files
been stored on a format called PARQUET. PARQUET is a format utilized when a big amount of data needs to
be processed and accessed in parallel to speed up the process.
Task 2 – Review and any fix Internal Checks (via SSH) listed
First thing we have to do is create the “datastore” folder in our Expedition instance:
Step 1: Using a SSH Client connect to your Expedition instance or use the Console link from the lab
portal. Log in with these credentials expedition / paloalto
• Create the folder /datastore and allow the web server user write on that folder (Default Expedition
web server user is www-data)
• Review the Task Manager now has been started and looks green.
• Following the recommendation from the Internal Checks for “OS settings” related to the SQL_Log_bin
flag we have to modify one configuration file:
i) sudo vi /home/userSpace/userDefinitions.php
ii) Add this line at the bottom by moving the cursor to the end of the file and pressing “o”
define ('DBSQL_LOG_BIN',0);
iii) To save the changes and exit the editor press ESC and type “:wq”, press enter.
End of Activity 1
The Jobs and Task Manager: This process is in charge to start any task we want to run and check if the job
is still alive from the backend, if we want to retrieve the running configuration from a device or we want to
push the API calls generated after a transformation to one or more devices the Task manager will be
responsible of that, so keep it UP and Running otherwise the task will be queued until you start it.
Expedition was conceived to run everything in a single VM or split in two. One piece will run the GUI and
basic database and another piece with more resources (CPU and RAM) will run the Data analytics and
Machine Learning. For this reason, in the case we were using two different VMs we can monitor the CPU,
RAM and DISK of each instance, ML HEALTH will show the status of the Analytics and the gauge charts shows
the status of the VM who is running the GUI and the database. In this exercise, all the charts will show the
same information because we are using a single instance to run both tasks.
Expedition is capable of ingesting CSV files generated directly from our firewalls but on
environments with a huge number of logs is preferred to use a syslog server and then, every day,
rotate the log and import into Expedition. Expedition comes with a syslog-NG to run that task as
well, but is preferred to run on a separate VM because the syslog has to write a lot to the disk and
that can cause Expedition to run slowly.
This task will show you how to configure your Next-Generation Firewall to export daily the logs to Expedition
by using SCP (using a secure channel ). With this process you don’t need to worry about doing any addition
task to get the logs out from your PanOS device.
Step 2: Go to Device, and select from the left panel “Scheduled Log Export”, click on Add button and fill the
fields with the information in the screenshot:
This step has already been done but would be needed for a new Expedition setup.
Step 4: Click again on Test SCP Server connection to validate we can write on that folder
• /home/expedition/logs
Note: With this our VM-Series Firewall will start sending logs every day at midnight to Expedition, for this
Lab we have already uploaded some log files to that folder and we don’t need to wait to start working
on this laboratory.
Click OK.
Expedition allow us to connect to a PanOS device using API keys. When it connects to the device and retrieve
the configuration it keeps that configuration encrypted on the hard drive. That said if you make changes in
the PanOS device and you want to update the configuration we had stored in Expedition you have to repeat
the process to retrieve the running or candidate config again.
Step 1: Connect to your Expedition instance via HTTPS. Navigate to the DEVICES option. This view shows
all defined Palo Alto Networks devices.
Step 3: Fill the fields with the following information and click on Save
Field Value
Device Name VMSeriesFW
Hostname/IP 10.30.11.1
Serial The serial captured on Task 1 – Step 5
Model vm-series
Step 4: Edit the Device by double-click on it or clicking on the Edit icon located near the row’s end.
Let’s add the credentials in order to generate the API calls to interact with our firewall.
• On Authentication API Keys, click on the “+” to fill the Username and Password. Use the credentials
admin / Ignite19. Click Add.
Step 6: Edit the VMSeriesFW again. Go to M.LEARNING TAB located at the end of the Device Edit
Window.
• Click On SAVE.
Step 1: Reopen the Device and come back to the M.LEARNING TAB
Note: This process can take up to 5 minutes. Notice the green check at the end when the file has been
processed correctly.
This process can be tracked from the cli as well, just enter via cli to Expedition and run as root.
# tail -f /tmp/error_logCoCo
This process will convert the CSV logs into a new file called PARQUET and will be stored in the folder
we created at the beginning of the lab “/datastore”. This process will reduce the amount of disk we
need for the data analysis. Example: A log file with 100MB, if we compress to zip it will be reduced to
If you click on the tab called “TRAFFIC” two times it will show the number of hits by application type
seen by day. This information can be exported as Excel file for further analysis since it will include all
the apps seen by Rule. This is useful to understand if the log files imported has logs generated for a
specific rule name.
Step 1: Navigate from the Expedition GUI to the tab called PROJECTS and click on the plus icon located
at the end of the header called LIBRARY to create a new one.
Field Value
Name ProjectOne
Source Select the firewall VMSeriesVM
Note: With these steps we have created the project and attached the Firewall to it. We can modify these
settings by clicking on “settings” on the selected Project.
Step 3: Double click on the Project called ProjectOne to get access on it.
Step 5: Expedition will show you a Summary about the objects and rules imported.
App-ID and User-ID charts show the percentage of rules found using them and the third chart is
related to the Best Practices Assessment Tool, this chart will not have any data until we run the
analysis.
On the bottom bar there are one or two combo boxes, one is related to the configuration we are
working on and the other in case of any will show the vsys/DG where we are focusing at the moment.
A Log Connector is a way to filter the information from all the logs we have stored in the PARQUET format
to focus only in the data produced by a specific vsys from a firewall or from a group of firewalls included in
a Panorama Device Group. When we create a Log Connector we are focusing only in some data and focusing
on a period of time.
Field Value
Connector Name Provide a name – e.g. ‘Last-30-Days’
Device Select your firewall
Virtual System vsys1
Period Custom
Start Date Add the start date
End Date Add the end date
Step 4: Click on Save. This will automatically create the Log connector and make it active.
Note: You can see the active log Connector from the bar below the grid. It’s mandatory to have one Active
in order to use Rule Enrichment.
We want to add Zones to some Rules. The goal of this task will be select the rules we want to enrich and let
Expedition to tell us which Zones our firewall found for those rules and bring the Zone From and Zone To to
our Security Policies.
Step 2: Change from the Bottom bar the vsys to vsys1. This will enable the Security Policies view
Step 3: Select the following Rule Names, use the Ctrl + click to do it:
• Remote Access FW
• Autofocus
• Outbound Dns
• Firewall Management
• VPN Didac
Step 4: With the Rules selected right click over one and select Rule Enrichment -> Monitor (Selection)
Step 5: Let’s use the right-click from one of the rules to apply a predefined filter to show only the Rules with
the Rule Enrichment Tag
Step 6: Located on the bottom bar, click on the green button called “Discovery” and select Rule Enrichment
This process will check for all the rules tagged with RE Enabled (Rule Enrichment) tag and start the
analysis in the backend. Rule Enrichment will group all the data seen by rule and show it grouped.
Step 9: Advanced – You can follow the progress from the GUI or CLI
• From GUI: After click on Analyze a URL will show up in the middle of the progress bar. If you click on
that URL you will see the progress from another html page.
Step 3: Enable Zone From to import all the Source Zones found
Step 5: Check you are Updating the Existing Rule (Replace Rule) and the Source, VSYS and Template
matches yours like in the screenshot.
(1) VMSeriesFW_007051000019156.xml
(2) vsys1
(3) template1
Note: With this activity you learned how to fulfill the missing parameters on your security rules like Zones but
It can be applied to Users / Applications / Services and Sources and Destinations IP address/Regions.
Try always to see if there is a chance to remove all the “any” from your Rules.
Let’s continue working with this Security Policies and reuse the analysis we did to reduce the attack
surface on the rule named Outbound DNS by adding the Source IP addresses seen.
Step 1: From the POLICIES TAB click on the Discovery button and select Rule Enrichment.
• Check Source
• Under the Source keep IPs to import only the ip address instead the Regions to the Rule
Check the original rule has been disabled to keep for review a new rule has been created with the
changes introduced by the Rule Enrichment. Notice now the rule name is CL (cloned) uA (No Users, Yes
Application) Original Rule Name. This is used by Expedition to know what type of rule we have created and
in case we run again the Rule Enrichment consider if the change to make requires to split the rule because
now we discovered users and we want that on a separate rule so the rule will be called CL (cloned) UA
(Yes User, Yes Application) Original Rule Name.
Step 1: From the POLICIES TAB click on the Discovery button and select Rule Enrichment.
• Check Application
We can export the analysis to Excel to be analyzed offline. To export the analysis:
Step 1: From the Discovery Window, select the RULE ENRICHEMENT TAB
Step 2: Click on Export Excel button from the bottom bar close the pagination buttons.
End of Activity 2
It’s important to understand the differences between Machine Learning and Rule Enrichment.
When we use Rule Enrichment Expedition will group all the data by Rule Name and it will create a
rule with all the traffic seen with Users, another one with the traffic seen without users, another one
with the traffic where the applications where found through their default-port and another one with
the traffic where the applications were found on a port different the default port.
That means with Rule Enrichment we will get a maximum of 4 rules by each rule we are analyzing.
In case to use Machine Learning, what Expedition will do is to create as much rules as consumptions
models we were able to identify from the traffic analyzed. That means we can get tons of rules from
any rule selected for the analysis.
Machine Learning must be used when a security policy can lead us to have like a new ruleset
basically because the rule itself was too wide open, like when we are on a Green Field and we have
mostly one rule that allows all the traffic from Trust to Untrust, there we want to know who the
Servers are and who is consuming what from the network.
To demonstrate this, we will show how a simple Rule that allows all the traffic between some Zones
will be transformed in many new Rules more specific to reduce the attack surface and create a new
security ruleset.
Step 1: Check we have a Log connector created and Active. Navigate to PLUGINS TAB.
Step 3: Select the Rule Name 19) VPN Didac (this rule is actually disabled because the Rule Enrichment
process)
Step 4: Right-click and select Machine Learning -> Monitor to enable the rule for being analyzed.
Step 6: Click on the green button Discovery and select Machine Learning
Step 7: The first TAB is called ANALISYS RESULT has on the right panel a button named Analyze Data.
b) Common: Are considered common applications those that are present in all the networks and
generates a huge volume of logs. Ex: ping, dns, ldap. In some cases, you won’t want to waste
resources to analyze logs related to those applications to speed up the analysis for other applications.
In case Expedition finds traffic regarding those applications will be considered as source “any” and
destination “any”. Ex: ping Rule suggested ANY – ANY – ping – ALLOW
c) Peer-to-Peer: All the applications classified as peer to peer by Palo Alto Networks.
d) Global: All the other applications. Expedition will analyze sources, destinations, users, service ports
to suggest Rules based on how they are consumed.
Step 8: Click on Analyze Data to start the analysis. Wait until a URL will be shown in the progress bar. In
this exercise you cannot click there because Expedition It’s behind a NAT and the URL its internal.
The Flow has been calculated after figure it out who are the servers on the networks.
Tag tells you the container for the APP. In this case all were Global.
This is the list of the servers we found from the logs after analyze who is who in the network. This is important
to Expedition to understand the flow of the communications. Expedition supports asymmetric traffic
environments.
• Point your mouse over the column “Server IP Address” and click on the arrow and click on the “Group
by this field” In this example nothing will change but in real life environments will give you the
applications served by each server we used.
• Transform: Unchecked
• SOURCE: VMSeriesFW_007051000019156.xml
• VSYS/DG: vsys1
Step 7: Scroll down on the Security Policy and select the new 3 rules created
Step 11: Let’s Merge 2 of the rules we just generated (EX27 and EX28) by just one by selecting them
and then right-click and select Rule Actions -> Merge
Step 12: Check the merged rule. The rule contains all the information merged and the rule is tagged
as merged for validation.
End of Activity 3
Expedition has gone one step beyond and has integrated the Best Practices Assessment Tool and
implemented some remediations inside to be automatically configured to be in compliant of the Assessment
Tool.
Step 1: Navigate from within the Project to the BEST PRACTICES tab
Note: The BPAT will only work if we have one Base Configuration loaded in the Project. Remember the base
Configuration it’s a Palo Alto Networks configuration and can be set as Base Configuration from the EXPORT
tab.
Step 3: After the process ends you can read the last Run date to confirm that was just executed
The next percentage is showing that, the percentage of the checks Passed, in this example 44,5% of the
total checks has been passed.
Take a look to the blue bag, that blue bar tells us there are 20 checks that Expedition can remediate
automatically. On summary we passed 53 checks and failed 46.
Next chart is showing us the amount of checks that Expedition can remediate by Topic, in our case all the
checks that can be remediated are under the Device Topic only and the percentage is telling us if we
remediate them we will increase the best practice adoption to 61,3% instead the 44,5% we have if we don’t
remediate them.
The Radar chart just shows us how we are doing the adoption by topic, our goal is always try to cover the
whole Radar chart with green (100% passed) but not all the environments are equal, in case we don’t need
SSL Decrypt or HA those will stay in 0 so we will never reach 100% of the checks passed but it will be ok
because we don’t need them.
Step 1: From within the Project and keeping selected the Best Practices Tab let’s select the next one called
Analysis.
Step 2: Open from the Tree located at the left the Device option and then select Administrators
This will show us two panels; the left panel shows the different topics (Device, Objects, Policies,
Network and Panorama in case the configuration comes from Panorama) and the right panel who
will show us the checks associated to the selected topic. The left panel will act as filter for the right’s
panel.
Step 3: Click on the Export Excel blue button located at the bottom right of the current view.
You can use this Excel file to track your changes and review them before plan how to remediate all you
can.
Step 1: Navigate now to the Authentication Settings option, check the gray bag icon at the right of the
view, if it’s dark gray indicates that check can be automatically remediate.
In this case the last 3 checks can be remediated by the recommended values.
Step 5: Point your mouse to one of the columns and when the arrow shows up click on Columns ->
Reference
Step 6: Go back to the Dashboard and recheck the percentage of the Passed Checks
Step 7: From the Analysis tab click on the device option and select all the checks
Step 8: Click on Remediate, that will remediate all the checks available under device.
Step 9: Go back to the Dashboard to check all the remediation was applied
This view shows some checks against the security policies configuration best practices, you can see how
your security policies has been implemented. Goal is to follow the best practices at the time to manage the
Security rules like ensure all the rules have Description, or you are not abusing of the LOG START that can
create tons of logs.
Step 2: Expand the Rule under vsys1 to view all the checks
Step 3: Point your mouse in the icon for pass / failed to see the check description
Step 4: Let’s Review the Rules to find the ones where the Log Start was enabled so the check will be
seen as Failed, because we recommend to don’t abuse of the Log Start.
Step 5: Select the Rules where the check Log Start failed and click on the Remediate button
Step 6: A new window will be shown. Select SELECTION and check UNSET LOG START Flag
Step 8: Let’s now review the Rules where the Flag DSRI failed, that is related to the function Disable
Server Response Inspection.
Step 9: Select the Rules affected by that flag and click on Remediate
Step 10: Click on SELECTION and check UNSET Disable Server Response Inspection flag
End of Activity 4
Iron-skillet can be used from Expedition to create base configuration files to import on top the policies and
objects migrated from other vendors or to grab some pieces like best practices security profiles and import
them in your current Palo Alto Networks configuration.
In this exercise we will create an Iron-Skillet configuration and we will import the security profiles to our
current project and apply them to all our rules with the Bulk change capability.
Step 1: From within the current Project go to the IMPORT tab. Select Palo Alto TAB and click on IRON-
SKILLET TAB
This will generate a full PanOS firewall configuration based on version 9.0 and holding a ton of the best
practices already configured. In this case we want to focus in the Security Profiles iron-skillet
provides and a list of custom reports ready to be consumed. Let’s import them into your project.
First check the configuration selected is panos_9.xml and the virtual system is vsys1 from the bottom bar
Step 1: Click on the Objects TAB and then select Contents TAB
Step 2: Review the profiles to see there are some like Inbound-AV, Inbound-AS, etc
Step 4: Go to EXPORT TAB and open the 2 trees to see the panos_9 and the Base Configuration files
Step 5: From the Left Panel select on vsys1 the Profiles object and inside it the Security and Security
Group
Step 7: Repeat the same with Reports under vsys1 from the left panel to the right
Step 1: Go to POLICIES
Step 2: Point your mouse in the rule where a source zone is Untrust like id 25 and with right-click select
Add to filters.
Step 3: A new window will show up with the filter click on Apply. That will show the rules where the Untrust
zone is the From Zone.
Step 5: Apply bulk change with right-click to add the profile group INBOUND to those rules
Step 6: Edit one of the rules to validate the Profile group has been attached.
End of Activity 5
Step 2: Check the Atomic option is selected from the bottom bar
• SubAtomic: A single API call will contain a single object, so that means for the case of the address
if you have 100 address you will get 100 API calls.
This will try to generate the XML output using your Base configuration plus the changes we made from
the GUI, after that the API calls will be shown on the view
The Id tells you the order in case you want to be selective at the time to push the API calls back to your
firewall, remember if we have address Groups we need to send first the Address because them can be
members of the groups. So, the order on how you send the API calls matters. Expedition with the ID will
in case you select some API calls it will send in the right order automatically.
If you don’t select any API call but you press the [Step 2] Send API Request ALL the API calls will be
sent in the right order.
Step 4: Don’t select any Rule and click on the [Step 2] Send API Requests
Step 5: A new window will be shown, then you can select the devices where to push the API calls, in our
Case only the VMSeriesFw will be shown
Step 8: You can see the content of the API calls by double clicking on each one
End of Activity 6