ISP Network Design

• PoP Topologies and Design

• Backbone Design
• ISP Systems Design
ISP Network Design • Addressing

ISP/IXP Workshops • Routing Protocols

• Security
• Out of Band Management
• Operational Considerations
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 1 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 2

PoP Topologies

• Core routers – high speed trunk connections

• Distribution routers and Access routers – high
port density
Point of Presence Topologies • Border routers – connections to other providers
• Service routers – hosting and servers
• Some functions might be handled by a single
router

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 3 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 4

PoP Design Modular PoP Design

Other ISPs
Web Cache
ISP Services Hosted Services
(DNS, Mail, News,
• Modular Design FTP, WWW)

Backbone link Backbone link

• Aggregation Services separated according to to another PoP to another PoP

Network
connection speed Core

customer service Consumer

Consumer cable,
xDSL and
DIAL Access wireless Access
contention ratio
security considerations Nx64 customer
aggregation layer
NxT1/E1 customer
aggregation layer
Network
Operations
Centre
Channelised T1/E1 circuits Channelised T3/E3 circuits
Nx64 leased line circuit delivery T1/E1 leased line circuit delivery

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 5 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 6

1
Modular Routing Protocol Design

• Modular IGP implementation

IGP “area” per module
aggregation/summarisation where possible into the core
• Modular iBGP implementation Point of Presence Design
BGP route reflector cluster per module
core routers are route-reflectors
clients peer with core only

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 7 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 8

PoP Modules PoP Modules

• Low Speed customer connections • High Speed customer connections

PSTN/ISDN dialup E1++ speeds
low bandwidth needs medium bandwidth needs
low revenue, large numbers high revenue, low numbers

• Medium Speed customer connections • Broad Band customer connections

56/64K to sub-T1/E1 speeds xDSL, Cable and Wireless
low bandwidth needs high bandwidth needs
medium revenue, medium numbers low revenue, large numbers

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 9 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 10

PoP Modules PoP Modules

• PoP Core
• ISP Services
Two dedicated routers
DNS (cache, secondary)
High Speed interconnect
News, Mail (POP3, Relay)
Backbone Links ONLY
WWW (server, proxy, cache)
Do not touch them!
• Hosted Services
• Border Network
Virtual Web, WWW (server, proxy, cache)
dedicated border router to other ISPs
Information/Content Services
the ISP’s “front” door
Electronic Commerce
transparent web caching

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 11 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 12

2
PoP Modules Low Speed Access Module

• Network Operations Centre

Web Cache
primary and backup locations
AS5300
Access Network
network monitoring Primary Rate T1/E1 Gateway Routers
statistics and log gathering AS2511
PSTN lines to
direct but secure access modem bank To Core Routers

• Out of Band Management Network 2600/3600

PSTN lines to
The ISP Network “Safety Belt” built-in modems

TACACS+/Radius
proxy, DNS resolver,
Content
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 13 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 14

Medium Speed Access Module High Speed Access Module

3800/7206/7600 7200/7600

Channelised T1/E1 Channelised T3/E3

64K and nx64K circuits T1 and E1 circuits

To Core Routers To Core Routers

Mixture of channelised
Mixture of channelised
T1/E1, 56/64K and
T3/E3 and T1/E1 circuits
nx64K circuits

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 15 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 16

Broad Band Access Module ISP Services Module

To core routers
Web Cache

61xx Service Network

Telephone Network 6400
Access Network Gateway Routers
Gateway Routers
IP, ATM

uBR7246 To Core Routers

The cable system

WWW
DNS Mail DNS
cache POP3 NEWS
secondary Relay cache
SSG, DHCP, TACACS+
or Radius Servers/Proxies,
DNS resolver, Content
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 17 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 18

3
 Hosted Services Module Border Module

To core routers

Hosted Network
Gateway Routers To local IXP - ISP1 ISP2
NB - no default route +
local AS routing table only

Network
Border Routers

Customer 1 Customer 3 Customer 5 Customer 7

To core routers
Customer 2 Customer 4 Customer 6

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 19 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 20

NOC Module Out of Band Network

Critical Services
To core routers Module Out of Band
Corporate LAN Management Network
Out of Band
Hosted Network
Management Network Firewall Router
Gateway Routers
consoles
2620/32async
To the NOC
2620/32async NetFlow
enabled
routers
NetFlow
Collector
Billing, Database
and Accounting
Systems Out of Band Ethernet
NetFlow TACACS+ SYSLOG Primary DNS
Analyser server server
Network Operations Centre Staff
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 21 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 22

Backbone Design

• Routed Backbone
• Switched Backbone
• Leased point-to-point circuits
Backbone Network Design nx64K, T1/E1, T3/E3, OC3, OC12,...
• ATM/Frame Relay service from telco
T3, OC3, OC12,… delivery
easily upgradeable bandwidth (CIR)

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 23 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 24

4
Distributed Network Design Distributed Network Design

Customer
ISP Services connections

• PoP design “standardised” Backup

POP Two
Operations Centre
operational scalability and simplicity
• ISP essential services distributed around Customer
connections
Customer
connections
backbone
• NOC and “backup” NOC ISP Services

POP Three POP One

• Redundant backbone links

ISP Services
External External
connections Operations Centre connections

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 25 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 26

Backbone Links Long Distance Backbone Links

• Tend to cost more

• ATM/Frame Relay • Plan for the future (at least two years ahead)
but stay in budget
now less popular due to overhead, extra equipment,
and shared with other customers of the telco Unplanned “emergency” upgrades can be disruptive
without redundancy
• Leased Line • Allow sufficient capacity on alternative paths
more popular with backbone providers for failure situations
IP over Optics and MPLS coming into the mainstream sufficient can be 20% to 50%

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 27 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 28

Long Distance Links Metropolitan Area Backbone Links

• Tend to be cheaper
POP Two
Long distance link Circuit concentration
Choose from multiple suppliers

• Think big
More redundancy

POP Three POP One Less impact of upgrades

Less impact of failures
Alternative/Backup Path

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 29 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 30

5
Metropolitan Area Backbone Links

POP Two
Metropolitan Links

ISP Services
POP Three POP One
DNS, Mail, News
design and location
Metropolitan Links

Cisco ISP
Traditional Point to Point Links Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 31 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 32

ISP Services: ISP Services:

DNS DNS

• Domain Name System • Primary nameserver

Provides name and address resolution Holds ISP zone files
forward zone (list of name to address mappings) for all
Servers need to be differentiated, properly ISP’s and any customer zones
located and specified reverse zone (list of address to name mappings) for all
ISP’s address space
Primary nameserver
One Unix server, fast I/O, reasonable amount of
Secondary nameserver memory (512Mbytes), reasonable disk
Caching nameserver – resolver Located in secure part of net, e.g. NOC LAN

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 33 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 34

ISP Services: ISP Services:

DNS Secondary DNS Example

• Secondary nameserver • apnic.net zone

Holds copies of ISP zone files primary DNS in Brisbane
At least two are required, more is better secondary DNS around the world
Unix server, fast I/O, reasonable amount of memory
$ dig apnic.net ns
(512Mbytes), reasonable disk
;; ANSWER SECTION:
Should be geographically separate from each other apnic.net. 50m44s IN NS svc00.apnic.net.
and the primary DNS apnic.net. 50m44s IN NS ns.ripe.net.
apnic.net. 50m44s IN NS rs.arin.net.
At different PoPs apnic.net. 50m44s IN NS ns.apnic.net.
Tokyo
On a different continent e.g. www.secondary.com ;; ADDITIONAL SECTION:
svc00.apnic.net. 1d23h53m25s IN A 202.12.28.131 Amsterdam
At another ISP ns.ripe.net. 1d23h54m46s IN A 193.0.0.193
rs.arin.net. 1d23h53m25s IN A 192.149.252.21 Washington
ns.apnic.net. 1d9h29m16s IN A 203.37.255.97
Brisbane
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 35 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 36

6
 ISP Services: ISP Services:
Secondary DNS Example DNS

• apnic.net zone
• Caching nameserver
primary DNS in Brisbane (ns.apnic.net)
This is the resolver – it is the DNS cache
secondary DNS run by APNIC in Tokyo
(svc00.apnic.net) Your customers use this as resolver, NOT your primary
or secondary DNS
zone secondaried by
RIPE NCC in Amsterdam
Provides very fast lookups
ARIN in Washington Does NOT secondary any zones
Geographical and service provider redundancy – this One, or preferably two per PoP (redundancy)
is the perfect example! Unix server, fast I/O, large amount of memory
(512Mbytes+ depending on number of zones)

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 37 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 38

ISP Services: ISP Services:

Caching Nameserver Anycasting the Caching Nameserver

Web Cache
Geek
Alert
• One trick of the trade
DIAL network
assign two unique IP addresses to be
for the two DNS resolver systems
To Core Routers
use these two IP addresses in every PoP
route the two /32s across your backbone
Switch redundancy even if the two resolver systems in the local PoP are
Router redundancy down, the IGP will ensure that the next nearest
DNS Cache redundancy Radius proxy
resolvers will be reachable
DNS Cache DNS Cache
Known as IP Anycast
DIAL users automatically given the IP addresses
of DNS caches when they dial in
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 39 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 40

ISP Services: ISP Services:

DNS DNS

• Efficient and resilient design

Primary DNS – keep it secure • Software
Secondary DNS – geographical and provider Make sure that the BIND distribution on the Unix system
redundancy is up to date
Don’t ever put them on the same LAN, switched or the vendor’s distribution is rarely current
otherwise
Pay attention to bug reports, security issues
Don’t put them in the same PoP
Reboot the DNS cache on a regular (e.g. monthly) basis
Caching DNS – one or two per PoP
clears out the cache
reduces DNS traffic across backbone
releases any lost RAM
more efficient, spreads the load
accepted good practice by system administrators

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 41 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 42

7
ISP Services: ISP Services:
DNS Mail

• Implementation • Must have at least two mail hosts (MX records) for
Put all your hosts, point-to-point links and loopbacks all supported domains
into the DNS geographical separation helps
under your ISP’s domain name
• POP3 server dedicated to that function
use sensible/meaningful names
DIAL users get mail from here
Put all your hosts, point-to-point links and loopbacks
into the REVERSE DNS also • SMTP gateway dedicated to that function
don’t forget about in-addr.arpa – many ISPs do DIAL users send mail via here
some systems demand forward/reverse DNS mapping
before allowing access • Mail relay open to CUSTOMERS only!

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 43 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 44

ISP Services: ISP Services:

Mail Example Mail

• telstra.net mail (MX records) • Software

primary MX is mako1
Make sure that the MAIL and POP3 distributions
backup MX is postoffice – two addresses on the Unix system are up to date
backup MX used if primary unavailable
the vendor’s distribution are rarely current
Pay attention to bug reports, security issues,
$ dig telstra.net mx unsolicited junk mail complaints
;; ANSWER SECTION:
telstra.net. 1H IN MX 10 postoffice.telstra.net.
telstra.net. 1H IN MX 5 mako1.telstra.net.

;; ADDITIONAL SECTION: IMPORTANT: Do NOT allow non-customers

postoffice.telstra.net.
postoffice.telstra.net.
1H IN A
1H IN A
139.130.4.7
203.50.1.76 to use your mail system as a relay
mako1.telstra.net. 1H IN A 203.50.0.28
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 45 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 46

ISP Services: ISP Services:

News News System Placement

Customer
News Feeder connections

• News servers provide a Usenet news feed to

POP Two
customers
• Distributed design required
Customer
Incoming newsfeed to one large server Customer
connections connections

Distributed to feed servers in each PoP

Feed servers provide news feed to customers POP Three POP One News Feeder

Outgoing news goes to another server

Separate reading news system
News Feeder
Separate posting news system External External
connections News Collector connections
News Distributor
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 47 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 48

8
ISP Services: ISP Services:
News System Placement News

Customer
News Feeder connections • Software
POP Two Make sure that the Internet News distribution on
the Unix system is up to date
Customer
the vendor’s distribution is rarely current
Customer
connections
connections
Pay attention to bug reports, security issues,
unsolicited junk posting complaints
POP Three POP One News Feeder

IMPORTANT: Do NOT allow non-customers

News Feeder
External External to use your news system for posting messages
connections News Collector connections
News Distributor
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 49 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 50

Where to get IP addresses and AS numbers

• Your upstream ISP

• Africa
AfriNIC – http://www.afrinic.net
• Asia and the Pacific
APNIC – http://www.apnic.net
Addressing • North America
ARIN – http://www.arin.net
• Latin America and the Caribbean
LACNIC – http://www.lacnic.net
• Europe and Middle East
RIPE NCC – http://www.ripe.net
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 51 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 52

Internet Registry Regions Getting IP address space

• Take part of upstream ISP’s PA space

or
ARIN • Become a member of your Regional Internet
Registry and get your own allocation
Require a plan for a year ahead
LACNIC General policies are outlined in RFC2050, more specific
details are on the individual RIR website
• There is plenty of IPv4 address space
registries require high quality documentation

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 53 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 54

9
Addressing Plans – ISP Infrastructure Addressing Plans – Customer

• Address block for router loop-back • Customers assigned address space

interfaces according to need
• Address block for infrastructure • Should not be reserved or assigned on a
per PoP or whole backbone per PoP basis
summarise between sites if it makes sense ISP iBGP carries customer nets
allocate according to genuine requirements, aggregation not required and usually not
not historic classful boundaries desirable

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 55 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 56

Addressing Plans
Addressing Plans – ISP Infrastructure Planning

Phase One • Registries will usually allocate the next

220.10.0.0/21
block to be contiguous with the first
220.10.0.1 220.10.6.255 /24 allocation
Customer assignments Instrastructure Loopbacks Minimum allocation is /21
Very likely that subsequent allocation will
Phase Two make this up to a /20
220.10.0.0/20
So plan accordingly
220.10.0.1
220.10.5.255 /24 /24 220.10.15.255

Original assignments New Assignments

Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 57 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 58

Addressing Plans (contd)

• Document infrastructure allocation

eases operation, debugging and management

• Document customer allocation

contained in iBGP
Routing Protocols
eases operation, debugging and management
submit network object to RIR Database

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 59 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 60

10
Routing Protocols Why Do We Need an IGP?

• IGP – Interior Gateway Protocol • ISP backbone scaling

carries infrastructure addresses, point-to-point links Hierarchy
examples are OSPF, ISIS, EIGRP... Modular infrastructure construction
• EGP – Exterior Gateway Protocol Limiting scope of failure
carries customer prefixes and Internet routes
Healing of infrastructure faults using dynamic
current EGP is BGP version 4 routing with fast convergence
• No link between IGP and EGP

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 61 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 62

Why Do We Need an EGP? Interior versus Exterior Routing Protocols

• Scaling to large network • Interior • Exterior

Hierarchy automatic neighbour specifically configured
Limit scope of failure discovery peers

• Policy generally trust your IGP connecting with

routers outside networks
Control reachability to prefixes
prefixes go to all IGP set administrative
Merge separate organizations routers boundaries
Connect multiple IGPs binds routers in one AS binds AS’s together
together

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 63 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 64

Interior versus Exterior Routing Protocols Hierarchy of Routing Protocols

Other ISPs
• Interior • Exterior BGP4
Carries ISP Carries customer
infrastructure prefixes
addresses only
Carries Internet prefixes
ISPs aim to keep the BGP4
EGPs are independent
IGP small for and OSPF/ISIS
of ISP network topology
efficiency and
scalability

BGP4 Static/BGP4
Local
FDDI IXP Customers

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 65 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 66

11
Routing Protocols: Routing Protocols:
Choosing an IGP IGP Recommendations

• Keep the IGP routing table as small as possible

• Review the “Introduction to Link State If you can count the routers and the point to point links
Protocols” presentation in the backbone, that total is the number of IGP entries
you should see
i.e. – OSPF and ISIS have very similar properties
• IGP details:
• ISP usually chooses between OSPF and ISIS
Should only have router loopbacks, backbone WAN
Choose which is appropriate for your operators’ point-to-point link addresses, and network addresses
experience of any LANs having an IGP running on them
In IOS, both OSPF and ISIS have sufficient “nerd Strongly recommended to use inter-router
knobs” to tweak the IGP’s behaviour authentication
Use inter-area summarisation if possible

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 67 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 68

Routing Protocols: Routing Protocols:

More IGP recommendations iBGP Recommendations

• To fine tune IGP table size more, consider:

Using “ip unnumbered” on customer point-to-point • iBGP should carry everything which
links – saves carrying that /30 in IGP doesn’t contribute to the IGP routing
(If customer point-to-point /30 is required for process
monitoring purposes, then put this in iBGP)
Internet routing table
Use contiguous addresses for backbone WAN links in
each area – can then summarise into backbone area Customer assigned addresses
Don’t summarise router loopback addresses – as iBGP Customer point-to-point links
needs those
Use iBGP for carrying anything which does not DIAL network pools, passive LANs, etc
contribute to the Link State Routing process
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 69 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 70

Routing Protocols:
More iBGP Recommendations

• Scalable iBGP features:

Use neighbour authentication
Use peer-groups to speed update process and
for configuration efficiency
Security
Use communities for ease of filtering
Use route-reflector hierarchy
Route reflector pair per PoP (overlaid clusters)
Use route flap damping at the network edges
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 71 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 72

12
 Security ISP Infrastructure Security

• ISP Infrastructure security

• ISP Network security • router security
• Security is not optional! usernames, passwords, vty filters, TACACS+
• ISPs need to: Disable telnet on vtys, only use SSH
protect themselves
vty filters should only allow NOC access, no
help protect their customers from the Internet
external access
protect the Internet from their customers
See IOS Essentials for the recommended
• The following slides are general practices for ISPs
recommendations
do more research on security before deploying any
network
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 73 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 74

ISP Infrastructure Security

ISP Infrastructure Security ISP Server Protection

• ISP server security Access-list examples: To core routers

usernames, passwords, TCP wrappers, IPTABLES Allow tcp/established to all servers

Service Network
protect all servers using routers with strong filters ICMP
Gateway Routers
applied DNS 2ary: udp/53 and tcp/53
POP3: tcp/110
Mail Relay: tcp/25 and ISP address
• Hosted services security range only
protect network from hosted servers using routers News: tcp/119 and ISP
with strong filters address range only
DNS Cache: udp/53
protect hosted servers from Internet using routers with Web server: tcp/80
strong filters
Other necessary filters: DNS Mail DNS Web
POP3 NEWS server
secondary Relay cache

All servers: SSH (tcp/22) from NOC LAN only

Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 75 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 76

ISP Infrastructure Security

Hosted Server Protection ISP Infrastructure Security

• premises security
Access-list examples: To core routers locks – electronic/card key preferred
Inbound secure access – 24x7 security arrangements
Allow tcp/established to all servers
Service Network
ICMP environment control – good aircon
Gateway Routers
Web server: tcp/80
SSH for customer access
• staff responsibility
Any other ports for services password policy, strangers, temp staff
sold to customers
employee exit procedures
Outbound • RFC2196
ICMP
Allow DNS udp/53 and (Site Security Handbook)
tcp/53
• RFC3871
Block all access to ISP Server1 Server2 Server3 Server4 Server5 Server6
address range (Operational Security Requirements for Large ISP IP
Network Infrastructure )
Cisco ISP Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 77 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 78

13
 ISP Network Security
ISP Network Security Secure external access

• Denial of Service Attacks • How to provide staff access from outside

eg: “smurfing” set up ssh gateway (Unix system with ssh daemon and
nothing else configured)
see http://www.denialinfo.com
provide ssh client on all staff laptops
• Effective filtering ssh available on Unix and Windows
network borders – see Cisco ISP Essentials ssh is Secure Shell – encrypted link
customer connections – unicast RPF • How not to provide access from outside
network operation centre
telnet, rsh, rlogin – these are all insecure
ISP corporate network – behind firewall
open host – insecure, can be compromised

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 79 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 80

Ingress & Egress Route Filtering

Your customers should not be

sending any IP packets out to the
Internet with a source address Out of Band Management
other then the address you have
allocated to them!

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 81 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 82

Out of Band Management Out of Band Management

• OoB Example – Access server:

• Not optional! modem attached to allow NOC dial in
• Allows access to network equipment in times of console ports of all network equipment connected to
failure serial ports

• Ensures quality of service to customers LAN and/or WAN link connects to network core, or via
separate management link to NOC
minimises downtime
• Full remote control access under all
minimises repair time
circumstances
eases diagnostics and debugging

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 83 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 84

14
Out of Band Network Out of Band Management
Equipment Rack Equipment Rack

Router, switch • OoB Example – Statistics gathering:

and ISP server
consoles Routers are NetFlow and syslog enabled
Management data is congestion/failure sensitive
(Optional) Out of band Ensures management data integrity in case of failure
WAN link to other PoPs

Modem – access
• Full remote information under all circumstances
to PSTN for out of
band dialin

Ethernet
Cisco ISP to the NOC Cisco ISP
Workshops © 2005, Cisco Systems, Inc. All rights reserved. 85 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 86

Test Laboratory

• Designed to look like a typical PoP

operated like a typical PoP
• Used to trial new services or new
Test Laboratory software under realistic conditions
• Allows discovery and fixing of potential
problems before they are introduced to
the network

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 87 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 88

Test Laboratory Test Laboratory

• Some ISPs dedicate equipment to the lab

• Can’t afford a test lab?
• Other ISPs “purchase ahead” so that
Set aside one spare router and server to trial new services
today’s lab equipment becomes
tomorrow’s PoP equipment Never ever try out new hardware, software or services on
the live network
• Other ISPs use lab equipment for “hot • Every major ISP in the US and Europe has a test lab
spares” in the event of hardware failure It’s a serious consideration

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 89 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 90

15
 Operational Considerations

Why design the world’s best network

Operational Considerations when you have not thought about what
operational good practices should be
implemented?

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 91 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 92

Operational Considerations Operational Considerations

Maintenance Support

• Never work on the live network, no matter how • Differentiate between customer support and the
trivial the modification may seem Network Operations Centre
Establish maintenance periods which your customers are Customer support fixes customer problems
aware of
NOC deals with and fixes backbone and Internet related
e.g. Tuesday 4-7am, Thursday 4-7am
problems
• Never do maintenance on a Friday • Network Engineering team is last resort
Unless you want to work all weekend cleaning up they design the next generation network, improve the
• Never do maintenance on a Monday routing design, implement new services, etc

Unless you want to work all weekend preparing they do not and should not be doing support!

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 93 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 94

Operational Considerations
NOC Communications

• NOC should know contact details for

equivalent NOCs in upstream providers
and peers
• Or consider joining the INOC-DBA system
Voice over IP phone system using SIP
ISP Network Design
Runs over the Internet
Summary
www.pch.net/inoc-dba for more information

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 95 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 96

16
ISP Design Summary

• KEEP IT SIMPLE & STUPID ! (KISS)

• Simple is elegant is scalable
• Use Redundancy, Security, and
Technology to make life easier for yourself ISP Network Design
ISP/IXP Workshops
• Above all, ensure quality of service for
your customers

Cisco ISP Cisco ISP

Workshops © 2005, Cisco Systems, Inc. All rights reserved. 97 Workshops © 2005, Cisco Systems, Inc. All rights reserved. 98

17