UIXP is currently involved in an upgrade of its
core network. This blog entry serves as a short technical report on the need for the change, the new network design as well as a current progress
report.
Where is the exchange coming from?
Illustration
1: UIXP Network prior to the upgrade
In the past the network was built around a pair of
HP ProCurve 3400CL switches. These switches offer 4 dual-personality
ports - each port can be used as either an RJ-45 10/100/1000 copper
port or an open mini-GBIC slot for fibre based transceivers plus 20
auto-sensing 10/100/1000 ports. The network was operated as a flat
switched network with no separation of traffic types. Services on a
Dell PowerEdge 750 Server were connected via a HP ProCurve 2524 100
Mb/s switch in the core which was interconnected with the peering
switches via a 100 Mb/s CAT5e copper cable.
What is the motivation to upgrade?
With the addition of the Akamai Content Delivery
Network (CDN) cache to the exchange and two Google caches located on
member networks but accessible through the exchange it became
necessary to re-look at network design as traffic levels rose
significantly.
Physical network
Illustration
2: UIXP - Physical layout
One of the limiting factors of the old model was
the interconnect between the switches. This was a single physical
IEEE 802.3z Type 1000Base-X giving a 1 Gb/s trunk. The distribution
of members between the switches meant the Ethernet bundle came
towards the limits of its bandwidth capacity.
The first thing necessary was a complete rebuild
of the Core and Peering elements of the network and a separation of
these functions to separate cabinets. A Juniper EX4300 was donated by
the Uganda Communications Commission (UCC) as the peering access
switch. With over 4 times the throughput of the HP ProCurve 3400CL
switches, 4 Small Form Factor Pluggable plus (SFP+) ports that
support IEEE 802.3ae Type 10GBASE-X as well as 48 port IEEE 802.3ab
1000Base-T for member peers. This switch is placed as a Top of Rack
(ToR) switch in the Peering cabinet facilitating interconnection by
the eXchange members as either 10, 100 or 1000 Mb/s. The Virtual
Chassis configuration feature of the EX4300 is attractive given the
potential to connect a second such switch in the future in the second
peering cabinet.
The main core switch is a Cisco Nexus 3548 that was donated to the
exchange by Packet Clearing House (PCH) with 48 fixed
SFP+ ports IEEE 802.3ae Type 10GBASE-X, 10GBASE-CU SFP+ with Twinax
Direct Attach Cables (DAC). Lower speeds are supported via Gigabit
Line Card (GLC) SFPs for both Fibre and Copper 1 Gb/s interfaces
configurable to lower speeds where necessary. This switch is
interconnected to the Juniper peering switch with a 10 GB/s link
configured as a Virtual Local Access Network (VLAN) trunk. The core
switch has currently the HP ProCurve 2524 connected to cater for
lower speed interfaces within the core network which reduces the
number of SFPs necessary in the Cisco Nexus. This switch will be
replaced by one of the HP ProCurve 3400CL switches once the members
begin to migrate to the new peering cabinets.
The Akamai CDN Cache connects to the Core switch
via a 10 Gb/s fibre interface while the Proxmox cluster nodes each
connect via 1 Gb/s copper interfaces.
The old Cisco 3500 router has given way for a less
power hungry Cisco C2801 router in the new core cabinet. As this
routers function is to facility the distribution of traffic between
the internal UIXP networks and the Internet, the bandwidth
requirement is actually quite small and the C2801 is quite adequate
for the function.
Infrastructure as a Service (IaaS) platform
To deliver core services it was necessary to build
a robust Hypervisor based Infrastructure as a Service (IaaS) that
could support the orchestration of both Virtual Machines (VM) and
Containers (CT) to support the functions required at the eXchange.
The selection criteria for the hypervisor platform
considered the need for it to be a Free and Open Source (FOSS)
platform that supports High Availability (HA) as well as both VMs and
CTs. The options explored were OpenStack and Proxmox. Both met the
requirements of HA and IaaS. OpenStack is released under a FOSS
Apache License, while Proxmox is licensed under the GNU is Not Unix
(GNU) Affero General Public License (AGPL) version 3, so both are
FOSS.
OpenStack however was considered more suitable for
a Service Provider wishing to provide cloud services to end
customers. This is not a requirement for the exchange and addes
significant complexity. While the Proxmox Virtual Environment (VE) is
not as fully featured as OpenStack it is powerful and simpler to
deploy and use with all the features required by the eXchange.
Proxmox is Debian GNU/Linux based and uses robust
Kernel Virtual Machine (KVM) technology and LinuX Containers (LXC). A
major plus of Proxmox is the HA Cluster features. When VM or CT
instances are configured as HA and the physical host fails, the
virtual instance is automatically restarted on the remaining Proxmox
VE Cluster nodes. It was considered that the Proxmox VE HA Cluster is
based on proven GNU/Linux HA technologies and would provide the
stable and reliable HA service required.
Initially the Proxmox cluster consists of the Dell
PowerEdge 750 and an old Dell Server, however thanks to a upcoming donation from the Internet Society of an additional Dell PowerEdge 750 it will be
possible to upgrade the Proxmox cluster hardware. This VE cluster is an essential element of the exchange and hosts the various
Virtual Network Functions (VNF) and Server instances as either VMs or
CTs.
Logical network
Illustration
3: UIXP Logical network design
Considering a number of items, the need to
separate traffic types and information/network security to name but a few it was decided
to split the network into logical elements, a peering Local Access
Network (LAN) to contain the member peering interfaces as well as the
Root Servers (RS) and the Autonomous System 112 (AS112) Nameserver. A
private management LAN for intercommunication between the functions
and a DeMilitarised Zone (DMZ) LAN to permit controlled access to the
various networking devices, VMs and CTs.
Current state
Well most of the physical network elements are
already in place and we await the migration of the peers to the new
peering cabinets. The Proxmox cluster is in place and will be beefed
up by the addition of the second Dell PowerEdge 750 and it supports
the core services that are built on VMs and containers. Once that is
complete the work of separating the LAN into the logical elements
just described will begin. Looking forward to a busy 2017.
Abbreviations
| AGPL | Affero General Public License |
| AS112 | Autonomous System 112 |
| CDN | Content Delivery Network |
| CT | Containers |
| DMZ | DeMilitarised Zone |
| FOSS | Free and Open Source |
| GLC | Gigabit Line Card |
| GNU | GNU is Not Unix |
| HA | High Availability |
| IaaS | Infrastructure as a Service |
| KVM | Kernel Virtual Machine |
| LAN | Local Access Network |
| LXC | LinuX Containers |
| PCH | Packet Clearing House |
| RS | Root Servers |
| SFP+ | Small Form Factor Pluggable plus |
| ToR | Top of Rack |
| UCC | Uganda Communications Commission |
| VE | Virtual Environment |
| VLAN | Virtual Local Access Network |
| VM | Virtual Machines |
| VNF | Virtual Network Functions |
Bibliography
Packet Clearing House. Available: https://www.pch.netUganda Communications Commission. Available: http://www.ucc.co.ug
The Internet Society. Available: http://www.internetsociety.org
Proxmox Server Solutions GmbH. Available: https://www.proxmox.com/en/
Akamai Technologies. Available: https://www.akamai.com
Cisco Nexus 3548 Switch. Available: http://www.cisco.com/c/en/us/products/switches/nexus-3548-switch
Juniper EX4300 Switch. Available: http://www.juniper.net/uk/en/products-services/switching/ex-series/ex4300
HP ProCurve 3400CL. Available: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c01809608
HP ProCurve 2524. Available: http://www.hp.com/ecomcat/hpcatalog/specs/J4813A.htm
Dell PowerEdge 750. Available: http://www.dell.com/downloads/global/products/pedge/en/750_specs.pdf
