Monday, 30 April 2018

Liberalization in Uganda: A Looming Regression

On Thursday, April 26th, a concerned network operator forwarded us a copy of a survey they received from the national telecommunications regulator, the Uganda Communications Commission (UCC), which seeks input on how the UIXP should be governed, managed, and financially sustained.

We were not previously aware of this survey and are deeply concerned by its contents. Its text is inaccurate and misleading; its questions are poorly formed; and its broad dissemination indicates that the UCC is planning to attempt a top-down intervention in the UIXP's governance model and, by extension, Uganda's nascent network interconnection industry.

April 2018: Cover letter for UCC survey seeking input on the how the UIXP should be governed, managed, and financially sustained.

The UCC issued this survey with full awareness that the UIXP is actively and transparently working to advance its own governance reform process in collaboration with the network operator community; that the issues we need to address have not adversely impacted our growth and do not present an immediate risk to the continuity of our operations; and that the neutral not-for-profit governance model which the UIXP adopted in 2001 has become the most dominant and successful model in Africa by far.

The UCC's justification for issuing this survey stems from a letter sent to them by two members of our Internet community that libelously accused the UIXP management team, Google, and Akamai of fraud and impropriety. The UCC quickly and conclusively learned that these claims were false but continued to interject in our affairs under the guise of a neutral intermediary intent on resolving a community conflict.

Meanwhile, in the background, we see a rising frequency of controlling and economically damaging government interventions in the telecommunications space. Recent examples include a controversial move to reclaim and prop up the terminally ill state owned telecommunications company; a proposal to limit the number of international gateway providers; an explicit order to tax social media users in order to curtail unfavourable on-line discourse; and a directive to block all online news providers that have not been granted a national license.

The government has also made multiple attempts to nationalize or directly involve themselves in the UIXP's governance in the past, including a 2014 vote by Parliament (which was never implemented) and other less formal/ethical efforts to achieve the same. They also launched a failed attempt to nationalize the .ug ccTLD and have displayed an increasing tendency to censor or block telecommunications services (including mobile money) during presidential elections and politically sensitive events.

In this context, it is hard to trust that the UCC's interest in the UIXP's governance model is benevolent. The nature of their current involvement, and this survey in particular, raises the specter of nationalization and other forms of government intervention that could ultimately deter or prohibit competition in Uganda's emerging network interconnection industry -- an outcome that would have significant long term consequences for Uganda's telecommunications market and national economy.

2014: Parliament adopts an ICT committee recommendation to nationalize the UIXP based on false information and without engaging UIXP management in any way.

The African network interconnection industry is evolving rapidly. There are now 42 IXPs in 32 countries which carry over 400 Gbps of Internet traffic on a daily basis -- up from only 160 Mbps in 2008. South Africa, Nigeria, and Angola all now have more than one IXP operator, with other countries soon to follow. In tandem, we are starting to see the deployment of carrier neutral datacenters, large-scale content, and cloud services.

In Uganda, the UIXP now interconnects 28 networks and carries over 6 Gbps of Internet traffic on a daily basis -- up from only 10 Mbps in 2008. This has made our market much more attractive to international content providers and carrier neutral datacenter investors. If we continue along this path, there will soon be enough demand for local network services to justify the entrance of Uganda's first carrier neutral datacenter and second IXP operator. This would mark a milestone in Uganda's telecommunications history and likely herald a golden era of Internet connectivity.


UIXP: Chart showing connected networks, traffic growth, and key governance events over time.

However, expropriating our private not-for-profit company, and nationalizing our nascent industry in the process, is a great way to make sure that never happens. It would significantly increase Uganda's investment risk profile; prevent new IXPs from forming; and make carrier neutral datacenters -- which rely on network interconnection for growth -- far less viable. As a result, large content providers would need to look elsewhere for hosting (e.g. Kenya) which, in turn, would ensure that Uganda must continue to pay other countries for access, thereby keeping end-user prices high and service quality low.

In our view this is a very real possibility that threatens everyone's interests. We therefore call on all network operators and other recipients of the UCC survey to take this into account when considering if, and how, to respond.

We further call on all network operators to more actively participate in our own sustainability and governance reform process. It should be clear to all by now that the only good way forward is to work together.

Monday, 25 September 2017

UIXP Portal: Launch Announcement

Today we are proud to announce the launch of the UIXP Portal; an open-source central management system and customer portal. It’s taken a few weeks to set up and modify -- and there are still a few bugs -- but it should be a vast improvement over the collection of independent systems we had before.

The software, formally known as IXP Manager, was developed by the team at INEX (The Internet Neutral Exchange Association). They provided valuable assistance throughout our deployment by working with us to resolve bugs and implement feature requests. We plan to continue actively working with them to improve this software as we feel it has clear value to the global Internet exchange community.

The system can be accessed via the “Portal Login” link at the top-right corner of our website and directly via this URL: https://portal.uixp.co.ug

Here’s a brief list of features:

  • Centralized management of customers, switches, ports, and other core infrastructure.
  • Automated configuration of MRTG, Reverse DNS (PTR), Route Servers, and Nagios. These systems were previously managed manually, which was becoming increasingly time consuming as the exchange has grown.
  • Public aggregate statistics and Looking Glass functionality. This helps outside networks better evaluate the value of joining our exchange.
  • Support for multiple sites and peering fabrics, making future expansion easier.
  • Automated reporting and alerts for traffic anomalies, congestion, etc.

The system also provides a customer portal with:
  • Port data and statistics: bits, packets, errors, discards, and broadcasts
  • Technical and contact data for all networks at the exchange
  • A peering matrix which shows network interconnectivity based on route server data. Sflow capability will be added in the future to improve accuracy and capture bilateral peering sessions.
  • A peering manager that helps customers keep track of bilateral sessions.
  • Integrated mailing list subscription management.
  • The ability to add or delete additional customer user accounts.
  • The ability to update customer NOC contact and billing data.

If you are one of the networks connected to our exchange, here’s how to get started:
  1. Send us an e-mail address that we can associate with your organization’s “master” account. This account will *only* have the ability to create regular accounts for your company. Regular accounts created by this master account will have access to the full customer portal functionality described above.
  2. Send us a generic technical/NOC e-mail address (e.g. peering@domain.com) that we can add to our core contact database for your company.
Once the master account has created at least one regular account, please use it to:
  1. Update your company’s full NOC contact and billing data.
  2. Subscribe to our mailing lists by visiting the account profile page.
Please contact us to submit the above data or if you have any questions when getting started.

Friday, 30 December 2016

UIXP Network development report for 2016/17

UIXP is currently involved in an upgrade of its core network. This blog entry serves as a short technical report on the need for the change, the new network design as well as a current progress report.

Where is the exchange coming from?


Illustration 1: UIXP Network prior to the upgrade

In the past the network was built around a pair of HP ProCurve 3400CL switches. These switches offer 4 dual-personality ports - each port can be used as either an RJ-45 10/100/1000 copper port or an open mini-GBIC slot for fibre based transceivers plus 20 auto-sensing 10/100/1000 ports. The network was operated as a flat switched network with no separation of traffic types. Services on a Dell PowerEdge 750 Server were connected via a HP ProCurve 2524 100 Mb/s switch in the core which was interconnected with the peering switches via a 100 Mb/s CAT5e copper cable.

What is the motivation to upgrade?

With the addition of the Akamai Content Delivery Network (CDN) cache to the exchange and two Google caches located on member networks but accessible through the exchange it became necessary to re-look at network design as traffic levels rose significantly.

Physical network 

Illustration 2: UIXP - Physical layout
One of the limiting factors of the old model was the interconnect between the switches. This was a single physical IEEE 802.3z Type 1000Base-X giving a 1 Gb/s trunk. The distribution of members between the switches meant the Ethernet bundle came towards the limits of its bandwidth capacity.

The first thing necessary was a complete rebuild of the Core and Peering elements of the network and a separation of these functions to separate cabinets. A Juniper EX4300 was donated by the Uganda Communications Commission (UCC) as the peering access switch. With over 4 times the throughput of the HP ProCurve 3400CL switches, 4 Small Form Factor Pluggable plus (SFP+) ports that support IEEE 802.3ae Type 10GBASE-X as well as 48 port IEEE 802.3ab 1000Base-T for member peers. This switch is placed as a Top of Rack (ToR) switch in the Peering cabinet facilitating interconnection by the eXchange members as either 10, 100 or 1000 Mb/s. The Virtual Chassis configuration feature of the EX4300 is attractive given the potential to connect a second such switch in the future in the second peering cabinet.

The main core switch is a Cisco Nexus 3548 that was donated to the exchange by Packet Clearing House (PCH) with 48 fixed SFP+ ports IEEE 802.3ae Type 10GBASE-X, 10GBASE-CU SFP+ with Twinax Direct Attach Cables (DAC). Lower speeds are supported via Gigabit Line Card (GLC) SFPs for both Fibre and Copper 1 Gb/s interfaces configurable to lower speeds where necessary. This switch is interconnected to the Juniper peering switch with a 10 GB/s link configured as a Virtual Local Access Network (VLAN) trunk. The core switch has currently the HP ProCurve 2524 connected to cater for lower speed interfaces within the core network which reduces the number of SFPs necessary in the Cisco Nexus. This switch will be replaced by one of the HP ProCurve 3400CL switches once the members begin to migrate to the new peering cabinets.

The Akamai CDN Cache connects to the Core switch via a 10 Gb/s fibre interface while the Proxmox cluster nodes each connect via 1 Gb/s copper interfaces.

The old Cisco 3500 router has given way for a less power hungry Cisco C2801 router in the new core cabinet. As this routers function is to facility the distribution of traffic between the internal UIXP networks and the Internet, the bandwidth requirement is actually quite small and the C2801 is quite adequate for the function.

Infrastructure as a Service (IaaS) platform

To deliver core services it was necessary to build a robust Hypervisor based Infrastructure as a Service (IaaS) that could support the orchestration of both Virtual Machines (VM) and Containers (CT) to support the functions required at the eXchange.

The selection criteria for the hypervisor platform considered the need for it to be a Free and Open Source (FOSS) platform that supports High Availability (HA) as well as both VMs and CTs. The options explored were OpenStack and Proxmox. Both met the requirements of HA and IaaS. OpenStack is released under a FOSS Apache License, while Proxmox is licensed under the GNU is Not Unix (GNU) Affero General Public License (AGPL) version 3, so both are FOSS.

OpenStack however was considered more suitable for a Service Provider wishing to provide cloud services to end customers. This is not a requirement for the exchange and addes significant complexity. While the Proxmox Virtual Environment (VE) is not as fully featured as OpenStack it is powerful and simpler to deploy and use with all the features required by the eXchange.

Proxmox is Debian GNU/Linux based and uses robust Kernel Virtual Machine (KVM) technology and LinuX Containers (LXC). A major plus of Proxmox is the HA Cluster features. When VM or CT instances are configured as HA and the physical host fails, the virtual instance is automatically restarted on the remaining Proxmox VE Cluster nodes. It was considered that the Proxmox VE HA Cluster is based on proven GNU/Linux HA technologies and would provide the stable and reliable HA service required.

Initially the Proxmox cluster consists of the Dell PowerEdge 750 and an old Dell Server, however thanks to a upcoming donation from the Internet Society of an additional Dell PowerEdge 750 it will be possible to upgrade the Proxmox cluster hardware. This VE cluster is an essential element of the exchange and hosts the various Virtual Network Functions (VNF) and Server instances as either VMs or CTs.

Logical network 

Illustration 3: UIXP Logical network design

Considering a number of items, the need to separate traffic types and information/network security to name but a few it was decided to split the network into logical elements, a peering Local Access Network (LAN) to contain the member peering interfaces as well as the Root Servers (RS) and the Autonomous System 112 (AS112) Nameserver. A private management LAN for intercommunication between the functions and a DeMilitarised Zone (DMZ) LAN to permit controlled access to the various networking devices, VMs and CTs.

Current state

Well most of the physical network elements are already in place and we await the migration of the peers to the new peering cabinets. The Proxmox cluster is in place and will be beefed up by the addition of the second Dell PowerEdge 750 and it supports the core services that are built on VMs and containers. Once that is complete the work of separating the LAN into the logical elements just described will begin. Looking forward to a busy 2017.

Abbreviations

AGPL Affero General Public License
AS112 Autonomous System 112
CDN Content Delivery Network
CT Containers
DMZ DeMilitarised Zone
FOSS Free and Open Source
GLC Gigabit Line Card
GNU GNU is Not Unix
HA High Availability
IaaS Infrastructure as a Service
KVM Kernel Virtual Machine
LAN Local Access Network
LXC LinuX Containers
PCH Packet Clearing House
RS Root Servers
SFP+ Small Form Factor Pluggable plus
ToR Top of Rack
UCC Uganda Communications Commission
VE Virtual Environment
VLAN Virtual Local Access Network
VM Virtual Machines
VNF Virtual Network Functions

Bibliography

Packet Clearing House. Available: https://www.pch.net
Uganda Communications Commission. Available: http://www.ucc.co.ug
The Internet Society. Available: http://www.internetsociety.org
Proxmox Server Solutions GmbH. Available: https://www.proxmox.com/en/
Akamai Technologies. Available: https://www.akamai.com
Cisco Nexus 3548 Switch. Available: http://www.cisco.com/c/en/us/products/switches/nexus-3548-switch
Juniper EX4300 Switch. Available: http://www.juniper.net/uk/en/products-services/switching/ex-series/ex4300
HP ProCurve 3400CL. Available: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c01809608
HP ProCurve 2524. Available: http://www.hp.com/ecomcat/hpcatalog/specs/J4813A.htm

Dell PowerEdge 750. Available: http://www.dell.com/downloads/global/products/pedge/en/750_specs.pdf

Diarmuid O'Briain joins our technical operations team

In November 2016, Diarmuid Ó'Briain, a lecturer at Makerere University, joined our team as Technical Operations Manager in order to help us address a significant human resource shortage which arose as a result of increased demand for our services.

Diarmuid is a Chartered Engineer with over 25 years experience in the sector. He previously worked as an engineering manager at multinational networking and telecommunications companies including US Robotics, 3Com, and UTStarcom where he designed and implemented next-generation networks and information security solutions for several clients, and was instrumental in the design and support for NGN solutions for carriers around the EMEA region.

His most recent role prior to arriving in Uganda in 2015 was Chief Technology Officer at Ripple Communications, an Irish Internet service provider, where he developed the core network and managed the design and delivery of fibre and licensed radio back-haul infrastructure for their national-scale access network.

Today he teaches Networking, Information Security, and Wireless Technologies at the Makerere College of Engineering, Design, Art, and Technology and is an active member of the Uganda Institute of Professional Engineers (UIPE).

Thursday, 29 December 2016

Welcome to the Uganda Internet eXchange Point blog

Since 2001, the Uganda Internet eXchange Point (UIXP) has been working to make the Internet in Uganda cheaper, faster, and more reliable.

This volunteer-led initiative, which began life as an esoteric engineering project, has since grown to become a core component of the national Internet ecosystem with the potential to become a new and competitive domestic industry. However, despite this success, our work remains largely invisible and not well understood.

The launch of this blog is an effort to address these shortcomings; to increase our public presence, provide insight into our work, and dispel common misconceptions. We also aim to publish tutorials relevant to developing IXPs, feature content from select external contributors, and share our thoughts on the Internet in Uganda.

We hope that you find it useful.