Tuesday, 25 June 2019

Nationalization in Uganda: A Looming Disaster

The Uganda Communications Commission (UCC) has proposed a new licensing framework that would effectively nationalize a core part of the country's telecommunications industry; Internet exchange points (IXPs).



Introduction to Internet Exchange Points (IXPs)

This news was communicated to us in a letter from the UCC dated June 7th, 2019. The letter included a draft of the licensing framework and a call for written feedback by July 5th, 2019 (click here to download a full copy).

Based on our analysis, the draft framework would establish a government controlled monopoly which all other market players would be subservient to. It would accomplish this with the following formula:

  • Establish a “Designated National Internet Exchange Point” that all other IXPs will be required to connect to (9.j);

  • Require government approval of contracts between IXPs and network operators (7.4.b);

  • Allow the government to arbitrarily compel IXPs to make operational and technical changes (7.5.c)

  • Allow the government to inspect, copy, or remove any data related to any IXP without a court order (7.5.b.i);

  • Require all licensed network operators to connect to an IXP (8.2.a).

Technically speaking, this policy would merge all IXPs into a single national peering LAN, with each IXP merely acting as a heavily regulated access point for the enlarged infrastructure. The resulting entity would suffer from all of the classic symptoms of a monopoly as well as significant technical challenges and security risks derived from having multiple operators control access points that form part of the same LAN. As a centralized service, it would also inherently lack the resilience that a diverse array of independent IXPs would provide.

Here are two diagrams to help illustrate the concept:




In addition, the draft framework contains language which suggests that the government intends for the "Designated National Internet Exchange Point" to establish itself by expropriating an existing private operation; namely, ours.

This appears to confirm some of our worst fears about Uganda's new National Broadband Policy; a government strategy document, reportedly drafted in isolation, that seemingly calls for a large-scale nationalization and centralization of Uganda's Internet infrastructure under the guise of infrastructure sharing.

We (and others) have repeatedly warned that such policies would have severe socioeconomic consequences for Uganda and the wider East African region. In this particular case, the UCC's planned regulatory intervention in our otherwise healthy industry has no successful parallel anywhere in the world -- and global experts widely regard the other attempts as textbook examples of regulatory failure.

In light of the obvious risks and highly technical nature of this proposal, we feel that this limited survey of the local Internet community is insufficient validation. Accordingly, we strongly urge the UCC to defer any further work on this project until there is a clear rationale and its viability can be transparently proven with case studies and corroborating input from credible global experts.

Meanwhile, we are preparing to submit detailed feedback to the UCC and will post a copy of our submission here once it is ready. We encourage anyone else that would like to submit feedback to do so through us electronically. We will collect, manually submit, and (unless anyone objects) electronically publish all that we receive in order to promote transparency.

Please feel free to contact us here: board@uixp.co.ug

[UPDATE: A copy of our formal feedback to the UCC can be downloaded here. Our general position is that the creation of a de jure IXP monopoly would be bad for our industry, Uganda, and the region. We argue that the regulatory framework should instead seek to create an enabling environment for competition.

We have also uploaded a number of supporting submissions made by the Internet eXchange Federation (IX-F), the African Network Information Centre (AFRINIC), the ICT Association of Uganda (ICTAU), Liquid Telecom, and a personal submission by Diarmuid O'Briain. A copy of those submissions can be downloaded here.]

Monday, 7 January 2019

Annual Update: Sustainability, Stability, and Growth

This is an annual update regarding the UIXP’s progress in 2018 and our ambitions for 2019.

2018 was a surprisingly good year: We overcame substantial challenges, attracted new peers, deployed a prototype Google cache, implemented a new sustainability model, upgraded our failing power system, and paid our legal debts. As a result, we are now significantly better positioned for future growth and, therefore, to deliver significantly more value to our members.


OLD POWER SYSTEM
NEW POWER SYSTEM


In 2019 we plan to build on these successes by lowering prices; developing internal structure; paying key staff; improving service quality; hosting quarterly events; supporting the local technology community; becoming fully tax compliant; and implementing governance reform.

The new pricing structure aims to attract more networks by making peering more affordable: We now offer 10 Mbps ports for free; have cut the cost of 100 Mbps ports by 60%; and have reduced the cost of 1 Gbps ports by 9%. These adjustments were possible to implement without negatively impacting our overall revenue because of growth in our paying membership base, and because many networks have transitioned to (or will soon transition to) more expensive 10 Gbps ports where our existing rates are still cost-effective.


PORT CAPACITY
2018 MRC
2019 MRC
10 Mbps
$100 / mo
FREE
100 Mbps
$250 / mo
$100 / mo
1 Gbps
$550 / mo
$500 / mo
10 Gbps
$1000 / mo
$1000 / mo


We are also excited to announce the impending arrival of a large social media network in Q1 2019. This network will peer directly and should significantly increase the amount of traffic networks generate from our exchange. We are still working out some of the technical details and will share more on this soon.

Finally, we would like to note that none of this would have been possible without the networks that supported the implementation of our sustainability model -- and those that have committed to do so in 2019. To these networks we are extremely grateful. We are heartened by your support.

We also thank everyone else for their participation and look forward to interacting with all of you in the coming year!

Monday, 30 April 2018

Liberalization in Uganda: A Looming Regression

On Thursday, April 26th, a concerned network operator forwarded us a copy of a survey they received from the national telecommunications regulator, the Uganda Communications Commission (UCC), which seeks input on how the UIXP should be governed, managed, and financially sustained.

We were not previously aware of this survey and are deeply concerned by its contents. Its text is inaccurate and misleading; its questions are poorly formed; and its broad dissemination indicates that the UCC is planning to attempt a top-down intervention in the UIXP's governance model and, by extension, Uganda's nascent network interconnection industry.

April 2018: Cover letter for UCC survey seeking input on the how the UIXP should be governed, managed, and financially sustained.

The UCC issued this survey with full awareness that the UIXP is actively and transparently working to advance its own governance reform process in collaboration with the network operator community; that the issues we need to address have not adversely impacted our growth and do not present an immediate risk to the continuity of our operations; and that the neutral non-profit governance model which the UIXP adopted in 2001 has become one of the most dominant and successful in Africa.

The UCC's justification for issuing this survey stems from a letter sent to them by two members of our Internet community that libelously accused the UIXP management team, Google, and Akamai of fraud and impropriety. The UCC quickly and conclusively learned that these claims were false but continued to interject in our affairs under the guise of a neutral intermediary intent on resolving a community conflict.

Meanwhile, in the background, we see a rising frequency of controlling and economically damaging government interventions in the telecommunications space. Recent examples include a controversial move to reclaim and prop up the terminally ill state owned telecommunications company; a proposal to limit the number of international gateway providers; an explicit order to tax social media users in order to curtail unfavourable on-line discourse; and a directive to block all online news providers that have not been granted a national license.

The government has also made multiple attempts to nationalize or directly involve themselves in the UIXP's governance in the past, including a 2014 vote by Parliament (which was never implemented) and other less formal/ethical efforts to achieve the same. They also launched a failed attempt to nationalize the .ug ccTLD and have displayed an increasing tendency to censor or block telecommunications services (including mobile money) during presidential elections and politically sensitive events.

In this context, it is hard to trust that the UCC's interest in the UIXP's governance model is benevolent. The nature of their current involvement, and this survey in particular, raises the specter of nationalization and other forms of government intervention that could ultimately deter or prohibit competition in Uganda's emerging network interconnection industry -- an outcome that would have significant long term consequences for Uganda's telecommunications market and national economy.

2014: Parliament adopts an ICT committee recommendation to nationalize the UIXP based on false information and without engaging UIXP management in any way.

The African network interconnection industry is evolving rapidly. There are now 42 IXPs in 32 countries which carry over 400 Gbps of Internet traffic on a daily basis -- up from only 160 Mbps in 2008. South Africa, Nigeria, and Angola all now have more than one IXP operator, with other countries soon to follow. In tandem, we are starting to see the deployment of carrier neutral datacenters, large-scale content, and cloud services.

In Uganda, the UIXP now interconnects 28 networks and carries over 6 Gbps of Internet traffic on a daily basis -- up from only 10 Mbps in 2008. This has made our market much more attractive to international content providers and carrier neutral datacenter investors. If we continue along this path, there will soon be enough demand for local network services to justify the entrance of Uganda's first carrier neutral datacenter and second IXP operator. This would mark a milestone in Uganda's telecommunications history and likely herald a golden era of Internet connectivity.


UIXP: Chart showing connected networks, traffic growth, and key governance events over time.

However, expropriating our private not-for-profit company, and nationalizing our nascent industry in the process, is a great way to make sure that never happens. It would significantly increase Uganda's investment risk profile; prevent new IXPs from forming; and make carrier neutral datacenters -- which rely on network interconnection for growth -- far less viable. As a result, large content providers would need to look elsewhere for hosting (e.g. Kenya) which, in turn, would ensure that Uganda must continue to pay other countries for access, thereby keeping end-user prices high and service quality low.

In our view this is a very real possibility that threatens everyone's interests. We therefore call on all network operators and other recipients of the UCC survey to take this into account when considering if, and how, to respond.

We further call on all network operators to more actively participate in our own sustainability and governance reform process. It should be clear to all by now that the only good way forward is to work together.

Monday, 25 September 2017

UIXP Portal: Launch Announcement

Today we are proud to announce the launch of the UIXP Portal; an open-source central management system and customer portal. It’s taken a few weeks to set up and modify -- and there are still a few bugs -- but it should be a vast improvement over the collection of independent systems we had before.

The software, formally known as IXP Manager, was developed by the team at INEX (The Internet Neutral Exchange Association). They provided valuable assistance throughout our deployment by working with us to resolve bugs and implement feature requests. We plan to continue actively working with them to improve this software as we feel it has clear value to the global Internet exchange community.

The system can be accessed via the “Portal Login” link at the top-right corner of our website and directly via this URL: https://portal.uixp.co.ug

Here’s a brief list of features:

  • Centralized management of customers, switches, ports, and other core infrastructure.
  • Automated configuration of MRTG, Reverse DNS (PTR), Route Servers, and Nagios. These systems were previously managed manually, which was becoming increasingly time consuming as the exchange has grown.
  • Public aggregate statistics and Looking Glass functionality. This helps outside networks better evaluate the value of joining our exchange.
  • Support for multiple sites and peering fabrics, making future expansion easier.
  • Automated reporting and alerts for traffic anomalies, congestion, etc.

The system also provides a customer portal with:
  • Port data and statistics: bits, packets, errors, discards, and broadcasts
  • Technical and contact data for all networks at the exchange
  • A peering matrix which shows network interconnectivity based on route server data. Sflow capability will be added in the future to improve accuracy and capture bilateral peering sessions.
  • A peering manager that helps customers keep track of bilateral sessions.
  • Integrated mailing list subscription management.
  • The ability to add or delete additional customer user accounts.
  • The ability to update customer NOC contact and billing data.

If you are one of the networks connected to our exchange, here’s how to get started:
  1. Send us an e-mail address that we can associate with your organization’s “master” account. This account will *only* have the ability to create regular accounts for your company. Regular accounts created by this master account will have access to the full customer portal functionality described above.
  2. Send us a generic technical/NOC e-mail address (e.g. peering@domain.com) that we can add to our core contact database for your company.
Once the master account has created at least one regular account, please use it to:
  1. Update your company’s full NOC contact and billing data.
  2. Subscribe to our mailing lists by visiting the account profile page.
Please contact us to submit the above data or if you have any questions when getting started.

Friday, 30 December 2016

UIXP Network development report for 2016/17

UIXP is currently involved in an upgrade of its core network. This blog entry serves as a short technical report on the need for the change, the new network design as well as a current progress report.

Where is the exchange coming from?


Illustration 1: UIXP Network prior to the upgrade

In the past the network was built around a pair of HP ProCurve 3400CL switches. These switches offer 4 dual-personality ports - each port can be used as either an RJ-45 10/100/1000 copper port or an open mini-GBIC slot for fibre based transceivers plus 20 auto-sensing 10/100/1000 ports. The network was operated as a flat switched network with no separation of traffic types. Services on a Dell PowerEdge 750 Server were connected via a HP ProCurve 2524 100 Mb/s switch in the core which was interconnected with the peering switches via a 100 Mb/s CAT5e copper cable.

What is the motivation to upgrade?

With the addition of the Akamai Content Delivery Network (CDN) cache to the exchange and two Google caches located on member networks but accessible through the exchange it became necessary to re-look at network design as traffic levels rose significantly.

Physical network 

Illustration 2: UIXP - Physical layout
One of the limiting factors of the old model was the interconnect between the switches. This was a single physical IEEE 802.3z Type 1000Base-X giving a 1 Gb/s trunk. The distribution of members between the switches meant the Ethernet bundle came towards the limits of its bandwidth capacity.

The first thing necessary was a complete rebuild of the Core and Peering elements of the network and a separation of these functions to separate cabinets. A Juniper EX4300 was donated by the Uganda Communications Commission (UCC) as the peering access switch. With over 4 times the throughput of the HP ProCurve 3400CL switches, 4 Small Form Factor Pluggable plus (SFP+) ports that support IEEE 802.3ae Type 10GBASE-X as well as 48 port IEEE 802.3ab 1000Base-T for member peers. This switch is placed as a Top of Rack (ToR) switch in the Peering cabinet facilitating interconnection by the eXchange members as either 10, 100 or 1000 Mb/s. The Virtual Chassis configuration feature of the EX4300 is attractive given the potential to connect a second such switch in the future in the second peering cabinet.

The main core switch is a Cisco Nexus 3548 that was donated to the exchange by Packet Clearing House (PCH) with 48 fixed SFP+ ports IEEE 802.3ae Type 10GBASE-X, 10GBASE-CU SFP+ with Twinax Direct Attach Cables (DAC). Lower speeds are supported via Gigabit Line Card (GLC) SFPs for both Fibre and Copper 1 Gb/s interfaces configurable to lower speeds where necessary. This switch is interconnected to the Juniper peering switch with a 10 GB/s link configured as a Virtual Local Access Network (VLAN) trunk. The core switch has currently the HP ProCurve 2524 connected to cater for lower speed interfaces within the core network which reduces the number of SFPs necessary in the Cisco Nexus. This switch will be replaced by one of the HP ProCurve 3400CL switches once the members begin to migrate to the new peering cabinets.

The Akamai CDN Cache connects to the Core switch via a 10 Gb/s fibre interface while the Proxmox cluster nodes each connect via 1 Gb/s copper interfaces.

The old Cisco 3500 router has given way for a less power hungry Cisco C2801 router in the new core cabinet. As this routers function is to facility the distribution of traffic between the internal UIXP networks and the Internet, the bandwidth requirement is actually quite small and the C2801 is quite adequate for the function.

Infrastructure as a Service (IaaS) platform

To deliver core services it was necessary to build a robust Hypervisor based Infrastructure as a Service (IaaS) that could support the orchestration of both Virtual Machines (VM) and Containers (CT) to support the functions required at the eXchange.

The selection criteria for the hypervisor platform considered the need for it to be a Free and Open Source (FOSS) platform that supports High Availability (HA) as well as both VMs and CTs. The options explored were OpenStack and Proxmox. Both met the requirements of HA and IaaS. OpenStack is released under a FOSS Apache License, while Proxmox is licensed under the GNU is Not Unix (GNU) Affero General Public License (AGPL) version 3, so both are FOSS.

OpenStack however was considered more suitable for a Service Provider wishing to provide cloud services to end customers. This is not a requirement for the exchange and addes significant complexity. While the Proxmox Virtual Environment (VE) is not as fully featured as OpenStack it is powerful and simpler to deploy and use with all the features required by the eXchange.

Proxmox is Debian GNU/Linux based and uses robust Kernel Virtual Machine (KVM) technology and LinuX Containers (LXC). A major plus of Proxmox is the HA Cluster features. When VM or CT instances are configured as HA and the physical host fails, the virtual instance is automatically restarted on the remaining Proxmox VE Cluster nodes. It was considered that the Proxmox VE HA Cluster is based on proven GNU/Linux HA technologies and would provide the stable and reliable HA service required.

Initially the Proxmox cluster consists of the Dell PowerEdge 750 and an old Dell Server, however thanks to a upcoming donation from the Internet Society of an additional Dell PowerEdge 750 it will be possible to upgrade the Proxmox cluster hardware. This VE cluster is an essential element of the exchange and hosts the various Virtual Network Functions (VNF) and Server instances as either VMs or CTs.

Logical network 

Illustration 3: UIXP Logical network design

Considering a number of items, the need to separate traffic types and information/network security to name but a few it was decided to split the network into logical elements, a peering Local Access Network (LAN) to contain the member peering interfaces as well as the Root Servers (RS) and the Autonomous System 112 (AS112) Nameserver. A private management LAN for intercommunication between the functions and a DeMilitarised Zone (DMZ) LAN to permit controlled access to the various networking devices, VMs and CTs.

Current state

Well most of the physical network elements are already in place and we await the migration of the peers to the new peering cabinets. The Proxmox cluster is in place and will be beefed up by the addition of the second Dell PowerEdge 750 and it supports the core services that are built on VMs and containers. Once that is complete the work of separating the LAN into the logical elements just described will begin. Looking forward to a busy 2017.

Abbreviations

AGPL Affero General Public License
AS112 Autonomous System 112
CDN Content Delivery Network
CT Containers
DMZ DeMilitarised Zone
FOSS Free and Open Source
GLC Gigabit Line Card
GNU GNU is Not Unix
HA High Availability
IaaS Infrastructure as a Service
KVM Kernel Virtual Machine
LAN Local Access Network
LXC LinuX Containers
PCH Packet Clearing House
RS Root Servers
SFP+ Small Form Factor Pluggable plus
ToR Top of Rack
UCC Uganda Communications Commission
VE Virtual Environment
VLAN Virtual Local Access Network
VM Virtual Machines
VNF Virtual Network Functions

Bibliography

Packet Clearing House. Available: https://www.pch.net
Uganda Communications Commission. Available: http://www.ucc.co.ug
The Internet Society. Available: http://www.internetsociety.org
Proxmox Server Solutions GmbH. Available: https://www.proxmox.com/en/
Akamai Technologies. Available: https://www.akamai.com
Cisco Nexus 3548 Switch. Available: http://www.cisco.com/c/en/us/products/switches/nexus-3548-switch
Juniper EX4300 Switch. Available: http://www.juniper.net/uk/en/products-services/switching/ex-series/ex4300
HP ProCurve 3400CL. Available: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c01809608
HP ProCurve 2524. Available: http://www.hp.com/ecomcat/hpcatalog/specs/J4813A.htm

Dell PowerEdge 750. Available: http://www.dell.com/downloads/global/products/pedge/en/750_specs.pdf

Diarmuid O'Briain joins our technical operations team

In November 2016, Diarmuid Ó'Briain, a lecturer at Makerere University, joined our team as Technical Operations Manager in order to help us address a significant human resource shortage which arose as a result of increased demand for our services.

Diarmuid is a Chartered Engineer with over 25 years experience in the sector. He previously worked as an engineering manager at multinational networking and telecommunications companies including US Robotics, 3Com, and UTStarcom where he designed and implemented next-generation networks and information security solutions for several clients, and was instrumental in the design and support for NGN solutions for carriers around the EMEA region.

His most recent role prior to arriving in Uganda in 2015 was Chief Technology Officer at Ripple Communications, an Irish Internet service provider, where he developed the core network and managed the design and delivery of fibre and licensed radio back-haul infrastructure for their national-scale access network.

Today he teaches Networking, Information Security, and Wireless Technologies at the Makerere College of Engineering, Design, Art, and Technology and is an active member of the Uganda Institute of Professional Engineers (UIPE).

Thursday, 29 December 2016

Welcome to the Uganda Internet eXchange Point blog

Since 2001, the Uganda Internet eXchange Point (UIXP) has been working to make the Internet in Uganda cheaper, faster, and more reliable.

This volunteer-led initiative, which began life as an esoteric engineering project, has since grown to become a core component of the national Internet ecosystem with the potential to become a new and competitive domestic industry. However, despite this success, our work remains largely invisible and not well understood.

The launch of this blog is an effort to address these shortcomings; to increase our public presence, provide insight into our work, and dispel common misconceptions. We also aim to publish tutorials relevant to developing IXPs, feature content from select external contributors, and share our thoughts on the Internet in Uganda.

We hope that you find it useful.