New Networks, New Team Members, Healthy Finances, and Improved Stability

This is a report on the UIXP’s status, progress, and other activities that took place during Q2 2019. We typically only send quarterly updates to the networks which use our infrastructure, but lot has happened in the last three months so we thought it would be prudent to share this one publicly.

27 networks currently exchange 8Gbps of peak daily traffic

New Members
We’re happy to announce that Facebook (AS63293) and Blue Crane Communications (AS328198) are now connected to the UIXP, bringing our total membership to 27 networks. Blue Crane is peering multilaterally via the route servers, but Facebook is not. If you would like to connect to their cache, please send us a message and we will facilitate an introduction. We may also reach out to you directly in order to encourage the same.

Staffing Changes

We’re sad to announce the looming departure of Diarmuid O’Briain as our Technical Operations Manager. Diarmuid has a relentless passion for his craft, and his contributions to the UIXP were both numerous and significant. His legacy will remain, though he will be missed, and we wish him well in all of his future endeavours.

Diarmuid is survived by Brian Masiga and Conrad Ekisa, the newest member of our engineering team. Conrad has a BSc in Telecommunications Engineering and was employed as a Graduate Research Assistant (GRA) at netLabs!UG Research Centre last year. During this time, Conrad qualified as Uganda's first MikroTik Certified consultant and academy trainer. In 2019 he was promoted to Graduate Researcher and led the internship programme for the year. Together with Brian, he will help manage our infrastructure and handle your technical support requests. Please give him a warm welcome!

Diarmuid O'Briain

Conrad Ekisa

Collections

We’re happy to report a substantial increase in the number of paying networks, and that payments have become more consistent overall. However, there are still a few legacy networks which have not joined our sustainability programme despite our best efforts to encourage them, and we will soon start migrating these networks to our free service tier in the interest of fairness.

Cash Reserves

If all goes well, we will have established cash reserves of USD $75,000 by the end of this year thanks, in part, to a generous three year pre-payment by one of our new members. We plan to sustain that amount or more indefinitely.

Salaries
We have started to pay our two IXP Engineers a $100/mo volunteer stipend in order to cover the various expenses associated with their work. These stipends are paid quarterly in order to minimize our administrative overhead. Note that we have not yet implemented remuneration for any other positions in order to prioritize other expenses and the development of cash reserves.

Taxes
We continue to remain in good-standing with URA thanks to support from our tax consultant. In light of his consistently good performance, we have started to pay his fees on a quarterly basis in order to minimize our administrative overhead.

Banking
We still do not have a UGX bank account because opening one would incur an additional monthly charge (in addition to our USD account which costs approximately $12/mo). We currently do few UGX transactions per month beyond URA payments and occasional cash purchases, so it’s cheaper for us to convert USD to UGX when these needs arise. If the volume of UGX transactions increases in the future, we will reconsider our options.

Accounting
We are recording all income and expenses in our accounting system, and we digitally archive all cheques and receipts for auditing purposes. However, we have not been logging URA payments in our accounting system as it’s not a straightforward process. As a result, the account balance in our system does not match the amount in our actual bank account. We are working to figure out the best way to resolve this issue systematically going forward.

Power Infrastructure
Our new inverter system is still working well. We have experienced no related downtime since its deployment, save for a building outage that exceeded the 12 hour runtime of our battery bank.

Air Conditioning
Our air conditioning system has been keeping things cool since we took over its management from the UCC following a series of extended outages. However, there have been a few reliability issues with one of the two new units we purchased which we’re still working with the service provider to resolve. Meanwhile, the new Raspberry Pi temperature monitor is working well and its automated alerts helped us avoid multiple heat related outages during the transition.

Community Engagement
We recently sponsored the Welcome Reception at The African Internet Summit (AIS) that took place in Kampala in June. We were happy to see many of you there and hope you enjoyed it. We also recently joined the ICT Association of Uganda and look forward to supporting their efforts to promote effective government policy.

Government Relations
As many of you are aware, Uganda’s telecommunications industry is facing an increasing number of regulatory challenges, and the UCC’s recent IXP licensing proposal is no exception. It took a significant amount of time and energy to organize the necessary local, regional, and global response to this proposal and we thank all who supported our cause. We will update you if and when we receive any feedback from the UCC.

For background and more information about this issue, please read our company blog post on the matter: http://blog.uixp.co.ug/2019/06/nationalization-in-uganda-looming.html

Governance Reform
We still have not received any feedback on the draft constitution that we proposed to the community in 2015 and again, in updated form, in 2018.

If you are interested in governance reform, please review this draft and submit feedback (or forward it to your legal teams for the same) so that we can move the process forward. A copy of the latest draft proposal and a summary document can be found here: https://uixp.co.ug/documentation/governance-reform-updated-draft-memarts-2018-12

That’s all for now. If anyone has any questions, please don’t hesitate to contact us.

Nationalization in Uganda: A Looming Disaster

The Uganda Communications Commission (UCC) has proposed a new licensing framework that would effectively nationalize a core part of the country's telecommunications industry; Internet exchange points (IXPs).

Introduction to Internet Exchange Points (IXPs)

This news was communicated to us in a letter from the UCC dated June 7th, 2019. The letter included a draft of the licensing framework and a call for written feedback by July 5th, 2019 (click here to download a full copy).

Based on our analysis, the draft framework would establish a government controlled monopoly which all other market players would be subservient to. It would accomplish this with the following formula:

• Establish a “Designated National Internet Exchange Point” that all other IXPs will be required to connect to (9.j);


• Require government approval of contracts between IXPs and network operators (7.4.b);


• Allow the government to arbitrarily compel IXPs to make operational and technical changes (7.5.c)


• Allow the government to inspect, copy, or remove any data related to any IXP without a court order (7.5.b.i);


• Require all licensed network operators to connect to an IXP (8.2.a).

Technically speaking, this policy would merge all IXPs into a single national peering LAN, with each IXP merely acting as a heavily regulated access point for the enlarged infrastructure. The resulting entity would suffer from all of the classic symptoms of a monopoly as well as significant technical challenges and security risks derived from having multiple operators control access points that form part of the same LAN. As a centralized service, it would also inherently lack the resilience that a diverse array of independent IXPs would provide.

Here are two diagrams to help illustrate the concept:

In addition, the draft framework contains language which suggests that the government intends for the "Designated National Internet Exchange Point" to establish itself by expropriating an existing private operation; namely, ours.

This appears to confirm some of our worst fears about Uganda's new National Broadband Policy; a government strategy document, reportedly drafted in isolation, that seemingly calls for a large-scale nationalization and centralization of Uganda's Internet infrastructure under the guise of infrastructure sharing.

We (and others) have repeatedly warned that such policies would have severe socioeconomic consequences for Uganda and the wider East African region. In this particular case, the UCC's planned regulatory intervention in our otherwise healthy industry has no successful parallel anywhere in the world -- and global experts widely regard the other attempts as textbook examples of regulatory failure.

In light of the obvious risks and highly technical nature of this proposal, we feel that this limited survey of the local Internet community is insufficient validation. Accordingly, we strongly urge the UCC to defer any further work on this project until there is a clear rationale and its viability can be transparently proven with case studies and corroborating input from credible global experts.

Meanwhile, we are preparing to submit detailed feedback to the UCC and will post a copy of our submission here once it is ready. We encourage anyone else that would like to submit feedback to do so through us electronically. We will collect, manually submit, and (unless anyone objects) electronically publish all that we receive in order to promote transparency.

Please feel free to contact us here: board@uixp.co.ug

[UPDATE: A copy of our formal feedback to the UCC can be downloaded here. Our general position is that the creation of a de jure IXP monopoly would be bad for our industry, Uganda, and the region. We argue that the regulatory framework should instead seek to create an enabling environment for competition.

We have also uploaded a number of supporting submissions made by the Internet eXchange Federation (IX-F), the African Network Information Centre (AFRINIC), the ICT Association of Uganda (ICTAU), Liquid Telecom, and a personal submission by Diarmuid O'Briain. A copy of those submissions can be downloaded here.]

Annual Update: Sustainability, Stability, and Growth

This is an annual update regarding the UIXP’s progress in 2018 and our ambitions for 2019.

2018 was a surprisingly good year: We overcame substantial challenges, attracted new peers, deployed a prototype Google cache, implemented a new sustainability model, upgraded our failing power system, and paid our legal debts. As a result, we are now significantly better positioned for future growth and, therefore, to deliver significantly more value to our members.

OLD POWER SYSTEM	NEW POWER SYSTEM

In 2019 we plan to build on these successes by lowering prices; developing internal structure; paying key staff; improving service quality; hosting quarterly events; supporting the local technology community; becoming fully tax compliant; and implementing governance reform.

The new pricing structure aims to attract more networks by making peering more affordable: We now offer 10 Mbps ports for free; have cut the cost of 100 Mbps ports by 60%; and have reduced the cost of 1 Gbps ports by 9%. These adjustments were possible to implement without negatively impacting our overall revenue because of growth in our paying membership base, and because many networks have transitioned to (or will soon transition to) more expensive 10 Gbps ports where our existing rates are still cost-effective.

PORT CAPACITY	2018 MRC	2019 MRC
10 Mbps	$100 / mo	FREE
100 Mbps	$250 / mo	$100 / mo
1 Gbps	$550 / mo	$500 / mo
10 Gbps	$1000 / mo	$1000 / mo

We are also excited to announce the impending arrival of a large social media network in Q1 2019. This network will peer directly and should significantly increase the amount of traffic networks generate from our exchange. We are still working out some of the technical details and will share more on this soon.

Finally, we would like to note that none of this would have been possible without the networks that supported the implementation of our sustainability model -- and those that have committed to do so in 2019. To these networks we are extremely grateful. We are heartened by your support.

We also thank everyone else for their participation and look forward to interacting with all of you in the coming year!

Liberalization in Uganda: A Looming Regression

On Thursday, April 26th, a concerned network operator forwarded us a copy of a survey they received from the national telecommunications regulator, the Uganda Communications Commission (UCC), which seeks input on how the UIXP should be governed, managed, and financially sustained.

We were not previously aware of this survey and are deeply concerned by its contents. Its text is inaccurate and misleading; its questions are poorly formed; and its broad dissemination indicates that the UCC is planning to attempt a top-down intervention in the UIXP's governance model and, by extension, Uganda's nascent network interconnection industry.

April 2018: Cover letter for UCC survey seeking input on the how the UIXP should be governed, managed, and financially sustained.

The UCC issued this survey with full awareness that the UIXP is actively and transparently working to advance its own governance reform process in collaboration with the network operator community; that the issues we need to address have not adversely impacted our growth and do not present an immediate risk to the continuity of our operations; and that the neutral non-profit governance model which the UIXP adopted in 2001 has become one of the most dominant and successful in Africa.

The UCC's justification for issuing this survey stems from a letter sent to them by two members of our Internet community that libelously accused the UIXP management team, Google, and Akamai of fraud and impropriety. The UCC quickly and conclusively learned that these claims were false but continued to interject in our affairs under the guise of a neutral intermediary intent on resolving a community conflict.

Meanwhile, in the background, we see a rising frequency of controlling and economically damaging government interventions in the telecommunications space. Recent examples include a controversial move to reclaim and prop up the terminally ill state owned telecommunications company; a proposal to limit the number of international gateway providers; an explicit order to tax social media users in order to curtail unfavourable on-line discourse; and a directive to block all online news providers that have not been granted a national license.

The government has also made multiple attempts to nationalize or directly involve themselves in the UIXP's governance in the past, including a 2014 vote by Parliament (which was never implemented) and other less formal/ethical efforts to achieve the same. They also launched a failed attempt to nationalize the .ug ccTLD and have displayed an increasing tendency to censor or block telecommunications services (including mobile money) during presidential elections and politically sensitive events.

In this context, it is hard to trust that the UCC's interest in the UIXP's governance model is benevolent. The nature of their current involvement, and this survey in particular, raises the specter of nationalization and other forms of government intervention that could ultimately deter or prohibit competition in Uganda's emerging network interconnection industry -- an outcome that would have significant long term consequences for Uganda's telecommunications market and national economy.

2014: Parliament adopts an ICT committee recommendation to nationalize the UIXP based on false information and without engaging UIXP management in any way.

The African network interconnection industry is evolving rapidly. There are now 42 IXPs in 32 countries which carry over 400 Gbps of Internet traffic on a daily basis -- up from only 160 Mbps in 2008. South Africa, Nigeria, and Angola all now have more than one IXP operator, with other countries soon to follow. In tandem, we are starting to see the deployment of carrier neutral datacenters, large-scale content, and cloud services.

In Uganda, the UIXP now interconnects 28 networks and carries over 6 Gbps of Internet traffic on a daily basis -- up from only 10 Mbps in 2008. This has made our market much more attractive to international content providers and carrier neutral datacenter investors. If we continue along this path, there will soon be enough demand for local network services to justify the entrance of Uganda's first carrier neutral datacenter and second IXP operator. This would mark a milestone in Uganda's telecommunications history and likely herald a golden era of Internet connectivity.

UIXP: Chart showing connected networks, traffic growth, and key governance events over time.

However, expropriating our private not-for-profit company, and nationalizing our nascent industry in the process, is a great way to make sure that never happens. It would significantly increase Uganda's investment risk profile; prevent new IXPs from forming; and make carrier neutral datacenters -- which rely on network interconnection for growth -- far less viable. As a result, large content providers would need to look elsewhere for hosting (e.g. Kenya) which, in turn, would ensure that Uganda must continue to pay other countries for access, thereby keeping end-user prices high and service quality low.

In our view this is a very real possibility that threatens everyone's interests. We therefore call on all network operators and other recipients of the UCC survey to take this into account when considering if, and how, to respond.

We further call on all network operators to more actively participate in our own sustainability and governance reform process. It should be clear to all by now that the only good way forward is to work together.

UIXP Portal: Launch Announcement

Today we are proud to announce the launch of the UIXP Portal; an open-source central management system and customer portal. It’s taken a few weeks to set up and modify -- and there are still a few bugs -- but it should be a vast improvement over the collection of independent systems we had before.

The software, formally known as IXP Manager, was developed by the team at INEX (The Internet Neutral Exchange Association). They provided valuable assistance throughout our deployment by working with us to resolve bugs and implement feature requests. We plan to continue actively working with them to improve this software as we feel it has clear value to the global Internet exchange community.

The system can be accessed via the “Portal Login” link at the top-right corner of our website and directly via this URL: https://portal.uixp.co.ug

Here’s a brief list of features:

• Centralized management of customers, switches, ports, and other core infrastructure.
• Automated configuration of MRTG, Reverse DNS (PTR), Route Servers, and Nagios. These systems were previously managed manually, which was becoming increasingly time consuming as the exchange has grown.
• Public aggregate statistics and Looking Glass functionality. This helps outside networks better evaluate the value of joining our exchange.
• Support for multiple sites and peering fabrics, making future expansion easier.
• Automated reporting and alerts for traffic anomalies, congestion, etc.

The system also provides a customer portal with:

• Port data and statistics: bits, packets, errors, discards, and broadcasts
• Technical and contact data for all networks at the exchange
• A peering matrix which shows network interconnectivity based on route server data. Sflow capability will be added in the future to improve accuracy and capture bilateral peering sessions.
• A peering manager that helps customers keep track of bilateral sessions.
• Integrated mailing list subscription management.
• The ability to add or delete additional customer user accounts.
• The ability to update customer NOC contact and billing data.

If you are one of the networks connected to our exchange, here’s how to get started:

1. Send us an e-mail address that we can associate with your organization’s “master” account. This account will *only* have the ability to create regular accounts for your company. Regular accounts created by this master account will have access to the full customer portal functionality described above.
2. Send us a generic technical/NOC e-mail address (e.g. peering@domain.com) that we can add to our core contact database for your company.

Once the master account has created at least one regular account, please use it to:

1. Update your company’s full NOC contact and billing data.
2. Subscribe to our mailing lists by visiting the account profile page.

Please contact us to submit the above data or if you have any questions when getting started.

UIXP Network development report for 2016/17

UIXP is currently involved in an upgrade of its core network. This blog entry serves as a short technical report on the need for the change, the new network design as well as a current progress report.

Where is the exchange coming from?

Illustration 1: UIXP Network prior to the upgrade

In the past the network was built around a pair of HP ProCurve 3400CL switches. These switches offer 4 dual-personality ports - each port can be used as either an RJ-45 10/100/1000 copper port or an open mini-GBIC slot for fibre based transceivers plus 20 auto-sensing 10/100/1000 ports. The network was operated as a flat switched network with no separation of traffic types. Services on a Dell PowerEdge 750 Server were connected via a HP ProCurve 2524 100 Mb/s switch in the core which was interconnected with the peering switches via a 100 Mb/s CAT5e copper cable.

What is the motivation to upgrade?

With the addition of the Akamai Content Delivery Network (CDN) cache to the exchange and two Google caches located on member networks but accessible through the exchange it became necessary to re-look at network design as traffic levels rose significantly.

Physical network 

Illustration 2: UIXP - Physical layout

One of the limiting factors of the old model was the interconnect between the switches. This was a single physical IEEE 802.3z Type 1000Base-X giving a 1 Gb/s trunk. The distribution of members between the switches meant the Ethernet bundle came towards the limits of its bandwidth capacity.

The first thing necessary was a complete rebuild of the Core and Peering elements of the network and a separation of these functions to separate cabinets. A Juniper EX4300 was donated by the Uganda Communications Commission (UCC) as the peering access switch. With over 4 times the throughput of the HP ProCurve 3400CL switches, 4 Small Form Factor Pluggable plus (SFP+) ports that support IEEE 802.3ae Type 10GBASE-X as well as 48 port IEEE 802.3ab 1000Base-T for member peers. This switch is placed as a Top of Rack (ToR) switch in the Peering cabinet facilitating interconnection by the eXchange members as either 10, 100 or 1000 Mb/s. The Virtual Chassis configuration feature of the EX4300 is attractive given the potential to connect a second such switch in the future in the second peering cabinet.

The main core switch is a Cisco Nexus 3548 that was donated to the exchange by Packet Clearing House (PCH) with 48 fixed SFP+ ports IEEE 802.3ae Type 10GBASE-X, 10GBASE-CU SFP+ with Twinax Direct Attach Cables (DAC). Lower speeds are supported via Gigabit Line Card (GLC) SFPs for both Fibre and Copper 1 Gb/s interfaces configurable to lower speeds where necessary. This switch is interconnected to the Juniper peering switch with a 10 GB/s link configured as a Virtual Local Access Network (VLAN) trunk. The core switch has currently the HP ProCurve 2524 connected to cater for lower speed interfaces within the core network which reduces the number of SFPs necessary in the Cisco Nexus. This switch will be replaced by one of the HP ProCurve 3400CL switches once the members begin to migrate to the new peering cabinets.

The Akamai CDN Cache connects to the Core switch via a 10 Gb/s fibre interface while the Proxmox cluster nodes each connect via 1 Gb/s copper interfaces.

The old Cisco 3500 router has given way for a less power hungry Cisco C2801 router in the new core cabinet. As this routers function is to facility the distribution of traffic between the internal UIXP networks and the Internet, the bandwidth requirement is actually quite small and the C2801 is quite adequate for the function.

Infrastructure as a Service (IaaS) platform

To deliver core services it was necessary to build a robust Hypervisor based Infrastructure as a Service (IaaS) that could support the orchestration of both Virtual Machines (VM) and Containers (CT) to support the functions required at the eXchange.

The selection criteria for the hypervisor platform considered the need for it to be a Free and Open Source (FOSS) platform that supports High Availability (HA) as well as both VMs and CTs. The options explored were OpenStack and Proxmox. Both met the requirements of HA and IaaS. OpenStack is released under a FOSS Apache License, while Proxmox is licensed under the GNU is Not Unix (GNU) Affero General Public License (AGPL) version 3, so both are FOSS.

OpenStack however was considered more suitable for a Service Provider wishing to provide cloud services to end customers. This is not a requirement for the exchange and addes significant complexity. While the Proxmox Virtual Environment (VE) is not as fully featured as OpenStack it is powerful and simpler to deploy and use with all the features required by the eXchange.

Proxmox is Debian GNU/Linux based and uses robust Kernel Virtual Machine (KVM) technology and LinuX Containers (LXC). A major plus of Proxmox is the HA Cluster features. When VM or CT instances are configured as HA and the physical host fails, the virtual instance is automatically restarted on the remaining Proxmox VE Cluster nodes. It was considered that the Proxmox VE HA Cluster is based on proven GNU/Linux HA technologies and would provide the stable and reliable HA service required.

Initially the Proxmox cluster consists of the Dell PowerEdge 750 and an old Dell Server, however thanks to a upcoming donation from the Internet Society of an additional Dell PowerEdge 750 it will be possible to upgrade the Proxmox cluster hardware. This VE cluster is an essential element of the exchange and hosts the various Virtual Network Functions (VNF) and Server instances as either VMs or CTs.

Logical network 

Illustration 3: UIXP Logical network design

Considering a number of items, the need to separate traffic types and information/network security to name but a few it was decided to split the network into logical elements, a peering Local Access Network (LAN) to contain the member peering interfaces as well as the Root Servers (RS) and the Autonomous System 112 (AS112) Nameserver. A private management LAN for intercommunication between the functions and a DeMilitarised Zone (DMZ) LAN to permit controlled access to the various networking devices, VMs and CTs.

Current state

Well most of the physical network elements are already in place and we await the migration of the peers to the new peering cabinets. The Proxmox cluster is in place and will be beefed up by the addition of the second Dell PowerEdge 750 and it supports the core services that are built on VMs and containers. Once that is complete the work of separating the LAN into the logical elements just described will begin. Looking forward to a busy 2017.

Abbreviations

AGPL	Affero General Public License
AS112	Autonomous System 112
CDN	Content Delivery Network
CT	Containers
DMZ	DeMilitarised Zone
FOSS	Free and Open Source
GLC	Gigabit Line Card
GNU	GNU is Not Unix
HA	High Availability
IaaS	Infrastructure as a Service
KVM	Kernel Virtual Machine
LAN	Local Access Network
LXC	LinuX Containers
PCH	Packet Clearing House
RS	Root Servers
SFP+	Small Form Factor Pluggable plus
ToR	Top of Rack
UCC	Uganda Communications Commission
VE	Virtual Environment
VLAN	Virtual Local Access Network
VM	Virtual Machines
VNF	Virtual Network Functions

Bibliography

Packet Clearing House. Available: https://www.pch.net
Uganda Communications Commission. Available: http://www.ucc.co.ug
The Internet Society. Available: http://www.internetsociety.org
Proxmox Server Solutions GmbH. Available: https://www.proxmox.com/en/
Akamai Technologies. Available: https://www.akamai.com
Cisco Nexus 3548 Switch. Available: http://www.cisco.com/c/en/us/products/switches/nexus-3548-switch
Juniper EX4300 Switch. Available: http://www.juniper.net/uk/en/products-services/switching/ex-series/ex4300
HP ProCurve 3400CL. Available: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c01809608
HP ProCurve 2524. Available: http://www.hp.com/ecomcat/hpcatalog/specs/J4813A.htm

Dell PowerEdge 750. Available: http://www.dell.com/downloads/global/products/pedge/en/750_specs.pdf

Diarmuid O'Briain joins our technical operations team

In November 2016, Diarmuid Ó'Briain, a lecturer at Makerere University, joined our team as Technical Operations Manager in order to help us address a significant human resource shortage which arose as a result of increased demand for our services.

Diarmuid is a Chartered Engineer with over 25 years experience in the sector. He previously worked as an engineering manager at multinational networking and telecommunications companies including US Robotics, 3Com, and UTStarcom where he designed and implemented next-generation networks and information security solutions for several clients, and was instrumental in the design and support for NGN solutions for carriers around the EMEA region.

His most recent role prior to arriving in Uganda in 2015 was Chief Technology Officer at Ripple Communications, an Irish Internet service provider, where he developed the core network and managed the design and delivery of fibre and licensed radio back-haul infrastructure for their national-scale access network.

Today he teaches Networking, Information Security, and Wireless Technologies at the Makerere College of Engineering, Design, Art, and Technology and is an active member of the Uganda Institute of Professional Engineers (UIPE).