Free 100Mbps Ports, New CDNs, and Infrastructure Upgrades

Message from the Executive Director

2023 was a good year for the UIXP despite some significant challenges.

We started off facing three long-term CDN outages and reduced income due to low demand but finished with four live CDNs, two new peers, and a small surplus. Peak daily traffic increased from 10 Gbps to 45 Gbps while IPv6 adoption increased to 33% of connected networks.

Along the way we restored Google's remote peering link to Mombasa, successfully deployed a new Meta cache, and will soon enable a new Netflix cache in collaboration with Lyca Mobile. We also upgraded our back-end virtualization platform, implemented new cross-site functionality, and enhanced our data storage systems with N+1 redundancy. In addition, new security measures were implemented as part of ongoing efforts to improve physical and digital security.

On the sustainability front, our cash reserves remain untouched and we maintain good standing with URA despite the usual challenges. However, we observed a rise in account delinquencies among smaller networks which we attribute to challenging macroeconomic conditions. In response, we are working with stakeholders to promote progressive government interventions.

We are also working to implement a new open-source accounting system which would provide a number of benefits including improved accessibility and automation. As we progress, we may seek support for testing purposes.

Looking forward, we are thrilled to announce the introduction of free 100 Mbps ports at Raxio and Communications House. This was made possible by the growing utilization of 1 Gbps, 10 Gbps, and 100 Gbps ports. In line with this, any networks that use 100 Mbps ports will no longer be charged while any networks that use free 10 Mbps ports will be upgraded automatically.

I encourage everyone to read the full report below for more details. As always, we are grateful for everyone's support and look forward to working together for the good of the Internet in 2024.

Kyle Spencer,
Executive Director

Free Ports & Raxio Discounts!

We are excited to announce that we now offer free 100 Mbps ports and note that Raxio customers can still access our service without any cross-connect charges.

We believe this move will facilitate growth and competition in our market without imposing any significant risk to our sustainability thanks to the growing utilization of 1 Gbps, 10 Gbps, and 100 Gbps ports. Any networks that use 100 Mbps ports will no longer be charged while networks that use free 10 Mbps ports will be upgraded to 100 Mbps.

As a reminder, we also offer a 6-month 25% discount on all new services at Raxio. If you would like to connect, upgrade your port, or learn more about our services, please visit our website.

Utilization Report

• We began 2023 with 32 connected networks but only ~10Gbps of peak traffic due to cascading CDN outages involving Google and Akamai (in addition to the ongoing Facebook ban which disabled their legacy cache in 2021).
  
  
• Akamai's cache remains offline due to challenges related to IP transit. If anyone would like to donate IP transit for this cache, please contact us.
   
• We helped Google stabilise their remote peering link to Mombasa following months of severe disruption. The successful restoration of their link provided significant benefits to our members and lowered the cost of delivering Google traffic within our catchment area. However, we understand that Google may withdraw remote peering links in the future due to a global policy change so we encourage everyone to prepare for this possibility.
  
  
• Meta deployed a new cache at the UIXP within the Raxio data center. This was a fast and straightforward deployment thanks to Meta's full support for the project. This lowered the cost and improved the performance of traffic related to Meta's various services. Due to demand growth, we are already working with Meta to upgrade their capacity.
  
  
  
• We are in the final stages of deploying a Netflix cache in partnership with Lyca Mobile and hope to bring it online within the coming weeks. This project is somewhat experimental for Netflix since they typically deploy caches within ISPs and MNOs. Netflix's prefixes will be announced to the route servers by the UIXP's ASN [328998] when it goes live due to the nature of their cache's architecture. If you wish to establish a bilateral peering connection with our ASN, please contact us.
   
• IPv6: 33% of all networks (10 out of 30) are peering with our route servers via IPv6. This reflects growing support in our market though most traffic is still exchanged via IPv4. We encourage all networks to transition to IPv6 and we are happy to provide support where possible. Click here for a live list of networks peering via IPv6 in Uganda.

Infrastructure Updates

• We upgraded our server virtualization platform, implemented cross-site functionality, and upgraded our data storage systems with N+1 redundancy throughout. We also took steps to improve physical and digital security, though this is a continuous process and additional measures will be implemented in 2024.
  
  
• We continue to maintain our facility at Communications House. Power, air-conditioning, and security systems are all operating well. Annual uptime exceeds 99.99%.
   
• As a reminder, we send technical alerts about notable changes and foreseeable outages to the members-only techies@uixp.co.ug mailing list. If your network is peering but is not a member and would like to be, please contact us at support@uixp.co.ug.

Invoicing, Collections, and Taxes

• We started the year with no surplus and depressed income due to three long-term CDN outages. Despite this, we ended the year with a small surplus and our cash reserves remain untouched.
  
  
• In 2023 we observed a small but growing number of account delinquencies. We believe these delinquencies reflect increasingly difficult macroeconomic market conditions and we are working with various stakeholders to encourage government interventions that would relieve pressure on our industry. We are also doing our best to accommodate affected networks while balancing our need to maintain fairness and sustainability.
  
  
• We are attempting to migrate to a new open-source accounting system and, as part of this, automate certain processes. We may contact some of you in the near future to request support for testing purposes.
  
  
• We remain in good standing with URA despite the typical challenges of compliance.
  

Environmental Issues

• The 2021 Facebook ban continues to suppress overall utilization of the Internet and our interconnection service. It continuously inflates the cost of Internet service delivery by forcing users and networks to import Facebook traffic from international sources rather than our local cache. This makes Uganda’s Internet more expensive and its market less attractive for regional interconnection & colocation relative to neighbouring countries.
  
  
• We remain concerned by the increasingly high taxes levied on Internet services which now account for over 50% of the total cost of access. Like the Facebook ban, this makes the Internet more expensive and hinders demand for Uganda’s digital economy.

Community Engagement

• All UIXP staff continue to play voluntary leadership and support roles within the African IXP Association (AFIX), the Africa Peering and Interconnection Forum (AfPIF), and the African Network Operators Group (AfNOG).
  
  
• The UIXP remains a member of the ICT Association of Uganda (ICTAU) and Internet Service Provider Association of Uganda (ISPAU). We work with both organizations to promote community development and good policy outcomes.
  
  
• The UIXP continues to facilitate donations for the Uganda Network Operators Group (UGNOG) until such time that they can formalize and register a bank account. Any organizations that wish to sponsor UGNOG should notify us by sending an e-mail to accounts@uixp.co.ug CC isabel.odida@gmail.com.

 

Disaster Recovery: CDN Outages, Financial Challenges, and the Way Forward

Message from the Executive Director

2022 was difficult for the UIXP.

The year started off strong. In January we had 27 networks exchanging 35 Gbps of peak daily traffic. In February we connected our management network to our route servers in order to deliver new low-latency services. In March we deployed a dark fibre link between Raxio and Communications House to facilitate multi-site peering. In May we helped Google extend their global network to Uganda to facilitate the distribution of their traffic via the UIXP.

By June our peak daily traffic reached a record high of 50 Gbps and we planned to start offering free 100M ports – but then a series of external problems destabilized and eventually disabled both Google and Akamai’s peering amid a lingering national Facebook ban. This multifaceted environmental disaster reduced our peak traffic to 6 Gbps and negatively impacted our income immediately after we incurred new costs related to our multi-site expansion.

We are doing our best to help Google and Akamai resolve their issues while we work with Netflix and Meta to deploy new caching systems. We have also assumed a more conservative budgetary posture in an effort to maintain financial stability during this difficult period. We faced an unusually high number of delinquencies at the end of the year and we will be working to address this in the coming months.

In the background, we continue to contribute to community initiatives at the local, regional, and global level. This includes the Uganda Network Operators Group, ISP Association of Uganda, ICT Association of Uganda, the African Network Operators Group, the African Peering & Interconnection Forum, the African IXP Association, and Internet Exchange Federation.

I encourage everyone to read the full report for more details. As always, we are grateful for everyone's support and look forward to working together for the good of the Internet in 2023.

Kyle Spencer,
Executive Director

Raxio Promotional Discounts

As a reminder, significant discounts are available to all Raxio customers including a free cross-connect to the UIXP and a 6-month 25% discount on all exchange services. If you would like to connect, upgrade your port, or learn more about our services, please contact us.

New Members & Connections

Google (AS 15169), Raxio (AS 328821), Group Vivendi Africa (AS 36924), and the Uganda Internet Exchange Point management network (AS 328998) joined the exchange in 2022. This diverse array of content, enterprise, and access networks are all connected to our new node in the Raxio data center, bringing the total number of peers at that location to five.

Looking ahead, we intend to engage a broader range of Raxio customers in order to expand our membership. The more networks that connect, the more valuable the exchange is to everyone.

Technical Updates

We upgraded our inter-site link to 100 Gbps with geographic redundancy in March 2022 in order to accommodate the anticipated flow of Google traffic between Raxio and Communications House where the majority of our customers are connected. C-Squared is providing this link and we are grateful for their reliable and attentive service.

We experienced multiple broadcast storms in 2022 that significantly disrupted some of our member networks. In response, we are gradually implementing technical changes to reduce the possibility of future disruptions. Most notably, we now restrict all ports to a single MAC address.

As a reminder, we send technical alerts about notable changes and foreseeable outages to the members-only techies@uixp.co.ug mailing list. If your network is peering but is not a member and would like to be, please contact us.

Content Networks: Outages & Upcoming Deployments

Four content networks are connected to the UIXP but only YoTV is actively peering. Facebook, Google, and Akamai have all been disabled by different circumstances detailed below:

• The Facebook cache (AS 63293) stopped serving traffic in January 2021 after a government ban forced network operators to block access to the service. This reduced local bandwidth production by 30% and damaged Uganda’s reputation in the global telecommunications industry. Meanwhile, many users continue to access Facebook via virtual private networks (VPNs) which force network operators to import traffic from more expensive international sources. In other words, the Facebook ban increased the cost of service delivery, reduced overall performance, and steered investment to neighbouring countries which are perceived to have more attractive and stable policy environments.
  
  
• Google (AS 15169) established a remote peering session in May 2022 after decommissioning their prototype “self serve” cache (AS 36040). However, Google’s regional transport suppliers were unable to deliver a stable link to Mombasa despite six months of joint troubleshooting. The session was ultimately disabled in December 2022 due to extreme instability and poor quality of service. Google is now working to change transport suppliers and we are actively supporting them with this process.
  
  
• Akamai’s cache (AS 20940) stopped serving traffic when Google’s instability began because it forced the cache-fill donor, NITA-U, to re-route Google related user traffic via international links. This overloaded their capacity and the resulting congestion forced the Akamai cache offline. NITA-U is working to upgrade their international capacity but the procurement timeline is uncertain so we are searching for a new donor and may need to implement a paid access model until Akamai can justify paying for the cache-fill themselves.
  
  

On a more positive note, Netflix and Meta are shipping us new edge caches which we hope to receive next month. Our goal is to bring them online in the second quarter of this year:

• Meta plans to procure their own cache-fill and will peer with our route servers. Their new system will reportedly make it easier for networks to distinguish between traffic related to Facebook and other Meta properties such as Instagram, Oculus, and WhatsApp.
  
  
• Netflix requires a cache-fill donor but their requirements are relatively easy to fulfill due to the static nature of their content catalogue. If your network has 500 Mbps of idle capacity at night and would like to donate it to the UIXP community for this purpose, please contact us to discuss the requirements in more detail.
  
  
  

Environmental Challenges

The 2021 Facebook ban continues to suppress overall utilization of the Internet and demand for our regional interconnection service. It also increases the cost of Internet service delivery by forcing network operators to import Facebook traffic via international VPNs rather than sourcing it locally from our cache. In general, it makes the Internet more expensive and constricts Uganda’s commercial gravity relative to neighbouring telecommunications markets.

We are also concerned by the high taxes levied on Internet services which now account for over 50% of the total cost of access. Like the Facebook ban, this also makes the Internet more expensive, suppresses overall utilization, and reduces demand for our service.

Financial Challenges

We unfortunately ended the year without any surplus or savings due to an unusually high number of delinquencies. The problem is limited to a minority of networks but the loss of income is significant due to the thin margins of our “cost-recovery plus” sustainability model.

While we are sympathetic to the difficulties that network operators face in our market, it is in everyone’s best interest for everyone to keep their accounts up to date. Income instability impacts our ability to pay suppliers, maintain our infrastructure, and plan for the future.

That aside, our cash reserves remain intact and we remain in good standing with URA despite occasional conflicts caused by erroneous assessments, improper penalties, and onerous bureaucracy.

Community Engagement

UGNOG: The UIXP continues to facilitate donations for the Uganda Network Operators Group (UGNOG) until they can formalize and register a bank account. Any organizations that wish to sponsor UGNOG should notify us by sending an e-mail to accounts@uixp.co.ug CC isabel.odida@gmail.com.

ISPAU & ICTAU: The UIXP is a member of the ISP Association of Uganda (ISPAU) and ICT Association of Uganda (ICTAU) and works through both organizations to promote community development and good policy outcomes for our industry.

AFIX, AfPIF, & AFNOG: Our team remains deeply involved in various voluntary leadership and support roles in the African IXP Association (AFIX), the African Peering and Interconnection Forum (AfPIF), and the African Network Operators Group (AfNOG).

UIXP Expands Into Raxio, Google Joins the UIXP, Switch Platform Upgrade, and more!

Message from the Executive Director

2021 was a good year for the UIXP. New networks connected, traffic levels doubled, and we expanded into Raxio; Uganda’s first Tier-III carrier neutral data center. We added a new member to our team, received new equipment donations, and increased our community development work. Our infrastructure and financial position also remained stable thanks to the support of our growing community of peers.

However, this success wasn’t easy. Our industry collectively endured a lingering pandemic, an Internet shutdown, a Facebook ban, and the imposition of a new 12% excise tax which all continue to hinder growth. We estimate that the Facebook ban alone has depressed Uganda’s local Internet bandwidth production by 30% to date and project that this figure will grow over time until the ban is lifted.

Despite these challenges we are optimistic about the future. We anticipate significant growth in 2022 and believe Uganda now has the potential to evolve into a regional content and interconnection hub. This would generate significant benefits for our industry, our country, and our region, so we plan to work aggressively with our members, partners, and stakeholders to promote this outcome. 

I encourage you to read the full report below for more details and to let us know if you have any comments or questions. As always, we are grateful for everyone's support and we look forward to working together for the good of the Internet in 2022.

Kyle Spencer,
Executive Director

UIXP Expands into the Raxio Data Center in Namanve

The UIXP recently expanded its peering infrastructure into Raxio, Uganda’s first Tier-III carrier neutral data center, and signed Google (AS15169) as the first peer in this new location. When the inter-site link goes live, all networks will be able to peer with each other from either of our two points of presence.

This collaboration between the UIXP, Raxio, and Google marks a significant milestone in the development of Uganda’s Internet ecosystem. It significantly increases domestic bandwidth production capacity; makes large-scale enterprise, content, and cloud infrastructure deployments viable; and promotes regional network interconnection.

Google will peer multilaterally via the UIXP route servers. If your network is also peering via the route servers, it will start to exchange traffic with Google automatically once their service goes live. The amount of traffic could be substantial so you will need to ensure that your network has sufficient backbone and port capacity in order to avoid congestion. If necessary, operators can filter out AS15169 prefixes to divert this traffic until they are prepared to route it via the UIXP.

In order to promote new connections in Raxio, we are offering multiple discounts including a free cross-connect for all Raxio customers plus a 6-month 25% discount on all UIXP services at that location. If you would like to connect, upgrade your port, or learn more about our services, please contact us.

The Internet Society Donates New Switches & Servers

We recently upgraded our core switching infrastructure to an Arista platform that was generously donated by The Internet Society. These new switches support 100G interfaces, advanced automation, and deep buffers; all features that will become critical as we scale up our operations and face new challenges related to operating a multi-site IXP.

The Internet Society also donated two new Dell servers which have been deployed in Raxio as a virtualization cluster. Aside from hosting our operational software systems, these servers will enable us to offer new locally hosted services including software mirrors.

We are extremely grateful to The Internet Society for these donations and their long-standing support for IXP development in our region. We would not be where we are today without them.

Patience Nagaba Joins the Engineering Team

Patience Nagaba has volunteered to join our team as an Exchange Engineer. Patience is a Senior Network Engineer at the Research and Education Network for Uganda (RENU) and has volunteered at AfNOG. She also holds an MSc. Communication Engineering and Signal Processing from the University of Manchester and a BSc. Telecommunications Engineering from Makerere University.

As an Exchange Engineer, Patience helps manage our infrastructure and customer support requests. She has already made a number of significant contributions and we are grateful to have her on the team.

Community Development Updates

PeerFest: In October 2021 the UIXP and Raxio hosted the first annual PeerFest in order to promote awareness and collaboration among industry stakeholders. This Oktoberfest themed event included live entertainment; a delicious barbeque buffet; and speeches by James Byaruhanga (Raxio), Mike Barnard (UIXP), Kyle Spencer (UIXP), and Irene Kaggwa (UCC). For more details, check out PC Tech's coverage of the event.

UGNOG: The UIXP has agreed to facilitate donations for the Uganda Network Operators Group (UGNOG) until they can formalize and register a bank account. Any organizations that wish to sponsor UGNOG should notify us about incoming transfers by sending an e-mail to accounts@uixp.co.ug CC isabel.odida@gmail.com with the relevant details.

ISPAU & ICTAU: The UIXP is a member of both organizations and works through them to promote community development and good policy outcomes for our industry.

AFIX, AfPIF, & AFNOG: Our team is voluntarily involved in various leadership and support roles in the African IXP Association (AFIX), the Africa Peering and Interconnection Forum (AfPIF), and the African Network Operators Group (AfNOG).

Infrastructure & Stability Report

The UIXP remained stable in 2021 though we experienced a few flaps which were generally the result of human action. Our Communications House power backup system remains reliable and our air conditioning system has kept the room cool despite occasional maintenance needs.

Our IP transit had some stability issues in 2021 and we are upgrading to a multi-homed configuration in order to address that. This project required the deployment of new routers because our old routers did not support 32bit ASNs and upgrading the OS was problematic.

In the near future there may be some brief service outages related to our multi-site expansion though we hope to keep this to a minimum. Our inter-site fibre link may also experience brief outages until we can afford to upgrade to a protected circuit.

As always, we will do our best to provide advance notice for any foreseeable outages via the members-only techies@uixp.co.ug mailing list. If you are not a member of this mailing list and would like to be, please contact us.

Financial Sustainability

The UIXP’s cash reserves remain intact and we ended the year with a small budget surplus thanks to reliable payments from our growing community of peers. We note that UTL, URA, and Smile are now the only non-free-tier networks that still do not contribute port fees.

Looking forward, we anticipate income growth from new ports and capacity upgrades related to our multi-site expansion. While this growth will be at least partially offset by new expenses for the same, we are optimistic that we will end the year with another surplus.

We remain in good standing with URA despite the burdens of EFRIS and a frequent need for reconciliation due to our cash based reporting method.

Government Relations

We enjoy an amicable relationship with the Uganda Communications Commission (UCC) under Irene Kaggwa’s leadership. We increasingly engage via events and regulatory development processes. Going forward we aim to enhance our collaboration in order to promote Uganda’s evolution into a competitive regional content and interconnection hub.

New Local CDN, 13 Gbps of Traffic, 100.000% Annual Uptime, EFRIS Concerns, and Other Updates

New Peer: Al Bayan Media
The UIXP would like to welcome Al Bayan Media to the exchange! Al Bayan is a locally owned IPTV provider. They are peering publicly via our route servers in order to deliver a high quality service to their customers across all networks in Uganda. If your network is also peering with our route servers, it should already be exchanging traffic with Al Bayan.

If you would like to establish a bilateral session or market Al Bayan’s IPTV service to your customers in order to promote utilization of local traffic, please let us know and we will arrange an introduction.


Peak Traffic Level: 13 Gbps

The UIXP’s traffic has returned to growth following a brief reduction caused by the strict COVID-19 lockdown measures imposed earlier this year. Our daily peak traffic level now exceeds 13 Gbps during some weekdays as can be seen in the graph above.


Annual Uptime: 100.000%
The UIXP recently achieved an annual up-time of 100.000%. This is an improvement over our previous record of 99.994% which was achieved in March. Reliability is a critical growth enabler for the UIXP so we aim to maintain a similarly high level of availability going forward.

This milestone is the result of equipment upgrades that were made possible by our port-fee based sustainability model and we are grateful to everyone who supported its implementation.


IPv6 Peering Reminder
We would like to remind everyone that the UIXP supports IPv6 peering. At the moment, only four out of the twenty six networks that peer with our route servers use IPv6 and we would like to increase this number. We will reach out to some of you directly in the coming months in order to encourage adoption, but if you would like to proactively enable this for your network, please let us know and we will gladly assist.


Conrad Ekisa Moves to Ireland

Conrad Ekisa, one of our esteemed engineers, has left Uganda for some post-graduate research in Ireland. He’s still part of our team but will transition to a more limited remote support role in order to focus on his studies.

Contrad has made very significant contributions to the UIXP that greatly improved our quality of service, customer experience, and internal organisation. We are sad to see him go but we know that he’s destined for great things and wish him well on the journey ahead.

Internal System Upgrades
We recently upgraded our hypervisor cluster to the latest version of its software in order to take advantage of new features, stability improvements, and security updates. We have also made a few other changes to our network and systems that should significantly improve the overall security of our infrastructure.

Looking forward, we are exploring a potential deployment of an open-source ERP system in order to automate more of our invoicing and accounting processes. We also aim to upgrade our core exchange management system in order to take advantage of new features that are available in the latest release. We’ll share more information about this as we progress.


Raxio Expansion: Latest News
Earlier this year we signed a deal to extend our peering network into Raxio’s carrier neutral data centre in Namanve. This will allow us to attract new peers, provide redundancy for the networks that rely on us, and facilitate large-scale content and cloud deployments in Uganda.

As a reminder, we plan to deploy a fully independent IXP node at Raxio and interconnect it with our existing node at Communications House. This will allow us to offer a unified peering LAN that spans both locations. Network operators will be free to peer at either, or both, as needed.

After consulting with other IXPs, we have decided to prioritise Arista’s switching platform for the new site due to its deep buffers and support for advanced automation through an integration with our existing exchange management system. ISOC has generously agreed to sponsor this procurement and we are working to negotiate an order through Arista’s authorized local supplier.

We have also reached a tentative agreement for a dark fibre link between the two sites. This will require us to pay a monthly fee, but we have negotiated a discounted rate and are confident that we can afford the cost so long as collections remain consistent. We plan to finalize and execute the contract for this once we get closer to deployment.


URA: Concerns About EFRIS
We continue to remain in good standing with URA despite the issues reported in our previous newsletter. Following a brief dialogue with their officers, they informally stopped imposing penalties on us for using a cash based reporting method. We do not know how long this reprieve will last, but we’re grateful for URA’s apparent accommodation.

However, we have serious concerns related to the upcoming implementation of EFRIS, a centralized online accounting system which all companies operating in Uganda will be required to use for invoicing and payments. In order to use the system, every company in Uganda will be required to import and maintain its list of materials, products, and services along with all associated pricing information. To learn more about EFRIS, see here: http://efris.ura.go.ug/

In order to comply, the UIXP will either be forced to (a) duplicate all invoicing and accounting work or (b) migrate to an alternative accounting system which can integrate with EFRIS via an API which would undoubtedly come at a cost.

We are also concerned that EFRIS will create a single point of failure for the national economy, dramatically increase the burden of tax compliance, and potentially expose sensitive information to the database operators and others who might inappropriately gain access to the system.

The deadline for implementation was July 1, 2020 but has been extended to January 1st, 2021 following complaints from the Uganda Manufacturers Association (UMA) and others.


Payments & Collections
Collections have generally been good although we did experience some delays during Q2 and Q3 as COVID significantly disrupted the income and working conditions for a number of network operators in our market. Once again, we thank everyone for the continued support!


Community Engagement
We remain committed to community development despite the limitations imposed by COVID. Members of our team have both organised and spoken at a number of virtual events this year including AfPIF’s Virtual Peering Series. We will also be speaking at the upcoming Uganda Network Operators Group (UGNOG) launch event. In addition, we recently made a donation to the ISP Association of Uganda (ISPAU) in order to support their industry development efforts.

That’s all for now. If you have any questions, please contact us.

New Uptime Record; New Traffic Record; and Raxio Expansion Plans

It's Q2 2020 and we have a lot of good news to share: We achieved a new uptime record, observed new peak traffic levels, and are expanding into Raxio's new carrier neutral data centre. Read on for details:

Coronavirus (COVID-19)

We would like to assure everyone that we are fully prepared to maintain operations during the lockdown period regardless of its duration. We have always had the ability to conduct most of our work remotely and the government has authorized a travel pass for us in order to accommodate any emergencies that require us to physically interact with our equipment.

Expansion Plans
We are excited to announce that we have signed a deal to extend our peering network into Raxio’s carrier neutral data center in Namanve. This will allow us to attract new peers, provide redundancy for the networks that rely on us, and make large-scale content and cloud deployments viable in Uganda.

To accomplish this, we will deploy a fully independent IXP at Raxio and interconnect it with our existing IXP at Communications House. This will allow us to offer a unified peering LAN that spans the two locations. Network operators will be free to peer at either, or both, as needed.

We believe this deal represents a significant milestone in the development of Uganda’s Internet ecosystem. Internet exchange points have a symbiotic relationship with carrier neutral data centers and we expect that this deal will catalyze a virtuous cycle of growth that will make the Internet significantly cheaper, faster, and more reliable for all.  

Please visit Raxio's website for a copy of the press release.

New Stability Record: 99.994%
We’re extremely happy to report that we have achieved 99.99% uptime since taking direct control of our power and air conditioning. This success would not have been possible to achieve without everyone’s support for our sustainability model -- so thanks to all once again!
 
Here are some graphs to demonstrate the UIXP’s reliability before and after:

BEFORE: 2018-06-01 to 2019-06-01

97.891% uptime (7d 17h 17m 46s of downtime)

AFTER: 2019-06-01 to 2020-04-02

99.994% uptime (26m 7s of downtime)

Note that the “BEFORE” data represents an extremely conservative downtime estimate as it does not account for partial outages including a multi-week CDN outage caused by an air conditioning system failure.

New Peak Traffic Record: 11Gbps

We recently observed the highest peak traffic levels ever recorded in the UIXP’s history:
 

However, the graph also indicates a significant reduction in peak traffic levels following the outbreak of Coronavirus (COVID-19) due to the Week 13 lockdown. We believe this occurred for the following reasons:

• The lockdown closed businesses and schools which caused a mass migration of users away from uncapped fixed connections to pay-by-the-byte mobile connections while simultaneously evaporating disposable income.
   
• Mobile Internet usage has less impact on the IXP because mobile carriers have on-net CDN cache nodes.
  
  
• Many companies were not able to implement work-from-home solutions, and many popular video conferencing solutions route traffic via relay servers hosted in other countries.
  
  

Tax Challenges

While we are generally in good standing with the Uganda Revenue Authority (URA), we have encountered an apparently insurmountable problem regarding the timing of our VAT payments that we thought was worth sharing.

URA expects companies to pay VAT when invoices are issued regardless of when payments are received. In other words, if we issue $100,000 worth of invoices in January, URA expects us to pay $18,000 before we receive any payments.

We cannot accommodate this for at least two reasons:

1. We cannot pay URA with money we don’t have. We are in a similar situation to most new businesses in that we don’t have enough free cash available to make advance tax payments.
   
2. We can’t force networks to pay because the UCC has prohibited us from doing so. This means that we cannot guarantee that we will be able to recover any advance tax payments even if we had enough free cash to make them.

In light of the above, we are unable to meet URA’s expectations and have opted to pay VAT based on when we receive payments instead. This unfortunately means that we must occasionally pay fines and perform additional administrative work when network operators claim VAT against invoices based on the date of the invoice.

We have attempted to lobby URA about this issue without success. If anyone has any suggestions for how to resolve this problem, please let us know. In the mean-time, we encourage network operators to file any VAT claims against our invoices using the date the payments are made instead of the invoice date.

 

Collections

We are happy to report that all but three non-pro-bono networks have agreed to participate in our sustainability model. We are actively working to find ways to encourage these networks to come on-board, but they are generally either unresponsive or actively refuse to pay. Our options are unfortunately somewhat limited as the UCC ordered us to provide free service to these networks beyond what is available in our free service tier even though we have made it clear that our paying members are unhappy subsidizing their service.

Cash Reserves

We successfully established cash reserves of $75,000 and have filed a board resolution that formalizes this. Any draw-down now requires majority approval from the board and a plan to replenish the reserves. It also requires the board to implement an emergency budget if the reserves fall below 75% of the target level for one year.

Regulatory Environment
We do not have any news to report regarding the UCC's proposed IXP licensing framework and related telecommunications regulations. There has been little movement on this issue, likely due to senior staffing changes within government agencies and the COVID-19 situation. However, we continue to seek an amicable way to resolve the issue and will provide updates if the situation evolves.

Governance Reform
We still have not received any feedback on the draft constitution proposed to the community in 2015 and again, in updated form, in 2018.

If you are interested in governance reform, please review the draft and submit feedback (or forward it to your legal teams for the same) so that we can move the process forward. A copy of the draft proposal and a summary document can be found here: https://uixp.co.ug/documentation/governance-reform-updated-draft-memarts-2018-12
 
Note that we have hired an additional technology oriented legal resource and plan to work with them on future revisions of the draft constitution.

That’s all for now. If anyone has any questions, please don’t hesitate to contact us.

New Networks, New Team Members, Healthy Finances, and Improved Stability

This is a report on the UIXP’s status, progress, and other activities that took place during Q2 2019. We typically only send quarterly updates to the networks which use our infrastructure, but lot has happened in the last three months so we thought it would be prudent to share this one publicly.

27 networks currently exchange 8Gbps of peak daily traffic

New Members
We’re happy to announce that Facebook (AS63293) and Blue Crane Communications (AS328198) are now connected to the UIXP, bringing our total membership to 27 networks. Blue Crane is peering multilaterally via the route servers, but Facebook is not. If you would like to connect to their cache, please send us a message and we will facilitate an introduction. We may also reach out to you directly in order to encourage the same.

Staffing Changes

We’re sad to announce the looming departure of Diarmuid O’Briain as our Technical Operations Manager. Diarmuid has a relentless passion for his craft, and his contributions to the UIXP were both numerous and significant. His legacy will remain, though he will be missed, and we wish him well in all of his future endeavours.

Diarmuid is survived by Brian Masiga and Conrad Ekisa, the newest member of our engineering team. Conrad has a BSc in Telecommunications Engineering and was employed as a Graduate Research Assistant (GRA) at netLabs!UG Research Centre last year. During this time, Conrad qualified as Uganda's first MikroTik Certified consultant and academy trainer. In 2019 he was promoted to Graduate Researcher and led the internship programme for the year. Together with Brian, he will help manage our infrastructure and handle your technical support requests. Please give him a warm welcome!

Diarmuid O'Briain

Conrad Ekisa

Collections

We’re happy to report a substantial increase in the number of paying networks, and that payments have become more consistent overall. However, there are still a few legacy networks which have not joined our sustainability programme despite our best efforts to encourage them, and we will soon start migrating these networks to our free service tier in the interest of fairness.

Cash Reserves

If all goes well, we will have established cash reserves of USD $75,000 by the end of this year thanks, in part, to a generous three year pre-payment by one of our new members. We plan to sustain that amount or more indefinitely.

Salaries
We have started to pay our two IXP Engineers a $100/mo volunteer stipend in order to cover the various expenses associated with their work. These stipends are paid quarterly in order to minimize our administrative overhead. Note that we have not yet implemented remuneration for any other positions in order to prioritize other expenses and the development of cash reserves.

Taxes
We continue to remain in good-standing with URA thanks to support from our tax consultant. In light of his consistently good performance, we have started to pay his fees on a quarterly basis in order to minimize our administrative overhead.

Banking
We still do not have a UGX bank account because opening one would incur an additional monthly charge (in addition to our USD account which costs approximately $12/mo). We currently do few UGX transactions per month beyond URA payments and occasional cash purchases, so it’s cheaper for us to convert USD to UGX when these needs arise. If the volume of UGX transactions increases in the future, we will reconsider our options.

Accounting
We are recording all income and expenses in our accounting system, and we digitally archive all cheques and receipts for auditing purposes. However, we have not been logging URA payments in our accounting system as it’s not a straightforward process. As a result, the account balance in our system does not match the amount in our actual bank account. We are working to figure out the best way to resolve this issue systematically going forward.

Power Infrastructure
Our new inverter system is still working well. We have experienced no related downtime since its deployment, save for a building outage that exceeded the 12 hour runtime of our battery bank.

Air Conditioning
Our air conditioning system has been keeping things cool since we took over its management from the UCC following a series of extended outages. However, there have been a few reliability issues with one of the two new units we purchased which we’re still working with the service provider to resolve. Meanwhile, the new Raspberry Pi temperature monitor is working well and its automated alerts helped us avoid multiple heat related outages during the transition.

Community Engagement
We recently sponsored the Welcome Reception at The African Internet Summit (AIS) that took place in Kampala in June. We were happy to see many of you there and hope you enjoyed it. We also recently joined the ICT Association of Uganda and look forward to supporting their efforts to promote effective government policy.

Government Relations
As many of you are aware, Uganda’s telecommunications industry is facing an increasing number of regulatory challenges, and the UCC’s recent IXP licensing proposal is no exception. It took a significant amount of time and energy to organize the necessary local, regional, and global response to this proposal and we thank all who supported our cause. We will update you if and when we receive any feedback from the UCC.

For background and more information about this issue, please read our company blog post on the matter: http://blog.uixp.co.ug/2019/06/nationalization-in-uganda-looming.html

Governance Reform
We still have not received any feedback on the draft constitution that we proposed to the community in 2015 and again, in updated form, in 2018.

If you are interested in governance reform, please review this draft and submit feedback (or forward it to your legal teams for the same) so that we can move the process forward. A copy of the latest draft proposal and a summary document can be found here: https://uixp.co.ug/documentation/governance-reform-updated-draft-memarts-2018-12

That’s all for now. If anyone has any questions, please don’t hesitate to contact us.

Nationalization in Uganda: A Looming Disaster

The Uganda Communications Commission (UCC) has proposed a new licensing framework that would effectively nationalize a core part of the country's telecommunications industry; Internet exchange points (IXPs).

Introduction to Internet Exchange Points (IXPs)

This news was communicated to us in a letter from the UCC dated June 7th, 2019. The letter included a draft of the licensing framework and a call for written feedback by July 5th, 2019 (click here to download a full copy).

Based on our analysis, the draft framework would establish a government controlled monopoly which all other market players would be subservient to. It would accomplish this with the following formula:

• Establish a “Designated National Internet Exchange Point” that all other IXPs will be required to connect to (9.j);


• Require government approval of contracts between IXPs and network operators (7.4.b);


• Allow the government to arbitrarily compel IXPs to make operational and technical changes (7.5.c)


• Allow the government to inspect, copy, or remove any data related to any IXP without a court order (7.5.b.i);


• Require all licensed network operators to connect to an IXP (8.2.a).

Technically speaking, this policy would merge all IXPs into a single national peering LAN, with each IXP merely acting as a heavily regulated access point for the enlarged infrastructure. The resulting entity would suffer from all of the classic symptoms of a monopoly as well as significant technical challenges and security risks derived from having multiple operators control access points that form part of the same LAN. As a centralized service, it would also inherently lack the resilience that a diverse array of independent IXPs would provide.

Here are two diagrams to help illustrate the concept:

In addition, the draft framework contains language which suggests that the government intends for the "Designated National Internet Exchange Point" to establish itself by expropriating an existing private operation; namely, ours.

This appears to confirm some of our worst fears about Uganda's new National Broadband Policy; a government strategy document, reportedly drafted in isolation, that seemingly calls for a large-scale nationalization and centralization of Uganda's Internet infrastructure under the guise of infrastructure sharing.

We (and others) have repeatedly warned that such policies would have severe socioeconomic consequences for Uganda and the wider East African region. In this particular case, the UCC's planned regulatory intervention in our otherwise healthy industry has no successful parallel anywhere in the world -- and global experts widely regard the other attempts as textbook examples of regulatory failure.

In light of the obvious risks and highly technical nature of this proposal, we feel that this limited survey of the local Internet community is insufficient validation. Accordingly, we strongly urge the UCC to defer any further work on this project until there is a clear rationale and its viability can be transparently proven with case studies and corroborating input from credible global experts.

Meanwhile, we are preparing to submit detailed feedback to the UCC and will post a copy of our submission here once it is ready. We encourage anyone else that would like to submit feedback to do so through us electronically. We will collect, manually submit, and (unless anyone objects) electronically publish all that we receive in order to promote transparency.

Please feel free to contact us here: board@uixp.co.ug

[UPDATE: A copy of our formal feedback to the UCC can be downloaded here. Our general position is that the creation of a de jure IXP monopoly would be bad for our industry, Uganda, and the region. We argue that the regulatory framework should instead seek to create an enabling environment for competition.

We have also uploaded a number of supporting submissions made by the Internet eXchange Federation (IX-F), the African Network Information Centre (AFRINIC), the ICT Association of Uganda (ICTAU), Liquid Telecom, and a personal submission by Diarmuid O'Briain. A copy of those submissions can be downloaded here.]