Showing posts with label sustainability. Show all posts
Showing posts with label sustainability. Show all posts

Monday, 13 January 2025

Endowment Established, Netflix Connected, Akamai Revived, New Services Launched

Message from the Executive DirectorProfile Picture of the Executive Director

2024 was an important year for the UIXP as we took significant steps to secure our long-term sustainability and enhance our service offerings.

We recently achieved a major milestone with the launch of an endowment fund that utilises conservative interest-bearing investments to generate income. This is a key part of our long-term strategy to secure the UIXP’s future; to reduce our service pricing; and to ensure that the UIXP can remain relevant amid looming competition from international players which can offer subsidised services.

We also expanded our service offerings, deploying a Netflix cache with donated cache-fill from Lyca Mobile, and restoring Akamai's cache to service using bandwidth contributions from RENU. In addition, we recently launched a locally hosted Jitsi video conferencing service and an AI social media bot to help improve our online presence. In the background, our infrastructure remains stable and we’ve made a variety of improvements to both physical and digital security.

Despite this progress, the year was not without challenges. Google's worldwide withdrawal from remote peering sessions impacted our ecosystem, while local market conditions such as high taxes and the lingering Facebook ban continue to suppress growth.

Looking ahead, we aim to grow the endowment fund, implement a new ERP with a self-service portal, expand our portfolio of locally hosted services, and continue our involvement in regional Internet communities and initiatives.

I encourage everyone to read the full report below for more details. As always, we are grateful for everyone's support and look forward to working together for the good of the Internet in 2025!

Kyle Spencer,
Executive Director


Endowment Fund Launched

We are proud to announce that we have launched an endowment fund in order to promote the UIXP’s long-term sustainability and reduce its dependence on traditional service fees.

The endowment is based on conservative interest-bearing investments which, starting this year, will cover 25% of the UIXP's annual non-salary operating expenses. The initial installment was made possible by years of savings, conservative financial management, and multi-year prepayments from some of our larger peers, including Meta.

We aim to grow this fund over time with the hope that it will eventually cover 100% of our total annual expenses. This would allow us to offer free or near-free services which would both facilitate the growth of Uganda’s digital economy and help the UIXP stay relevant in the future as our region faces increasing competition from international IXPs that can offer subsidised services.

 

Free Ports & Raxio Discounts

We would like to remind everyone that we offer free 100 Mbps ports at Raxio with no cross-connect charges. We also offer a 6-month 25% discount on all new services at Raxio so, if you would like to connect, upgrade your port, or learn more, please contact us.

 

New Value Added Services 

  • Encrypted Video Conferencing: We recently launched a free web-based open-source video conferencing system which offers features similar to Zoom and Google Meet. All user traffic is encrypted and routed via the UIXP which helps to protect users from mass surveillance. We encourage everyone to give it a try and send us feedback: https://kafunda.uixp.co.ug

  • AI Marketing Bot: We recently introduced an AI marketing bot that posts occasional promotional updates on Twitter/X. This complements our manual posts and helps to ensure more consistent engagement with our community and the public.

 

Peering Updates 

  • UIXP Traffic Graph for 2024
    Summary: We began and ended 2024 with 32 connected networks and approximately 40Gbps of peak traffic. There was some churn during this period with some networks disconnecting and others connecting. There was also a mid-year traffic depression following the disconnection of Google’s remote peering session with a subsequent boost in traffic in November caused by the restoration of the Akamai cache (see below for more details).

  • Raxio: We now have six networks connected at the Raxio carrier neutral data center with another connecting soon. As a reminder, we offer free 100M ports and significant discounts on new services at this facility so please contact us if you would like to connect or migrate your network.

  • IPv6 Adoption: 33% of all networks (10 out of 30) are peering with our route servers via IPv6 though most traffic is still exchanged via IPv4. We encourage all networks to transition to IPv6 and we are happy to provide support where possible. Click here for a live list of networks peering via IPv6 in Uganda.

  • New Peers: Netflix and Kampala Siti Cable [AS328727] connected to the UIXP in 2024, though the Netflix cache is connected to the UIXP via the UIXP’s ASN [AS328998].

  • Netflix: We deployed a Netflix cache this year, made accessible to all connected networks through our route servers, with donated cache-fill from Lyca Mobile. However, this cache requires manual activation. To connect, networks must append their BGP announcements to the UIXP route servers with the community string 40027:4000. For more detailed instructions, read the documentation on our website or contact us via e-mail.

  • Akamai: After a period of downtime, the Akamai cache was restored on an experimental basis using donated cache-fill from RENU. However, its output is currently limited by the amount of donated cache-fill bandwidth that is available as well as the capacity of the cache cluster hardware.

    Traffic from this cache is automatically served to connected networks via the UIXP route servers. However, Akamai’s systems automatically moderate traffic distribution based on a variety of factors (including cache capacity) which can influence whether or not your particular ASN gets traffic from this cache cluster.

    We are working with Akamai to upgrade the cache cluster hardware and expect this to occur in Q2 2025. However, in order for the upgraded cluster to increase its output, we will need more cache-fill bandwidth. If we are unable to get additional donated bandwidth, we may try to implement a cache-fill cost-sharing model.

  • Google: Due to a global policy change, Google withdrew from all remote peering sessions around the world. This impacted the UIXP as Google was connected via a long-distance fibre link to their nearest PoP in Mombasa. We believe this decision was largely driven by the cost and complexity of maintaining these links.

    We explored the possibility of either (a) inheriting the management and cost of the transport link to Mombasa in order to maintain Google’s remote peering session, or (b) replacing Google's remote peering session with a shared Google cache but, in this case, we would need to source cache-fill IPT. Both options were difficult logistically and/or financially. In either case, we would not be able to serve Google traffic for significantly less cost than our peers can already get it via wholesale IPT and, in the remote peering scenario, Google's traffic would still originate in Mombasa so there would be no latency benefit.

 

Facility Updates 

  • Our switching infrastructure and data center facilities continue to operate reliably. We’ve also strengthened physical and digital security to ensure a safer and more resilient service environment.

  • As a reminder, we send technical alerts about significant events, planned maintenance, and unplanned outages to the UIXP Techies mailing list. If your network is peering but is not a member of this list, please contact us to get added.


Accounting & Tax Updates 

  • In 2023 we experienced a number of account delinquencies, particularly from smaller networks, which we attributed to challenging market conditions. Throughout 2024 we worked with the affected networks to reduce these balances and are happy to report that almost all have returned to good standing. We sincerely appreciate everyone for honouring their commitments and will do our best to ensure that we can continue to be accommodating should similar challenges arise in the future.

  • Our efforts to migrate to a new open-source accounting system were delayed and did not happen in 2024 as we had hoped. However, this remains on our agenda and we hope to undertake this project in 2024. Our goal is to streamline accounting and operational processes through increased automation while providing networks with a convenient self-service portal.

  • We remain in good standing with URA and have not faced any significant compliance issues during the past year.

 

Market Challenges

  • The 2021 Facebook ban continues to suppress overall utilisation of the Internet and our interconnection service. It continuously inflates the cost of Internet service delivery by forcing users and networks to import Facebook traffic from international sources rather than our local cache. This makes Uganda’s Internet more expensive and its market less attractive for regional interconnection & colocation relative to neighbouring countries.

  • We remain concerned by the increasingly high taxes levied on Internet services which now account for over 50% of the total cost of Internet access in Uganda. Like the Facebook ban, this hinders demand and deters investment in Uganda’s digital economy.

 

Community Engagement

Thursday, 23 February 2023

Disaster Recovery: CDN Outages, Financial Challenges, and the Way Forward

Message from the Executive Director

2022 was difficult for the UIXP.

The year started off strong. In January we had 27 networks exchanging 35 Gbps of peak daily traffic. In February we connected our management network to our route servers in order to deliver new low-latency services. In March we deployed a dark fibre link between Raxio and Communications House to facilitate multi-site peering. In May we helped Google extend their global network to Uganda to facilitate the distribution of their traffic via the UIXP.

By June our peak daily traffic reached a record high of 50 Gbps and we planned to start offering free 100M ports – but then a series of external problems destabilized and eventually disabled both Google and Akamai’s peering amid a lingering national Facebook ban. This multifaceted environmental disaster reduced our peak traffic to 6 Gbps and negatively impacted our income immediately after we incurred new costs related to our multi-site expansion.

We are doing our best to help Google and Akamai resolve their issues while we work with Netflix and Meta to deploy new caching systems. We have also assumed a more conservative budgetary posture in an effort to maintain financial stability during this difficult period. We faced an unusually high number of delinquencies at the end of the year and we will be working to address this in the coming months.

In the background, we continue to contribute to community initiatives at the local, regional, and global level. This includes the Uganda Network Operators Group, ISP Association of Uganda, ICT Association of Uganda, the African Network Operators Group, the African Peering & Interconnection Forum, the African IXP Association, and Internet Exchange Federation.

I encourage everyone to read the full report for more details. As always, we are grateful for everyone's support and look forward to working together for the good of the Internet in 2023.

Kyle Spencer,
Executive Director


Raxio Promotional Discounts

As a reminder, significant discounts are available to all Raxio customers including a free cross-connect to the UIXP and a 6-month 25% discount on all exchange services. If you would like to connect, upgrade your port, or learn more about our services, please contact us.


New Members & Connections


Google (AS 15169), Raxio (AS 328821), Group Vivendi Africa (AS 36924), and the Uganda Internet Exchange Point management network (AS 328998) joined the exchange in 2022. This diverse array of content, enterprise, and access networks are all connected to our new node in the Raxio data center, bringing the total number of peers at that location to five.

Looking ahead, we intend to engage a broader range of Raxio customers in order to expand our membership. The more networks that connect, the more valuable the exchange is to everyone.


Technical Updates

We upgraded our inter-site link to 100 Gbps with geographic redundancy in March 2022 in order to accommodate the anticipated flow of Google traffic between Raxio and Communications House where the majority of our customers are connected. C-Squared is providing this link and we are grateful for their reliable and attentive service.

We experienced multiple broadcast storms in 2022 that significantly disrupted some of our member networks. In response, we are gradually implementing technical changes to reduce the possibility of future disruptions. Most notably, we now restrict all ports to a single MAC address.

As a reminder, we send technical alerts about notable changes and foreseeable outages to the members-only techies@uixp.co.ug mailing list. If your network is peering but is not a member and would like to be, please contact us.


Content Networks: Outages & Upcoming Deployments


Four content networks are connected to the UIXP but only YoTV is actively peering. Facebook, Google, and Akamai have all been disabled by different circumstances detailed below:

  • The Facebook cache (AS 63293) stopped serving traffic in January 2021 after a government ban forced network operators to block access to the service. This reduced local bandwidth production by 30% and damaged Uganda’s reputation in the global telecommunications industry. Meanwhile, many users continue to access Facebook via virtual private networks (VPNs) which force network operators to import traffic from more expensive international sources. In other words, the Facebook ban increased the cost of service delivery, reduced overall performance, and steered investment to neighbouring countries which are perceived to have more attractive and stable policy environments.

  • Google (AS 15169) established a remote peering session in May 2022 after decommissioning their prototype “self serve” cache (AS 36040). However, Google’s regional transport suppliers were unable to deliver a stable link to Mombasa despite six months of joint troubleshooting. The session was ultimately disabled in December 2022 due to extreme instability and poor quality of service. Google is now working to change transport suppliers and we are actively supporting them with this process.

  • Akamai’s cache (AS 20940) stopped serving traffic when Google’s instability began because it forced the cache-fill donor, NITA-U, to re-route Google related user traffic via international links. This overloaded their capacity and the resulting congestion forced the Akamai cache offline. NITA-U is working to upgrade their international capacity but the procurement timeline is uncertain so we are searching for a new donor and may need to implement a paid access model until Akamai can justify paying for the cache-fill themselves.

On a more positive note, Netflix and Meta are shipping us new edge caches which we hope to receive next month. Our goal is to bring them online in the second quarter of this year:

  • Meta plans to procure their own cache-fill and will peer with our route servers. Their new system will reportedly make it easier for networks to distinguish between traffic related to Facebook and other Meta properties such as Instagram, Oculus, and WhatsApp.

  • Netflix requires a cache-fill donor but their requirements are relatively easy to fulfill due to the static nature of their content catalogue. If your network has 500 Mbps of idle capacity at night and would like to donate it to the UIXP community for this purpose, please contact us to discuss the requirements in more detail.


Environmental Challenges


The 2021 Facebook ban continues to suppress overall utilization of the Internet and demand for our regional interconnection service. It also increases the cost of Internet service delivery by forcing network operators to import Facebook traffic via international VPNs rather than sourcing it locally from our cache. In general, it makes the Internet more expensive and constricts Uganda’s commercial gravity relative to neighbouring telecommunications markets.

We are also concerned by the high taxes levied on Internet services which now account for over 50% of the total cost of access. Like the Facebook ban, this also makes the Internet more expensive, suppresses overall utilization, and reduces demand for our service.


Financial Challenges

We unfortunately ended the year without any surplus or savings due to an unusually high number of delinquencies. The problem is limited to a minority of networks but the loss of income is significant due to the thin margins of our “cost-recovery plus” sustainability model.

While we are sympathetic to the difficulties that network operators face in our market, it is in everyone’s best interest for everyone to keep their accounts up to date. Income instability impacts our ability to pay suppliers, maintain our infrastructure, and plan for the future.

That aside, our cash reserves remain intact and we remain in good standing with URA despite occasional conflicts caused by erroneous assessments, improper penalties, and onerous bureaucracy.


Community Engagement

UGNOG: The UIXP continues to facilitate donations for the Uganda Network Operators Group (UGNOG) until they can formalize and register a bank account. Any organizations that wish to sponsor UGNOG should notify us by sending an e-mail to accounts@uixp.co.ug CC isabel.odida@gmail.com.

ISPAU & ICTAU: The UIXP is a member of the ISP Association of Uganda (ISPAU) and ICT Association of Uganda (ICTAU) and works through both organizations to promote community development and good policy outcomes for our industry.

AFIX, AfPIF, & AFNOG: Our team remains deeply involved in various voluntary leadership and support roles in the African IXP Association (AFIX), the African Peering and Interconnection Forum (AfPIF), and the African Network Operators Group (AfNOG).