It is my annual get away and I find myself in the Cayman Islands, listening to one of my favorite BBC radio shows “Liza Tarbuck” streamed live from the UK. We sometimes take for granted the sheer logistics that it takes for the mutterings of Liza talking into a microphone, to then up popping out on my Bluetooth speaker, on a sandy beach on the other side of the world, for free.
I thought why not deconstruct this technical marvel and shine a light on some of the steps that some may not appreciate or even realize. Come with me on this amazing journey to see what the “inter” in “internet” really mean.
#1 Content Generation
Liza sits in her studio in the UK, doing what a DJ does, coordinating the show with a mixture of music and talk. Talking into a microphone, her voice is converted from analogue into digital, which ultimate results in a stream of 1’s and 0’s that is then relayed on to a broadcasting server. There will be a whole host of software going on here, as noise is removed, levels are equalized and what have you (interesting aside – this is the world I started out my software career, writing embedded software for professional audio hardware for Harman Kardon).
The broadcasting server, will store this stream (for later playback) and then relay it out to the various relay servers dotted around the globe. An organization the size of the BBC will use a CDN (Content Delivery Network) and at the time of writing it is Fastly to get it’s content as geographically as close to its consumers.
A CDN is a technique to reduce network traffic by relaying, or proxying, data as close to the users as possible. That way not everyone is connecting from where they are back to the UK (in the case of the BBC).
The audio stream will be broken up into individual TCP packets, no larger than 64KB in size, and sent using high-speed backbone networks to the Fastly CDN. Even getting the content to the CDN, can involve many hops, from being routed through the BBC internal network, as it goes through various routers and hubs, from cable-to-cable, mixture of copper and fiber as it leaves the building out onto the global network.
#2 Delivery
Once it leaves the building, it will do down one of the peering ISP’s that the BBC will contract. This will be the first step out on the internet. The BBC will pay for its bandwidth connectivity just like the rest of us for our home; except their bill will be significantly higher as they will be demanding a lot more assured bandwidth that the average home user.
Once it gets to a peering server, they will then handle the relay/delivery to each end user. As each user, using their BBC Sounds Mobile App (or the website), tunes into Lisa, they are routed to the nearest CDN node.
#3 Peering
What is happening under the covers a connection is made out to https://r.bbci.co.uk. This domain, will resolve to an IP address using DNS. Instead of always returning the same IP address, the service will, based on the IP making the request (i.e. your address) give back the nearest Fastly node to connect to.
To see which one I am closest to, I can open a terminal/cmd window and run a ping to see which IP address is returned.
With that, we can put it in a tool that lets us map the approximate location of an IP address – https://whatismyipaddress.com/ip/199.232.208.81
California – that is where my app will be connecting to start receiving a stream of audio packets from Lisa sitting in the UK. Now given my app connects to a peering node of Fastly, I can’t trace the path of packets from the BBC to California. But I can from where I am now to California.
#4 To California
Using the tool tracert/traceroute, we can trace the route of each TCP packet as it moves through the network. The internet is just a connection of networks (hence the “inter”) that relays packets of data from network onto another.
This illustrates all the different networks my packet moves along to get to the final destination in California. We can do a reverse lookup on each IP address (https://whatismyipaddress.com/ip/209.27.60.97) to get a sense of where each network is located.
What we first discover is that we need to get out of the hotel network which then relays onto the Island’s main ISP. We go through a couple of hops there before we find ourselves going over to Miami in the USA. How do we get over there?
Today most of the Internet is routed through undersea cables and we can have a look at where those cables are laid using the site https://www.submarinecablemap.com/. We can see the Cayman Islands are connected by two cables – one to Miami and another to Jamaica and given our traceroute utility determined our packets are headed to Miami we can see our data is going via Maya-1 cable.
The website also tells us where the cable terminates on land – in our case Half Moon Bay. Interestingly, Half Moon Bay is actually only a few miles away where I actually am, but because of the networking of the island, my packets bounced from side of the island to another.
Once in Miami, the packet is routed through 3 more networks before it reaches California.
#5 The Last “Mile”
My app has successfully connected to California, and is now receiving data packets (in 64KB chunks). Before we celebrate success we have a few steps still to go.
Imagine sending a handwritten letter through the post. You write the letter, put it in envelope, address it, put a stamp on it and send it. The letter, after some time, will reach its destination. The internet operates on the same principal. However, each envelope can only hold so much data. What happens if what you are sending is more than can fit in an envelope?
It is split up into multiple envelopes, with a sequence number, and all sent separately. As such, they can arrive out of order (most likely), because the network (or your Post Man) does not care about the order of the envelopes.
My phone, will be receiving Lisa’s packets individually, and before it can do something with it, it must ensure the sequence is correct. Not doing this, would make for a very disjointed audible experience as packets would be played out of sequence.
By the time it gets to the BBC Sound App running in my app, the underlying network software in the phone, has re-ordered the sequence.
Buffering: We’ve all experienced this at some point, streaming video, audio or a web conference. What does it mean though? The most common reason is that the expected packet for a given sequence number has not yet been received. As soon as that magical packet comes in, it all seems to work for a period of time. The reason is that, the app has already received the subsequent packets, but was waiting on that particularly one. Some apps, will gracefully error and simply jump over that and go to the next sequence of packets, resulting in a jumpy experience.
#6 To Audio
So our packet of data, has made it all the way from the UK, popped over the Atlantic, gone to California, then found itself back to Miami, headed under the sea, popped up on the island and made it to your device. Now comes the dreaded Bluetooth connection. With the data stream in the phone, it now has to be relayed to the Bluetooth speaker, that will turn the steam of 1’s and 0’s into something the human ear can interpret.
Can you imagine taking that length of a journey, only to trip up at the last, literally, few feet? Bluetooth is wireless, on a specific frequency (2.4G), that can be good for 1,000 ft, but we all know in reality, it’s really only 20ft. My little Anker speaker, sitting on the beach, is responsible for delivering the sound of Lisa after a very long journey.
Summary
Think about this whole chain of events for a minute. The internet makes the world feel very small. It is a wonder it works at all, but for it to work in real time is nothing short of miraculous.
This is one of the unsung wonders of the world – little packets of data, are flying all over the planet, taking milliseconds (notice the time in the traceroute output above), and we take this for granted. Even as I write this blog entry, sitting on the Cayman Islands, the webapp auto-saves my writing, which means small packets of data are heading under the sea, competing/waiting for all the other traffic leaving the island as it heads over to the servers at WordPress.
28ms from Cayman Islands to California, via Miami. 0.028 of a second, to route data from my hotel here in Cayman, as it goes under the sea, converted from WiFi, to copper, to fiber and back again. All this technology, all this ingenuity, and cost, and what use do I put it to? Listening to the ramblings of a BBC Radio 2 DJ while sitting on a beach, trying not to get burned (failed by the way).
I often wonder what the likes of Marconi (who was the first person to send data over the Atlantic) would think of our modern day Internet. Thanks to him, my ass can sit on a beach, listening to music from across the planet and not pay a single cent for the pleasure.
So the next time you complain about buffering or interruptions in the network, pause for a moment and appreciate all the moving parts that has been put in place to make this even plausible.
One book I would recommend is “The Victorian Internet” by Tom Standage – he tells the story of how all this is a reality and evolution that has happened to get to where we are.
PS @Lisa .. we thank you (and Pauline) for your weekly output
Alan Williamson 