Engineers are always looking for new and better ways to stream large events. Whether you're streaming a live event on Facebook or YouTube Live, or using your custom-built video streaming infrastructure, there's always a chance that something could go wrong and cause your video feed to stop working and crash!
But, let's not let that happen, shall we?
This article will explore some of the top reasons why live stream crashes happen so you can be prepared next time!
Weak Wi-Fi connection / Internet Connection
Using a solid internet connection is a critical requirement when live streaming either from your phone, laptop, a concert, or an event at your church. In some situations, your camera will directly feed video to your video processing computer (local or cloud).
In others, your camera will probably communicate with a computer where additional processing occurs before the video is transcoded and delivered either locally or in the cloud.
In either situation, if the internet connection is weak and it drops, then the live video input to the rest of the pipeline gets interrupted. The live stream interruption will kick off a chain reaction of starving the encoder, the packager, and ultimately the video player, which will start buffering and annoy the audience.
So, always test and ensure that your connectivity is top-notch!
Secondly, apart from a reliable and robust internet connection, ensure that you have multiple networking options (Wi-Fi, Ethernet, or a 4G hotspot/dongle) configured, tested, and ready to use in the worst-case scenarios.
Poorly Configured Encoder
A poorly configured encoder can genuinely ruin a live stream since it is a vital cog in the live streaming pipeline. An encoder takes in a live video stream and compresses it to produce one or more input versions in various bitrate and resolution combinations. E.g., the input might be 1080p @10mbps, and the output from the encoder would be 480p @ 1mbps, 640p @ 1.2 Mbps, 720p @ 2mbps, and 1080p @ 4mbps.
So, what can go wrong with the encoder? For one, if the encoder is configured wrong, then the encoder might crash due to out-of-memory/space/IO issues - apart from producing poor-quality video.
The encoder's configuration depends on the server configuration, egress capabilities, the average bandwidth where the audience lives, etc.
Some examples of poorly configured settings include setting a very high bitrate for the target geography and bandwidth conditions, very high framerate, too many ABR profiles, etc. All of these can overload & crash the encoder by causing it to run out of memory/space/IO-bandwidth.
In addition to dedicated encoder software or hardware, it is quite common to see people use browser-based encoders nowadays. These encoders can also lag and create issues, especially if the computer has multiple programs running in the background that eat into the available RAM.
Not using a CDN
A CDN or Content Delivery Network caches your videos to servers around the world. When a user requests a video segment, the CDN serves it instead of reaching your web servers. Using a CDN is very important in video delivery because a significant spike in traffic can easily crash a web server that isn't sitting behind a CDN. Using a CDN also protects your web/origin servers from DDoS attacks that can easily crash your live stream.
Not using Redundant and Properly Sized Streaming Servers.
Do you plan on delivering or streaming the video without a CDN? If yes, you need to ensure that you have sized your streaming servers appropriately after "guesstimating" the traffic your event is likely to receive. You can "guesstimate" this by either looking at traffic numbers from previous events or similar events from your competition.
Using the traffic numbers and the length of each segment, you can arrive at a rough estimate of your live traffic and use that to size your servers.
It's wise to have a standby server (with hot switch-over) and a load-balancer in front of the streaming endpoints to distribute the incoming traffic. Redundant servers will ensure that the incoming traffic is allocated correctly to your streaming servers with the option of diverting traffic to other servers in case one of them starts to misbehave.
Honestly, if all of the above sounds too difficult, you should consider using a CDN and avoid the hassle and complexity of DIY.
Redundant Encoders, Ingress Streams, and Path Redundancy
Redundancy is a simple but powerful concept that can protect your live streams from catastrophic failures at different touchpoints of the streaming pipeline. Let's see how, next.
Encoders: It helps to use a redundant encoder configuration - i.e., use another encoder ready to switch over if the first one fails (hot-switching). Such designs do not require manual intervention to switch the encoders in the event of failure.
Ingress: Along the same lines as the encoder, it's always safe to produce a redundant stream from your streaming software (or cameras) to your encoders. Redundancy ensures that there is always a video feed to your encoder even if one feeds fails.
Egress: Finally, the output of the encoder can also be made redundant on two different paths (different IP addresses, ports, etc.)
Failure in Authentication Services
Authentication services are potential bottlenecks in TVOD (PPV) and SVOD services as they can get overwhelmed by sudden traffic spikes. Consequently, these services might authenticate users very slowly or even crash, thus preventing your audience from watching the event.
An apparent root cause for such problems is that most of your audience joins simultaneously, i.e., at the start of an event, resulting in a traffic spike.
Services can avoid these traffic spikes by providing updates or push notifications to the subscribers and encouraging them to log in early for pre-event programming. Such soft approaches work well in preventing authentication spikes.
Not using a multi-CDN configuration.
Suppose you plan to live stream a significant event to a geo-distributed audience with high traffic/volumes. In that case, it is worth exploring a multi-CDN distribution strategy if you are already using a CDN for streaming.
As seen during Fastly's 48 min downtime in June 2021, several high-profile websites like StackOverflow, Reddit, Vimeo, and others went down and were not reachable. All of this occurred due to a bad service configuration made by Fastly!
Websites and services could have avoided the downtime by using multiple CDN vendors to deliver their content instead of pinning their hopes on just one!
A multi-CDN architecture can protect your streams from events affecting one of the vendors, protect you against DDoS attacks, improve your streaming performance, and reduce your streaming costs by taking advantage of cost differences between the different CDN vendors.
Warming up the Live Stream’s Cache
Cache warming is a technique where video segments are pre-emptively loaded into the CDN cache even before anyone requests them. Cache warming is a technique that works well for VOD where the video is available in advance.
But, in live streaming, you can't cache something that hasn't occurred yet, right?
In Live Streaming, one strategy is to ensure that your viewers (players) are all streaming the same segment of video and that the CDN or the caching layer pre-fetches (warms) itself with the next segment(s) - pre-emptively if they are already available and haven't been pushed to the CDN's cache yet.
Cache warming is beneficial when re-streaming a pre-recorded event (as a live event). The first couple of minutes of video can be cached and served via a CDN or a caching layer.
However, incorrectly warming the cache is also a problem that you should be aware of. If you run a script to warm the cache from your local computer in your office, you might be warming only the PoP closest to your office. But if your audience is primarily from a location served by another PoP, they could be starved of content due to cold caches. So, it helps to create an intelligent cache-warming strategy.
Unscalable Digital Rights Management (DRM) Infrastructure
It is imperative to use DRM to protect your live streams from piracy and illegal restreaming. And, when you do decide to use DRM, you need to test it to ensure that your DRM vendor is reliable and has a well-engineered system.
Like how we cautioned you against traffic spikes affecting your streaming server and authentication systems, it is likely that your DRM vendor could be overwhelmed as well. If your DRM implementation or vendor cannot scale in sync with your traffic, your audience won't watch the live stream, will eventually get frustrated, and leave the event.
Another advantage of using DRM is to keep illegal streamers at bay! Suppose a hacker gets hold of your streams and restreams them to a large audience illegally. In that case, the rest of your infrastructure will be overwhelmed quickly because it would not have been provisioned for illegal traffic!
At BuyDRM, we have decades of experience dealing with large-scale deployments of Digital Rights Management systems and recently served more than 57 million licenses in a single day using our KeyOS multi-DRM platform.
Contact us to learn more about how we can help you secure your assets using our studio-approved multi-DRM and watermarking platform KeyOS.
Here is more content that you might be interested in:
- [BLOG] Speed Up Your Streaming With Low-Latency DRM
- [BLOG] Multi-DRM Best Practices for Live Sports Streaming and Other Live Events
- [BLOG] Every Stream Unique: The Rise of Watermarking with DRM for OTT Streaming
