Rtmp Load Testing Tool Comparison Chart
I was on the ASP.NET Community Standup this morning and Jon mentioned a new tool for load testing called 'Netling.' This got me to thinking about simple lightweight load testing in general. I've used large enterprise systems like SilkTest as well as the cloud based load testing tools like those in Azure and Visual Studio. I've also used command-line tools like WCAT, an old but very competent load testing tool.
Load Testing Tools to Consider. Depending on their complexity, some load testing tools can be quite expensive. However, there are cheaper options in the market and some are even free for use. I’ve included a mixture of these below for your reference, including a couple of open source options. Loadview by Dotcom Monitor. Free tool for web application load testing that allows for the simulation of concurrent connections to your web application's APIs.
The whole idea with load testing is that you’re trying to mimic your own workloads. If you’re just trying to test a server with generic workloads, start with my post, “How to Check Performance on a New SQL Server.” Single-Query Load Tests. These tools let you write a query, and then run that query lots of times to stress load. RTMP is a live streaming protocol that transmits video files from the encoder to an OVP. Most encoders use the format RTMP. RTMP stands for Real-Time Messaging Protocol. RTMP streaming is a delivery method designed for live-streaming. It’s not a codec, but RTMP commonly uses the H.264 codec. This article will provide a summary of the top API testing tools in 2020 covering both open-source and commercial solutions that testing teams can select to suit their needs. TL;DR: Katalon Studio.
I thought I'd take a moment and look at two tools run locally. The goal is to see how easily I can do quick load tests and iterate on the results.
Netling
Netling is by Tore Lervik and is a nice little load tester client for easy and quick web testing. It's open source and on GitHub which is always nice. It's fun to read other people's code.
Netling includes both a WPF and Console client and is cleanly factored with a Core project that does all the work. With the WPF version you do test and then optionally mark that test as a baseline. Then you can make small changes as you like and do a quick retest. You'll get red (bad) or green (good) results if things get better. This should probably be adjusted to ensure it is visible for those with red-green color blindness. Regardless, it's a nice clean UI and definitely something you'll want to throw into your utilities folder and call upon often!
Do remember that it's not really nice to do load testing on web servers that you don't own, so be kind.
Note that for now there are no formal 'releases' so you'll need to clone the repo and build the app. Fortunately it builds very cleanly with the free version of Visual Studio Community 2015.
The Netling console client is also notable for its cool ASCII charts.
I'm sure that Tore would appreciate the help so head over to https://github.com/hallatore/Netling and file some issues but more importantly, perhaps chat with him and offer a pull request?
WebSurge
WebSurge is a more fully featured tool created by Rick Strahl. Rick is known in .NET spaces for his excellent blog. WebSurge is a quick free download for personal use but you should register it and talk to Rick if you plan on using it commercially or a lot as an individual.
WebSurge also speaks the language of the Fiddler Web Debugging Proxy so you can record and playback web traffic and generate somewhat sophisticated load testing scenarios. The session files are just test files that you can put in source control and share with other members of your team.
I realize there's LOT of choices out there. These are just two really quick and easy tools that you can use as a developer to easily create HTTP requests and then play back at will and iterate during the development process.
What do YOU use for load testing and iterating on performance during development? Let us all know in the comments.
Sponsor: Big thanks to Redgate for sponsoring the feed this week. Could you deploy 1,000 databases? Imagine working in a 70-strong IT team, with 91 applications and 1,000+ databases. Now imagine deployment time. It’s not fiction, it’s fact. Read FlexiGroup's story.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
AboutNewsletter
There are two types of test I’ll describe below. First of all using Apple HLS streams, which is HTTP Live Streaming via port 80, supported by iOS and Safari, and also by Android (apps and browser). Then we have Adobe’s RTMP over port 1935, mostly used by Flash players on desktop, this covers browsers like Internet Explorer and Chrome on desktop. These tests apply to Wowza server but I think it’ll also cover Adobe Media Server.
All links to files and software mentioned are duplicated at the end of this post.
It’s worth noting that you can stick to HLS entirely by using an HLS plugin for Flash video players such as this one, and that is what we’re doing in order to make good use of Amazon’s CloudFront CDN.
For the purpose of testing you may also wish to simulate some live camera streams from static video files, see further down this post for info on how to do that on your computer, server or EC2.
Testing RTMP Live Streaming with Flazr
In this test we want to load test a Wowza origin server itself to see the direct effect of a lot of users on CPU load and RAM usage. This test is performed with Flazr, via RTMP on port 1935.
Assuming you’ve set up your Wowza or Adobe Media server already, for example by using a pre-built Wowza Amazon EC2 AMI. We’re using an m3.xlarge instance for this test as it has high network availabilty and a tonne of RAM – and we’re streaming 4 unique 720p ~4Mbit streams to it, transcoded to multiple SD and HD outputs (CPU use from this alone is up to 80%).
Installing flazr
First up, for the instance size and test configuration I’m using I modified flazr’s client.sh to increase the Java heap size to 8GB, otherwise you run out of RAM. Next up FTP and upload (or wget) flazr to a directory on your server/EC2 instance. Then SSH in and:
The order of parameters does seem to matter in later versions of flazr, but either way this test runs for 60 seconds, with a load of 1000 viewers. Given all the transcoding our CPU was already feeling the pain, but there was no sign of trouble. We managed 4500 before anything started to stutter in our test player from another m3.xlarge instance.
Of course this only matters if you are not using a CDN, but it’s good to know this EC2 instance can handle a lot of HD viewers.
Testing HLS Live Streaming (or a CDN such as Amazon CloudFront) with hlsprobe
Onto HLS streaming, the standard for mobile apps and sites. We have used Wowza CloudFront Formations to set up HLS caching for content delivery, so that we can handle a very large number of viewers without impacting on the CPU load or network throughput of the origin server, and to giver us greater redundancy. Given CloudFront works with HLS streams we are not using RTMP for this test, so we cannot use Flazr again. To test HLS consumption –that being the continuous download of .m3u8 files and their linked .ts video chunks– we can use a tool called hlsprobe, which written in python.
If you’re on a Mac and don’t have python I recommend you install it via brew to get up and running quickly. If you don’t have brew, get it here.
hlsprobe also relies on there being an SMTP server running, not that you need a fully functional one but:
Then to install hlsprobe’s dependencies and hlsprobe itself:
A sample config is linked at the end of the post.
Running hls probe is as simple as this (note the -v verbose mode, you can turn that off once you have it working).
Now if you fire up the Wowza Engine Manager admin interface, you should still see the connection count and network traffic, but the traffic. If you’re testing your CDN such as with CloudFront, you should note that your CPU usage does not increase substantially as you add thousands of clients.
Rtmp Download
Simulating cameras to Wowza via nodeJS
It’s good to be able to simulate live streams at any time, either from your computer or in my case, from some EC2 instances. To do this I’ve written a simple nodejs script which loops a video, optionally transcoding as you go. I recommend against that due to high CPU use and therefore frame-loss; in my sample script I am passing through video and audio directly, the video is already using the correct codecs, frame size and bitrate via Handbrake.
Cek pbb bekasi. The script runs ffmpeg, so you’ll need to install that first:
Edit the js script to point to your server, port, and video file, the run the script with:
If the video completes, it’ll restart the stream but there will be a second of downtime, some video players automatically retry, but make sure your video is long enough for the test to be safe.
These are just a couple of ways of load testing a live streaming server, there are 3rd parties out there but we’ve not had great success so far, and this way you have a lot more control over the test environments.
Links
fakestream.js – NodeJS script to simulate live streams
config.yaml – Sample config for hlsprobe
hlsprobe – Tool for testing HLS streams
Flazr – Tool for testing RTMP streams
OSMF-HLS – OSMF HLS Plugin to support HLS in Flash video players