Performance notes

Performance of the busybox-w32 shell and other utilities has been tested by timing how long it takes to run the BusyBox test suite. The test suite consists of a set of shell scripts which exercise various features of the build. It is certainly not representative of typical use of BusyBox nor is it comprehensive but it does provide a large body of shell scripts which are known to work with BusyBox.

Timings were obtained by running the test suite in virtual machines. Each VM was given 2 CPUs and 1GB of memory (apart from Windows 10 which had 2GB of memory). The host system had an Intel Core i5-4460 running at 3.2 GHz with 8GB of memory.

As a benchmark the FRP-3532 release was compiled as a native Linux binary on Fedora 31 with the default settings for busybox-w32. The test suite took 18.7 seconds in this case.

Here are the raw results for a 64-bit Windows binary of the same release on a fully updated 64-bit Windows 10 installation:

   196.93 203.71 212.42 215.79 221.03
This is a rather poor performance. Each run is more than ten times slower than on Linux and the times show a steady increase from one run to the next.

Here's a screenshot from Sysinternal's Process Explorer during a run:

The actual work of running the test suite is off the bottom of the display. The interesting entries are those for PIDs 368 and 1460 which together are consuming more than a CPUs worth of processor time (2 CPUs = 100%).

PID 368 is responsible (among other things) for the Program Compatibility Assistant which

monitors applications run by the user. When a potential compatibility issue with an application is detected, the PCA will prompt the user with recommended solutions.
PID 1460 is the Windows Defender anti-malware service.

It appears that each of these checks every instance of an application that starts up. The test suite is constantly creating new processes for subshells and commands, each of which is subject to inspection by these two services. Clearly PCA and Windows Defender haven't been optimised for this situation.

Happily, mitigations are available. You just have to find them.

For Windows Defender go to:

   Settings
   Update & Security
   Windows Security
   Virus & threat protection
   Virus & threat protection settings
   Manage settings
   Exclusions
   Add or remove exclusions
and you'll be able to add an exclusion to prevent the anti-malware service from inspecting every process that runs busybox.exe. (Note that the exclusion is for a process not a file.)

Running the test suite again after this change gave the following raw times:

   146.79 153.01 158.31 165.84 176.14
That's a substantial improvement, a saving of about 50 seconds, but the times still increase from one run to the next. Process Explorer showed that the PCA process was using more and more memory and consuming more and more CPU. When I shut down the console window I'd been using the PCA process spent several tens of seconds at 49% CPU until it eventually stabilised with a reduced working set. It appears that PCA was recording data about each child of the original shell until it terminated, at which point it tidied up.

When the console window was shut down between each run of the test suite the times were much more stable:

   144.82 145.47 145.17 144.58 145.26
Obviously having to shut down the console from time to time to maintain performance isn't an ideal workflow. Unfortunately the PCA doesn't seem to have a mechanism to exclude certain processes. The best that I could find was to turn it off completely. Depending on your circumstances this may not be an acceptable solution.

To turn off PCA run the Group Policy Editor (gpedit.msc) and navigate to

   Local Computer Policy
   Computer Configuration
   Administrative Templates
   Windows Components
   Application Compatibility
Double click on 'Turn off Program Compatibility Assistant' and set the checkbox for 'Enabled'.

Following a restart the test suite ran with an average elapsed time of 133.8s. Still much slower than on Linux but a reasonable improvement on the initial results.

Other observations:


Ron Yorston
20th July 2020