MRG has been conducting flash tests for nearly two years now. The purpose of the tests was to give a basic indication of a products performance against zero day threats over a period of time.
Whilst the tests used only a single sample each time, we attempted to ensure their validity by using samples which were found on active URLs and by using IE as an infection vector. Each sample used is a single snapshot of a pool of some hundreds of variants of that specific malware type.
The flash tests have become quite popular among users and some vendors, so in an effort to increase their relevance, we increased the number of samples used from one to four. As of the 29th of August, we will be introducing significant changes to the tests to further increase their validity.
The new flash tests will be run as two separate tests. We will continue with the dynamic tests, using samples from live URLs with IE as the infection vector, however, these will now be run once each week with eight samples per test.
To help give greater statistical relevance, we will include a static component to the flash tests. Twice each month, we will test using 100,000 malicious samples which are less than 72 hours old. Whilst static testing does not always assess efficacy as accurately as dynamic, it remains a convenient way to get a loose indication of performance against a large number of samples.
We will continue to run the dynamic tests using the existing cohort of security applications, however, vendors who are not existing clients will need to contact us to ask for their product to be included in the bi-monthly static tests.
We welcome input / feedback from users and vendors concerning the new flash tests. Please feel free to contact us with your comments.








