MRG will shortly be starting an ongoing testing programme for a cohort of ten anti malware products. Every day the applications will be tested against 250 random samples which emerged within the preceding 24 hours. Each day, a new batch of 250 samples will be used to test the applications, along with the samples from the preceding two days.

The cohort will be composed of five cloud based and five traditional anti malware products. The purpose of this testing programme is to give an overview of which products detect new threats most effectively and to compare the different speed at which the products develop detection for early life samples.

The daily testing will be on demand, so we will express the usual caveats concerning this limitation and in order to add another dimension to the testing, will, at regular periods conduct live infection prevention tests using samples missed during on demand scanning.

As a precursor to the above programme, we thought we would put the five cloud based products through an informal on demand test.

The products tested were:

• Bluepoint Security
• Hitman Pro
• Immunet Protect
• Panda Cloud
• Prevx

The test was conducted on 04/11/09 and made use of the most current versions of the applications at that time. The malware sample used was 500 random samples which were new and received within 24 hours of the test being conducted.

The testing was conducted using fully updated XP Pro SP3 VMs, with live internet connections.

The percentage of malware detected by each application was as follows:

1) Hitman Pro = 66%
2) Bluepoint Security = 60%
3) Prevx = 29%
4) Panda Cloud = 15%*
5) Immunet Protect = 7%*

Please note, applications with * are beta products and this should be taken in to consideration when comparing results.

To give some point of comparison, we tested Microsoft Security Essentials as well, which detected 24%.

It is important to remember that the samples used were very new and it is to be expected that the majority would be missed in an on demand scan as there can be no behavioural analysis etc, which would allow the applications to improve detection. This said however, we feel Hitman and Bluepoint did very well indeed and we look forward to seeing how they perform in our long term project.

Regards,

Chris

We applaud MRG for being one of the first testing organizations to step into the cloud testing arena!

Keep up the good work guys!

As an addendum, we just ran the same sample by A-Squared on a 22 hour old VM, with no internet connection (to ensure the signatures were from the same time as the cloud apps were tested. A-squared detected 68% of the sample.

As a matter of interest, we will run all these apps against these same samples within the next 3-4 hours to test for improvement.

Regards,

Chris

It has come to our attention that with reference to this test, Prevx have implied in their support forum, that our methodology is incorrect and that we do not understand how Prevx works.

In the post they state:

“We haven't gotten any details regarding the test samples so we honestly have no idea if they actually are malicious”

“Prevx's right-click scanner provides only a very small fraction of the detection/protection which Prevx fully provides”

“Testers trying to test Prevx on-demand are incorrect in their methodology and clearly don't understand how Prevx works”

Whilst we would agree, very few people outside Prevx fully understand how Prevx works, we would suggest we do in fact have a good understanding of the product, having tested it behind the scenes for some time (on demand and live infection tests) and also having performed beta testing for a number of releases.

Prevx has provided MRG with licenses so we can test the product in our VMs and I personally liaise with them regularly to discuss results.

MRG is fully aware of the limitations of the standard right click scan performed by Prevx and understand it does not employ all the detection techniques the product has available and that in order to perform a more thorough test, a full system scan is required.

I spoke with Prevx yesterday via Windows Messenger to discuss the results of the test. During that conversation:

1) We confirmed that we had tested the samples using several complete system scans and had not relied on a right click scan. We even detailed the exact numbers detected with each subsequent full system scan, resulting in the total detected.

2) We sent them 25 randomly selected pieces of malware prevx had missed in the sample so that they could analyse them and have an understanding of its makeup.

3) Prevx confirmed receipt of these and stated they were analysing them and indeed gave us feedback on them.

From this, we can prove that shortly after the test was conducted, Prevx:

1) Knew MRG had not relied on a right click scan to conduct the test.
2) Had received a sample of the test malware.

As a matter of routine, we keep logs of all Messenger, Skype and email communications and are happy to provide a full transcript of the conversation, with timestamps, if Prevx wants to try and deny any of the above took place.

MRG has been careful to make clear in these informal tests that on demand testing has limitations. Nowhere have we stated or implied that on demand tests represents exactly how well an application protects a system –this is why we conduct live execution tests as well.

We have contacted Prevx and stated that in the light of our communication with them yesterday and the various caveats we make in the tests and made clear we feel their statement in their forum is unreasonable and have asked them to amend it. Unfortunately, they have not done this and we have not received any communication from them, therefore, we have been forced to detail the information above in order to prove our case.

Even though the tests are unofficial, it would be nice if two things were made a little clearer. I know your time is valuable but this may help to clear up confusion in future quick tests.

1). Make clear if programs were tested with default settings, if settings were modified. which settings.

2). Briefly how the scan was conducted (right click, full system scan, quick scan, folder scan etc.)

Just our 2cents..

Thanks!

Thanks for the feedback.

As you state, these are informal tests, so we have not outlined the methodology in detail, however, I can confirm that:

1) All applications were tested with dedault settings

2) Where right click scanning was available it was used, with the exception of Prevx, where a full system scan was used following an initial right click scan - as we had been informed by Prevx that right click scanning did not use as many detection techneques as a full system scan. (they tell us they are considering including the methods used in a full scan, in the right click scan at a future time)

3) For applications with no right click scanning, a full system scan was used.

4) In the case of Immunet Protect, the samples were loaded in to the system and in so doing, detected by on access scanning. The samples were repetedly copied to and from the system in order to ensure none were missed. This method was used as no malware was detected when performing a system scan.

Best regards,

Chris

In the future we will disclose all the details of our unofficial tests, we will even create videos so that everybody is able to see what we are doing.

Great, thanks guys!

The approach to security has changed a lot in the last few years where in days of old the on demand scanner was the workhorse of the AV system because a lot of infections made it past the on access scanners the on demand scan had to try to find and remove most infections.

Today we have in place much better on access scanners with behavioral abilities and AV vendors are making this the forefront of their systems as it should be prevention is the best cure then if for some reason the on access prevention fails let's see if the on demand scan picks it up.

For those of us who deal with and understand this stuff no big deal but Prevx is worried about the perception of their product by the general public based on just the demand numbers which I understand people just need to educated a little more about how security applications work and not to choose a security application based on just the on demand detection rates which can be deceiving as to the products true ability to actually protect you and but these numbers are the ones you usually see posted everywhere when you start comparing products hence the point of this and other forums to discuss and educate.

Wilders thread with unkind words about this

http://www.wilderssecurity.com/showthread.php?t=260781