Welcome to Project #2 of the MRG Ongoing Early Life Testing Project.
Project #2 uses 100 malware samples which we downloaded 24 hours before the test which was run on 07/11/09.

The applications tested are in three categories; cloud based Anti-Malware, traditional Anti-Malware and specialist complementary antimalware.

The cloud based Anti-Malware products were:

• Bluepoint Security 1.0.0.75
• Immunet Protect Beta 1.0.18
• Panda cloud Beta 3
• Prevx 3.0.5.10

The traditional antimalware products were:

• A-Squared Antimalware 4.5.0.27
• Avira AntiVir Premium 9.0.0.447
• GData 20.0.1.1
• Kaspersky Antivirus 9.0.0.736
• Microsoft Security Essentials 1.0.1611.0
• Nod32 4.0.468

The complementary antimalware products were:

• IOBit Security 360 1.20.10
• Malwarebytes Antimalware 1.41


Methodology used in this test:

1. Windows XP Professional Service Pack 3 is installed and updated with all important updates.
2. An image of the Operating System is created with internet access.
3. A clone of the Imaged system is made for each program to be used in the test.
4. An individual program is installed with default settings on each of the Cloned systems .
5. On each Cloned system the package containing the samples of malware is placed.
6. All the programs are fully updated.
7. Real Time protection/On Access scanners as well as all other methods of detection/prevention used by various Security Applications are turned on prior to the start of the test.
8. The test is conducted by performing two complete system scans and copying the sample set to and from the test machine twice.
9. Any malware samples detected are removed as it is a given that if they are detected in this way, they will also be detected on execution. (This has been confirmed by several of the vendors)
10. The remaining samples are executed individually. If the malware executes, the next sample is tested on a clean, uninfected machine by following steps 3-7, excluding step 5, above.
11. Any missed samples are tested again (executed)

Test results. Samples missed out of 100:

The cloud based Anti-Malware products:

• Bluepoint Security 1.0.0.75 = 0
• Panda cloud Beta 3 = 7
• Prevx 3.0.5.10 = 8
• Immunet Protect Beta 1.0.18 = 42*

The traditional Anti-Malware products:

• A-Squared Antimalware = 4
• Kaspersky Antivirus = 6
• Avira AntiVir = 9
• GData = 9
• Nod32 = 10
• Microsoft Security Essentials = 15

The complementary Anti-Malware products:

• IOBit 360 = 97
• Malwarebytes Antimalware = 22

* = Please note, we have discovered an occasional fault with Immunet Protect, which can result in a reduced detection rate. On one of the test runs, Immunet missed 9. We have discussed this with the vendor and point out that this result is not representative of the products usual performance.

Please note, this test is the property of Malware Research Group and may not be reproduced without permission.

Could have we a checklist of that 100 malware samples in test used

+++ which sort of malicious software (a category of malware) were missed by the products, and how much high is its dangerousness (of that missed smaples)


Thanks for comprehensive methodology and conditions , according AMTSO (I believe)

What does the * stands for?

Well spotted Woodrowbone. Please see the edited post.

We have spoken with Al Huger about this and their cloud logic will be up and down die to some work they are doing. It is possible the varied results we are seeing is ecause of this.

In Project #3, we may be moving Immunet Protect to the "complementary" section as the vendors state it should be used with a full AV/AM

Another reflection on the test or tests u guys perform here is regarding to A-Squared Antimalware.
The times I have used their portable version to scan and heal infected computers I have to say that a lot more then the infected files would be deleted if I did follow their instructions.

I am surprised if the computers would have started after the cleaning process, thats how many false positives I did encounter.

Do u guys take this into consideration when testing?
I mean if all vendors did turn upp their heuristics over the roof they would all be on topp of the detection list or am I at fault here?

Woodrow

Hi, interesting that you make this observation as we have been looking at FPs recently and only today, I have been in communication with a vendor (not one being tested here) about FP testing. This is something we will be doing in the not too distant future and may even incorporate it as part of this project so as to provide more meaningful data.

We will be providing some deeper detail about the exact methodology we are going to apply to Project#3 shortly, along with reasons and explanations.

We welcome your thoughts and any feedback.

Regards,

Chris

Hello and welcome. For the time being, we will be sticking to the set of applications as they are in Project#2. We will of course continue to run other tests as well and be happy to consider other applications in these.

If you have an application you would like to see tested, let us know, as one of the team could probably put it through its paces for you.

Regards,

Chris.

Hi chris, im really curious for see it.

thank you!

Not to beat on Bluepoint Security, but I don't think having it in this test had any relevance: last time I checked, BS (no pun intended) had no separate antivirus/malware engine and relied solely on whitelisting for detection. While still a cloud-based solution, it's not a true antivirus/malware as it has no detection engine other than its whitelisting mechanism, and, in my opinion, should be tested along the lines of HIPS products like DefenceWall, ThreatFire, etc.
In other words, you tested a product that blindly blocked everything it didn't recognize; it didn't rely on detection engine or behavioral/heuristic scanning like most other products in your test did.
Also, I am curious if A-sqaured was tested with Community Detection option turned on. I'm not sure if it's on by default, however.
Thank you.