Welcome to Project #1 of the MRG Ongoing Early Life Testing Project.

The purpose of this and the projects to come in this section, is to assess the effectiveness of a set of antimalware products against early life threats.

MRG, like other security organisations, receives a large number of zero day and early life samples every day. The majority of our behind the scenes testing and research has focused on this early life malware and the team has become almost desensitised to the fact that a good proportion of samples in our lab go undetected by nearly all security applications.

Whilst we must point out that a good proportion of what we see does not seem to circulate in the wild, there is no escaping the fact that it is zero day and early life malware that represents one of the greatest threats to IT users and systems.

2009 has seen a dramatic explosion in the number of unique samples of malware and the challenge security vendors face is to create signatures and / or devise other systems to counter this growing threat at a rate that at least keeps up.

There has been an increase in the number of “cloud” based antimalware applications and applications that include some cloud based or community component in an attempt to help increase the speed of detection and response. One of the purposes of these projects is to compare applications using cloud technology against the more traditional offerings.

Each week, we will select a batch of 250 early life samples and test our cohort of antimalware applications against them. The tests will all be live infection prevention tests. We will start each new project on a Monday and test the cohort every day for a week on the same batch of malware to chart detection improvements. The following Monday, we will start with a new batch of early life malware. After a week of testing, the sample set will then be tested every seven days.

In project #1 and Project #2, we will not be conducting further testing, these will be snapshots only. Testing will begin properly with Project #3.

MRG, may, from time to time submit missed samples to respective vendors at our discretion.

Project #1 uses 50 malware samples which we downloaded on 06/11/09, 24 hours before the test.

The applications tested are in three categories; cloud based antimalware, traditional antimalware and specialist complementary antimalware.

The cloud based Anti-Malware products were:

• Bluepoint Security 1.0.0.75
• Immunet Protect Beta 1.0.18
• Panda cloud Beta 3
• Prevx 3.0.5.10

The traditional Anti-Malware products were:

• A-Squared Anti-Malware 4.5.0.27
• Avira AntiVir Premium 9.0.0.447
• Kaspersky Antivirus 9.0.0.736
• Microsoft Security Essentials 1.0.1611.0
• Nod32 4.0.468

The complementary Anti-Malware products were:

• IOBit Security 360 1.20.10
• Malwarebytes Anti-Malware 1.41


Methodology used in this test:

1. Windows XP Professional Service Pack 3 is installed and updated with all important updates.
2. An image of the Operating System is created with internet access.
3. A clone of the Imaged system is made for each program to be used in the test.
4. An individual program is installed with default settings on each of the Cloned systems .
5. On each Cloned system the package containing the samples of malware is placed.
6. All the programs are fully updated.
7. Real Time protection/On Access scanners as well as all other methods of detection/prevention used by various Security Applications are turned on prior to the start of the test.
8. The test is conducted by performing two complete system scans and copying the sample set to and from the test machine twice.
9. Any malware samples detected are removed as it is a given that if they are detected in this way, they will also be detected on execution. (This has been confirmed by several of the vendors)
10. The remaining samples are executed individually. If the malware executes, the next sample is tested on a clean, uninfected machine by following steps 3-7, excluding step 5, above.
11. Any missed samples are tested again (executed)

Test results. Samples missed out of 50:

The cloud based Anti-Malware products:

• Bluepoint Security 1.0.0.75 - PASSED
• Immunet Protect Beta 1.0.18 - FAILED (missed 1 sample)*
• Panda cloud Beta 3 - FAILED (missed 1 sample)
• Prevx 3.0.5.10 - FAILED (missed 2 samples)

The traditional Anti-Malware products:

• A-Squared Anti-Malware - PASSED
• Avira AntiVir - PASSED
• Kaspersky Antivirus - PASSED
• Microsoft Security Essentials - FAILED (missed 11 samples)
• Nod32 - FAILED (missed 5 samples0

The complementary Anti-Malware products:

• IOBit Security 360 missed 49 samples
• Malwarebytes Anti-Malware missed 5 samples

Please note, we have discovered an occasional fault with Immunet Protect, which can result in a reduced detection rate. On one of the test runs, Immunet missed 9. We have discussed this with the vendor and will be conducting further tests with them.

Additional information:

1. We used only 50 samples which can sound as a very small sample set, but please keep in mind that these samples were zero day and ,as such, there was no need to use large amounts of samples.

2. Malwarebytes Anti-Malware and IObit Security 360 were tested On Demand only, this was done to create a direct comparison of their detection ratios.