Friday, July 30, 2010

New Firefox add-on to protect against Blackhat spam SEO

There are currently no ultimate solutions for end-users to protect themselves against fake AV pages, fake videos and other malicious spam SEO: antivirus have a low detection rate, blacklist (such as Google Safe Browsing) lag behind the creation of new malicious domains.

In a previous post, I pointed out the vast majority of malicious spam SEO sites check the referrer string of the visitor. If this string does not include bing.com, yahoo.com or google.com, the user is not redirected to a malicious page.

We are releasing a Firefox add-on that uses this trick to protect Firefox users against most spam SEO threats, including fake AV and fake Video pages. This plugin works with Firefox 3.x. Click on the image below to install it. It is called "Search Engine Security".

Install Search Engine Security add-on for Firefox 3.x

Search Engine Security add-on installed

How it works

This Firefox add-on handles Bing, Yahoo and Google search engines, in all languages. Normally, if a user clicks on a link within search engine results, the HTTP request to the external site contains a Referrer string from the search engine within the HTTP header. For example, if a user searches for "this is a test" in Google, any request to a search result will include the following Referrer:

Referer: http://www.google.com?q=this+is+a+test&hl=en&safe=active
For these requests, the add-on changes the Referrer header to a different value. This means that the requested page does not know that a given request came from a Google, Yahoo or Bing search. This is critical as Blackhat SEO pages only deliver malicious content (fake AV, Flash/Java updates, codecs, etc.) when requests come from the SEO results. Changing the Referer header, breaks the attack.

The add-on does not change the referrer if you navigate within the same site (for example, inside google.com), or if your Referrer does not include a Bing, Yahoo or Google domain.


Install Search Engine Security add-on for Firefox 3.x


Configuration

You can customize the behavior of the Search Engine Security add-on. In the preferences menu, you can change the following values:

Search Engine Security preferences

-Protect

Select the search engines for which you wish to enable protection.

- Use Referer header

Choose the Referer value to use for overriding the Google/Bing/Yahoo Referer. You can use an empty value, but it is recommended that you use a valid URL.

- Modify User-Agent (NEW in 1.0.8)

Most spam pages look at the Referrer value to decide whether or not to redirect users to a malicious page. However, in some cases like the Hot Video pages, only the User-Agent value is used. One common check is to look for "slurp" in the user-agent string to flag the request as coming form the Yahoo crawler. If you check the "Modify User-Agent" checkbox in the options, the string "slurp" is added to the User-Agent header when you leave Google/Bing/Yahoo in addition to overriding the Referrer header.

This option provides additional protection against malicious spam SEO.

-Whitelist

Some websites display a different page if you come from a search engine. When you use this add-on, the websites can no longer detect that you come from Google/Yahoo/Bing. If you are sure that a website is safe, you can add it to the whitelist. This will disable the add-on for this website.

If the URL matches any of the elements in the whitelist, the add-on does not change the Referer value. This is a string match and the match occurs if the URL includes one element of the whitelist. For example, http://www.expert-exchange.com/ can be whitelisted by adding:

  • http://www.expert-exchange.com/ (also matches http://www.expert-exchange.com/foo)
  • expert-exchange.com/ (matches any subdomain)
  • expert-exchange. (matches the domains expert-exchange.net, expert-exchange.org, and paths like http://example.com/expert-exchange.html/)
  • etc.

Notification (NEW in 1.0.4)

A notification is shown on Bing, Yahoo, and Google to let users know whether the SES protection is enabled for this search engine. The notification is shown under the search input.

Search Engine Security notification in Google search
Search Engine Security notification in Bing search


If you find any problem with this add-on, please let me know at jsobrier@zscaler.com.

Install Search Engine Security add-on for Firefox 3.x

-- Julien

37 comments:

Anonymous said...

I have the feeling you will piss of a lot of web masters who use the Google referrer to determine how many people are coming from Google, what search terms they are using, etc.

Anonymous said...

why are there a whole host of versions? The firefox add site has 1.03. This site has 1.04, updated to 1.05?

Anonymous said...

There are an older, widly customizable addon, named RefControl:

https://addons.mozilla.org/hu/firefox/addon/953/

And yeah, and sorry, I'll piss on all webmasters, who modify their webpage's content by the visitors referrer.

Julien Sobrier said...

The current versions, as of 08/13/2010, is 1.0.5. I have updated the post to point to the latest version.

I released a couple of updates to the plugins:
* 1.0.1, 1.0.2: Better descriptions
* 1.0.3: link to this blog post
* 1.0.4: SES on/off shown in Google searches
* 1.0.5: SES on/off show in Yahoo and Bing as well

If you want the latest version, go to Tools > Add-ons > extensions. Look for Search Engine Security and right click > Find Update.

Anonymous said...

@ julian

Thanks for the info. I downloaded the add on via the official Firefox add on site. Right now it is still 1.0.3 and it says no update found when I try forced update. I guess the addon from the Firefox add on site is not updated yet to 1.0.5 or even 1.0.4.

https://addons.mozilla.org/en-US/firefox/addon/207647/

Anonymous said...

You can also disable Firefox's referrer manually: type about:config in the address bar and press Enter. Click through the reminder to be careful and navigate in the resulting list to network.http.sendRefererHeader. Double-click the entry to open the "Edit integer value" dialog. Change the value to 0 and click OK. The change will take effect when you restart Firefox.

Anonymous said...

Please update this add-on on addons.mozilla.org to the latest version.

Julien Sobrier said...

Search Engine Security is still being reviewed by the official Firefox Add-ons sites. I did not realize that it was actually available to anybody before it was reviewed. I've uploaded version 1.0.6 to our website and the official Add-on site, so you should be able to update the plugin. 1.0.6 is functionally the same as 1.0.5, it only contains some internal code refactoring.

Julien Sobrier said...

"You can also disable Firefox's referrer manually[...]"

This solution is less intrusive. It changes the Referrer only when the user is leaving Google/Yahoo/Bing. It is also easier to whitelist websites which display different content when you come from a search engine (like expert-exchange.com)

Anonymous said...

Does this add-on work with Google SSL Search? Google SSL Search removes referer when going to a http site, but not when going to a https site.

Anonymous said...

I'm using Firefox 4 beta 3 with Add-on Compatibility Reporter.
The "SES on" is visible on Google search page, but how can I test if it works? How can I know the referer isn't sent?

Julien Sobrier said...

About "Google SSL Search": the add-on does modify the Referer for https website in Google SSL Search.

Julien Sobrier said...

"I'm using Firefox 4 beta 3 with Add-on"

The add-on is set to work with Firefox 3.0 to 3.6.*, so I do not know how it will behave with Firefox 4.0

The easiest way to test SES is to install the Firefox plugin "Live HTTP Headers" This extension will show you the headers sent with each request. Or you can use a proxy such a Fiddler.

Anonymous said...

Hi,

Any idea as to when Mozilla will have finished testing this Addon?

I'm patiently waiting for it to get the green light.

Julien Sobrier said...

The extensions has been approved by Mozilla today: https://addons.mozilla.org/en-US/firefox/addon/207647/

I need to upload a newer version.

Anonymous said...

Thanks julien, installing now.

Nice work btw :).

Anonymous said...

Hello Julien,

Sorry for my ignorance, but I'm not the developer, like you. :)

You wrote: 'There are currently no ultimate solutions for end-users to protect themselves against fake AV pages, fake videos and other malicious spam SEO ..'.

My experience tells me two easy solutions against these fake antivirus automatic redirections.

If I pass my infected link by a web proxy anonymizer, I do not get the malicious content, but innocent spam page only.
If I click on malicious link with a search engine using the New Window feature(eg. Yippy Search, Zuula Search), I do not get the malicious content, but innocent page ..

So: by proxy - no infection, by New Window in search engine - no infection.

But why - this is my question for you ..

Thank you for your response,

PROROOTECT

Anonymous said...

Yes, I wait patiently for your response, sorry :)

Prorootect :)

Anonymous said...

Sorry for my English, yes, it's me who asks the question: how to act proxies and search engines with 'New Window' feature - to do their job, which I described.

P.

Julien Sobrier said...

@anonymous I don't think using a free anonymizer is a solution. You have no idea what they do with your data: do they add malicious content to the reply, how do they handle credentials and other private information you may need to enter on the final site, etc.

ae6dx said...

I have this on 3.6.13 Mac OS X. With it installed Firefox uses a lot more CPU.

Anonymous said...

@ae6dx, can you elaborate? The addon itself is only 11 k, so I doubt it.

However, I would like Julien Sobrier for this free addon. Became security conscious after hitting a fake AV link from a google search.

Now In my current browser config, Firefox+ Norton DNS+ WOT+ Zscaler SES = pretty secure browser all for free.

Anonymous said...

This seems like a great idea - my users are getting hit with these SEO malware a lot. Is there any way to do something like this in IE?

Anonymous said...

Please update the addon and mark it compatible with Firefox 5.*

Anonymous said...

This may seem like a stupid question, but WHY does the referral heading (e.g., Bing, Google, Yahoo, (or whoever), determine the instance of "Blackhat spam SEO"?

Julien Sobrier said...

@all: the plug-in has been updated to work with Firefox 5.*

@anonymous: check out the this post to learn how Blackhat SEO posts differentiate between users and bots:http://research.zscaler.com/2010/09/hot-video-pages-analysis-of-hijacked_29.html

Anonymous said...

Please update Search Engine Security on AMO, and mark it compatible with Firefox 6

Anonymous said...

Please update the addon and mark it compatible with Firefox 6.*

Anonymous said...

ZScaler SES needs an update for FF6....

Julien Sobrier said...

@Anonymous Thank you for the reminder, I've updated the plugin.

Anonymous said...

Why isn't version 1.2.1 on AMO?

Anonymous said...

The "Modify User-Agent" doesn't work. It has no effect, the user agent doesn't get modified.

Julien Sobrier said...

Version 1.2.1 was released to include a newer maximum version of Firefox. There is no functional change. AMO took care of increasing the version number, so there was no need to create a new version.

Julien Sobrier said...

@anonymyzer: the add-on adds "slurp" at the user agent string. Look for "referer mobilefish" in Google. Click on the first search result. The page from mobilefish.com shows your User Agent string as well as Referer. You can try it with Search Engine Security truned on and off for Google to see the different values.

Anonymous said...

Just thank you note for the addon. Precisely what I wanted--don't want to get myself notified for private search keyword but also keep most websites working untouched. For me the reason choosing was not because of advertisement websites but more of privacy. Regardless it still works fine for my use. Refcontrol was popular result when I searched referer for extension, but it's impossible to set refcontrol like SES or vice versa. So please keep up your good work (which I understand to be not as eas with current FF numbering scheme).

Anonymous said...

SES is marked incompatible and disabled in Firefox for Mobile 9
Please fix!

Julien Sobrier said...

@All SES 1.2.3 is marked as compatible with Firefox Mobile 11 on our website, and version 15 on the Mozilla Add-on site