Tutorial

Firefox Search Engine Cautions, Recommendations

See the revision history at the end for a list of changes to this document.

Introduction

The scope of this tutorial is limited primarily to preserving your privacy when using the default Firefox (or derivative thereof) search engine plugins, as well as discovering new search engines and finding alternative ways to use the major search indexes, such as Google, Bing and Yahoo. For a more in-depth tutorial regarding Firefox privacy issues and customization, please see the article, Firefox Configuration Guide for Privacy Freaks and Performance Buffs.

When ‘Free’ software isn’t

Many of us probably tend to associate the open-source software community with individuals or small organizations that freely give away their work and expect little or nothing in return, however nothing could be further from the truth regarding the multi-million dollar Mozilla Foundation. The hundreds of millions of dollars that Mozilla rakes in annually is largely due to its partnership with search engine companies such as Google, Yahoo and others. These ethically challenged mega-corporations then track your web activities and sell the data they collect to advertisers and governments and who knows who else. It seems to me that these kinds of partnerships are clearly at odds with Mozilla’s statement, “Committed to you, your privacy and an open Web“.

So how does Mozilla get paid by the mega-giants like Yahoo and Google? Simple: Any time you use any of the default search engine plugins that are packaged with the browser, parameters like these are added to your search query:

client=firefox-a
name="appid" value="ffd"
name="hspart" value="mozilla"

These parameters tell the search engine that you are using a Firefox/Mozilla product and that’s all it takes to rake in the dough. If you do not wish to participate in these affiliate schemes and/or value your privacy, read on.

Types of search engines

It appears the two basic types of search engines are meta search engines and search indexes and it is important to understand the difference. Google, Yahoo and Bing for example, use software robots known as “crawlers” to discover and index web content. In other words these companies actively seek out updated and fresh content to store in their databases so that it’s ready for you to find. Meta search engines do not typically index the web however and instead rely primarily on third parties like Google, Yahoo and Bing to provide their search results and therefore when you use so-called “alternative” search engine such as DuckDuckGo, Startpage, Searx, etc., you are still subject to the content manipulation and censorship employed by the big corporations. While these alternative meta search engines do indeed make a great deal of sense from a privacy perspective since one can avoid being tracked by the big indexes directly, they are not true alternatives as they are often described, but more like search engine proxies which simply provide their own interface in order to display the search results as provided by third parties. These alternative search engines are also subject to local laws, such as secret surveillance requests issued by a government.

Indexing the web and storing the massive amount of data that results is an incredibly expensive proposition and this is why meta-search companies like DuckDuckGo, Startpage, Searx and others rely heavily upon corporations like Google. There is an even better solution however, one which both respects your privacy and is censorship resistant and these are distributed search indexes. Imagine a free, open-source, decentralized search engine where the search index is distributed among millions of personal computers like yours, each storing a piece of the whole. This is what the developers behind YaCy have done with their peer-to-peer search engine and i think it’s a great way to move forward and avoid corporate censorship and attacks on personal privacy.

Adding search engines to Firefox

Possibly the easiest way to mitigate risks to your anonymity posed by using the default Firefox search engines is to simply disable all of them and find alternatives. One of my favorite choices is the open source and highly configurable Searx meta search engine which you can host on your own web server if you wish, but you certainly don’t have to in order to use it. Like DuckDuckGo, Startpage and others, Searx is not an index and so it does not crawl the web seeking out and storing content like Google does. The primary difference between Searx and most of the other meta search engines however, is that Searx is capable of pulling results from many other indexes, including decentralized peer-to-peer indexes such as YaCy, as well as centralized indexes like Google Yahoo and Bing, any of which you can enable or disable using the excellent preferences interface that Searx provides.

One easy way to add Searx to Firefox is to locate a hosted instance which you like and which is preferably close to you geographically, and then from the Firefox search bar, simply click the “add” menu item. While searx.me is the original instance of Searx as provided by the developers, it is best not to use it because it can become overloaded. The Searx developers simply cannot afford to have too many people using their particular instance without your help and so they will disable it at times in order to promote the many other third party instances. That said, a potential pitfall to using a third party Searx instance is that the server may be logging traffic, such as IP addresses, countries of origin, etc., so you’ll have to decide what’s best for you.

Most other search engines can be added to Firefox in the same way as described above, but there are other ways also. The Mycroft Project hosts tens of thousands of preconfigured search engine plugins for a variety of web browsers and they  have a form for writing your own plugins which can then be added by simply by clicking on their names, the top 100 of which are listed here. Unfortunately it is not possible to review the code from the main listing of search plugins before adding it to your browser, however you can use their submission form to do so by mousing over the plugin name to reveal its numeric ID and then filling in that ID in the submission form page.

Privacy-centric search engines

For a list of search engines that focus on privacy, see the article, Alternative Search Engines That Respect Your Privacy.

Sanitizing the default search engine plugins

If you choose to use the default search engine plugins provided by Mozilla, you may want to sanitize them in order to circumvent some risks to your privacy, however you should be aware that sanitizing the default plugins will not prevent tracking or privacy risks when clicking on the search results provided by those companies. If you are going to use the default search engine plugins, then you should also use something like the Neat URL add-on which at least strips the tracking parameters from URLs.

If you are running Firefox version 56 or less and want to sanitize the default search engine plugins yourself, you might want to install the XML Search Engines Exporter/Importer add-on which makes it super easy to export and import the modified plugin code wherein we will be removing the monetization schemes and browser fingerprinting employed by Mozilla. You will also want a decent code editor, such as Kate for Linux (already included in some distributions) or Notepad++ or PSPad for Windows. If you do not want to sanitize the default search engine plugins yourself, you can download my pre-sanitized plugins below (what search engine plugins ship with Firefox depends on your language and i have sanitized only the U.S. English ones).

From Firefox version 57 and newer, Mozilla has dropped support for importing search engine plugins from the /searchplugins folder in your profile directory and so another method is needed. Though the XML Search Engines Exporter/Importer add-on will not work in v57+, the developer, ‘nohamelin’, has kindly made available two Scratchpad scripts on his GitHub page which include instructions for exporting and importing search engine plugins. They are xseei.export-all.js and xseei.import.js. You can use the import script to import your manually sanitized search plugins, or you can import my pre-sanitized search plugins below.

Installing pre-sanitized search plugins

If you would rather avoid the hassle of sanitizing the default search plugins manually, you can use my pre-sanitized plugins which are from the U.S. English version of Firefox and include Amazon, Bing, DuckDuckGo, Google, Startpage, Twitter, Wikipedia and Yahoo. Each will have an “(s)” appended to its name in your search bar and search preferences to indicate it has been sanitized. Note that you should still sanitize the prefs.js preferences corresponding to these search plugins as described below.

In all cases the search suggestion code has been removed and _s was appended to the XML file names. In the case of Google it has been further customized to disable safe search, auto-complete, personalized search and other annoyances. Google has also been configured to use only google.com, thus disabling the automatic country redirect (in other words if you live in Canada and want to use google.com, Google will no longer redirect to google.ca) and both the displayed name and file name include “NCR” which stands for “No Country Redirect”.

Download: sanitized-searchplugins.zip

Sanitizing manually

If you are using the XML Search Engines Exporter/Importer add-on for Firefox version 56 or less, open the Firefox preferences UI and navigate to the “Search” section, or enter about:preferences#search in the address bar. You should probably make sure that all of the search engines are enabled at this point, even those you don’t use, else you will not be able to edit them. Click the “Restore Default Search Engines” button if necessary. Near the bottom of the search preferences UI you will notice some new buttons, but the one we are interested in is labeled “Export All Search Engines to File…”. This will export all of the default search engine plugins, as well as any others you may have added, to a ZIP archive, the extraction of which will reveal a bunch of XML files and it is these which we will be editing.

If you are using the Scratchpad scripts by ‘nohamelin’, follow the instructions in the script comments to export the search plugins.

The following uses the yahoo.xml file as an example (note that the base64 image code for the Yahoo icon was removed ( [icon code removed] ) to shorten the example, but it will be present in your file):

<?xml version="1.0" encoding="UTF-8"?>
<Searchlugin xmlns="http://www.mozilla.org/2006/browser/search/" xmlns:os="http://a9.com/-/spec/opensearch/1.1/">
  <os:ShortName>Yahoo</os:ShortName>
  <os:Description>Yahoo Search</os:Description>
  <os:InputEncoding>UTF-8</os:InputEncoding>
  <os:Image width="16" height="16">data:image/x-icon;base64,[icon code removed]</os:Image>
  <SearchForm>https://search.yahoo.com/yhs/search?p=&ei=UTF-8&hspart=mozilla&hsimp=yhs-001</SearchForm>
  <os:Url type="application/x-suggestions+json" method="GET" template="https://search.yahoo.com/sugg/ff" resultDomain="search.yahoo.com">
    <os:Param name="output" value="fxjson"/>
    <os:Param name="appid" value="ffd"/>
    <os:Param name="command" value="{searchTerms}"/>
  </os:Url>
  <os:Url type="text/html" method="GET" template="https://search.yahoo.com/yhs/search" rel="searchform" resultDomain="yahoo.com">
    <os:Param name="p" value="{searchTerms}"/>
    <os:Param name="ei" value="UTF-8"/>
    <os:Param name="hspart" value="mozilla"/>
    <!--The original definition of this search engine included too some non-standard 'MozParam' parameters. As they are recognized by Firefox only if they are found in an engine included by default in the application, they were omitted here.-->
  </os:Url>
</SearchPlugin>

Examining the above code reveals added parameters that are not necessary for performing a search with Yahoo. Following are the parameters which we want to edit:

In the <SearchForm> tag, everything after /search in the URL, beginning with the question mark, can be removed so we end up with the following:

<SearchForm>https://search.yahoo.com/yhs/search</SearchForm>

The entire block beginning with <os:Url type="application/x-suggestions+json" and ending with the first of the two </os:Url> closing tags can be removed. This will disable some fingerprinting as well as search suggestions.

The line <os:Param name="hspart" value="mozilla"/> can be removed.

The last thing you must to do is rename the search engine since the XML Search Engines Exporter/Importer add-on will not import a search engine plugin with the same name as a default plugin included with Firefox. This will also allow you to be sure you are using the sanitized version of Yahoo rather than the default one as provided by Mozilla since the modified name will be the one displayed in your search bar and in your search preferences. To rename the plugin, simply edit the following line and change Yahoo to whatever you wish:

<os:ShortName>Yahoo</os:ShortName>

I might suggest something like this, where the (s) stands for “sanitized”:

<os:ShortName>Yahoo (s)</os:ShortName>

The above is how the Yahoo search engine will be displayed in your search bar and in the Firefox preferences.

The name of the XML file does not have to be changed, but you may want to append something like _s to it to indicate it has been sanitized so you don’t get it mixed up with the default search plugin.

For the remaining search engine plugins you basically want to repeat what we have done above with Yahoo. The parameter names and values will sometimes differ from the examples above, but basically you want to look for anything that identifies your browser, operating system and, potentially, your local and remove it. For example, if you exported the Wikipedia search plugin XML file, you will find in it &amp;sourceid=Mozilla-search and <os:Param name="sourceid" value="Mozilla-search"/>, both of which can be removed. The exception with all of the search plugin XML files is that you do not need to alter the <SearchPlugin xmlns= line even though it may contain a mozilla.org URL.

To import the sanitized search engine files with the XML Search Engines Exporter/Importer add-on (FF v56 or less):

  1. Start Firefox, open the search preferences UI (about:preferences#search) and import your sanitized search plugins. Optionally you may want to disable/remove the default search plugins.

To import the sanitized search engine files manually (FF v56 or less):

  1. Place your sanitized search plugin XML files in the /searchplugins folder within your Firefox profile folder (create the folder if necessary).
  2. In your Firefox profile folder, rename the search.json.mozlz4 file to search.json.mozlz4.bak.
  3. Restart Firefox and, optionally, open the search settings in the preferences UI (about:preferences#search) and disable/remove the default search plugins.

To import the sanitized search engine files using the Scratchpad script by ‘nohamelin’ (FF v57 or greater):

Simply follow the instructions in his script.

Sanitizing the prefs.js search engine preferences

Another step we need to perform is to sanitize any browser.search.param. preferences in the prefs.js file. To do this it is best to create a custom user.js file to store our modified preferences if you don’t already have one. You can read the tutorial, Firefox Configuration Guide for Privacy Freaks and Performance Buffs, for information on creating the file, as well as many additional things we can do to protect our privacy and tighten the security of Firefox.

To sanitize the search engine preferences, open the about:config URL in your browser and enter browser.search.param. in the search field. At the time of this writing there are only two preferences that will be displayed and they are browser.search.param.yahoo-fr and browser.search.param.yahoo-fr-ja. The default values may be different in your case, but in mine they are data:text/plain,browser.search.param.yahoo-fr=linuxmint and an empty string, respectively. What you need to do is copy both preference names to your user.js file and set the values to an empty string:

user_pref("browser.search.param.yahoo-fr", ""); // sanitize Yahoo
user_pref("browser.search.param.yahoo-fr-ja", ""); // sanitize Yahoo

If you are going to sanitize the other default search engine XML files, you may as well sanitize any other prefs.js preferences related to the other search engines as we did above. Simply enter browser.search.param. in the search field and copy all the preference names to your user.js file and set the empty the values as shown above (again, at the time of this writing, the only two preference names refer to Yahoo).

Removing the ‘Follow On Search’ system add-on

Mozilla packages some system add-ons (browser extensions) with Firefox and installs them without your permission and these system add-ons are not listed in the Extensions section of the Preferences UI (about:addons). The technology behind one of these system add-ons is called “Follow-on Search” (also see the Mozilla GitHub repository page) and it is used to collect data about the way you use the Google, Bing and Yahoo search engines. Note that this add-on works independently of the search engine plugins discussed in this article and therefore it too must be addressed if we value our privacy.

To see if the Follow-on Search add-on is installed, open about:support in the Firefox address bar and look under the section titled “Firefox Features”. In my particular case under Linux Mint, there was no “Firefox Features” section on the support page, nor was the add-on found on my system, however there were other system add-ons found.

The Follow On Search add-on, which has the file name followonsearch@mozilla.com.xpi, is located at /usr/lib/firefox/browser/features on Linux Mint (and very likely other flavors of Linux as well) and at \Program Files (x86)\Firefox\browser\features or \Program Files\Firefox\browser\features on Windows and i would suggest either deleting or disabling it. Under Linux, i am not aware of how it can be disabled, but it can certainly be deleted. Here’s how delete all of the system add-ons under Linux:

cd /usr/lib/firefox/browser/features
sudo rm *.xpi

On Windows you can apparently use CCleaner to disable these system add-ons from the Tools > Browser Plugins menu.

Note that these system add-on files will be recreated when Firefox is updated and therefore you will need to delete/disable them after each update. On Windows, CCleaner may keep these add-ons disabled after a Firefox update but you should check to be sure. To avoid this hassle, i suggest using a privacy-centric custom build of Firefox, such as Waterfox, which does not support system add-ons.

We’ve only scratched the surface…

Sanitizing the Firefox search engine plugins is a good start, but there is much more to do if you’re interested in circumventing the risks to your privacy and computer security that are inherent in any of the popular web browsers. For further information, please refer to my article, Firefox Configuration Guide for Privacy Freaks and Performance Buffs. You may find it helpful even if you aren’t using Firefox.

Resources

Special mention goes to ‘Thorin-Oakenpants’ (aka ‘Pants’) as well as the ‘ghacks’ crew and their GitHub repository where they host an excellent privacy and security centric custom user.js for Firefox, as well as a Wiki which is full of valuable information.

Change log

Click to expand...

15-Sep-2017

  • first publish

16-Sep-2017

  • added this change log
  • corrected an error in the pre-sanitized Wikipedia search plugin and re-uploaded sanitized_search_plugs.zip
  • added information as suggested by ‘Pants’ in his comment below, particularly details and resources regarding the followonsearch@mozilla.com.xpi system add-on in a new section titled “Removing the ‘Follow On Search’ system add-on
  • added Hulbee and MetaGer to the search engine list
  • added a “Decentralized” column to the search engine table
  • added resource: 5 Best Search Engines That Respect Your Privacy – BestVPN.com
  • misc. cleanup and edits

17-Sep-2017

  • corrected typo in metager URL
  • added “Requires JS / Cookies” column in search engine table
  • changed links for search engines in table to point to company/about page and added links to point to search page
  • added link to the ‘lite’ version of DDG
  • added a link to the uBO filters to block Startpage/Ixquick tracking images
  • misc. minor edits

18-Sep-2017

  • added “Client Required” column to search engine table
  • corrected some info regarding the search engines in the table
  • minor misc. edits

24-Sep-2017

  • added a link to the Duck Duck Go: Illusion of Privacy article
  • added findx to the search engine list
  • minor edits

27-Sep-2017

  • added Qwant to the search engine table

29-Sep-2017

  • misc. edits and added info, nothing really important

3-Oct-2017

  • very minor edits

23-Oct-2017

  • moved the list of alternative search engines to it own page
  • minor edits

5-Dec-2017

  • minor change to the section ‘Sanitizing the default search engine plugins’ thanks to commenter ‘nohamelin’ – more changes coming shortly thanks to this persons comments

23-Dec-2017

  • updated search plugin import/export instructions as per the very helpful comment left by ‘nohamelin’, the developer of the XML Search Engines Exporter/Importer add-on in which he made available Scratchpad scripts that work with FF v57+
  • corrected an error in the pre-sanitized search engine archive, added Startpage and re-uploaded a new archive
  • misc. minor edits

28-Jan-2018

  • polishing

11 thoughts on “Firefox Search Engine Cautions, Recommendations

  1. (XML Search Engines Exporter/Importer developer here)

    Nice text. I want to add that the list of search plugins included by default in Fx is highly dependent of the build installed: that given list is for the en-US build, I think, and it changes for other locales, according to regional deals done by Mozilla, between other things.

    Now, given that no WebExtensions add-ons interacting with your search engines can be written yet, that leaves us for Fx57+:

    * First, you can get easily the original XML opensearch definition of all the engines included by default navigating to the “resource://search-plugins/” uri. From here, you can inspect them and save them to disk for manual sanitizing.
    * I adapted from the add-on some scripts to import/export engines via Scratchpad; it’s a bit awkward to use, but it should help:
    https://gist.github.com/nohamelin/8e2e1b50dc7d97044992ae981487c6ec
    https://gist.github.com/nohamelin/6af8907ca2dd90a9c870629c396c9521

    Also, be aware that Firefox dropped recently the support for adding engines from the searchplugins/ folder after deleting search.jzon.mozlz4; it will aply since Fx58:
    https://bugzilla.mozilla.org/show_bug.cgi?id=1405670

  2. do you know of any alternative commands in the terminal to delete this add-on ( unfortunately waterfox approved it ) ? it did not work for me in ubuntu 16.04

      1. I got the message, that it couldn’t be removed as the file/ filecatalog did not exist. english is not my first language, so i’ve translated the message. tech terms may not be correct. I’m no savvy myself, so I’m only familiar with the most common terminal commands. I do attend a local community. they might know about alternatives. yet, I’m the only one pro privacy. off subject: this is a great blog. very user friendly and in depth. thank you @pants for linking at ghacks

  3. Excellent article 12bytes. Really well done mate :)

    Some items for thought:
    – users should note some engines will require stripping tracking of search *results* (eg utms on google)
    – Firefox has a system add-on called Follow On Search, you should kill that (see ghacks user.js)
    – XML Search Engines Exporter/Importer is not Web Extension (yet?) so for FF57+ users, I guess we can use a portable legacy FF to create the file(s)?

    1. Also remember that in a lot of cases it is better in the first place to use a site specific search engine as well. For example, I have added an iTunes search engine so no-one but iTunes knows I search for Taylor Swift .. #GoTayTay :)

    2. thanks for your input Pants! much appreciated – i updated the article to address this stuff and added a section “Removing the ‘Follow On Search’ system add-on”

      Q: do you know how these system add-ons are loaded – are they loaded dynamically each time FF starts like other add-ons, or are they actually installed/cached some place? i’m wanting to be sure that deleting them is sufficient

      1. System add-ons are listed under about:support>”Firefox Features” – they act just like normal extensions AFAIK (except no disable/uninstall options). Delete the xpi and restart FF and you’re good to go.

        System add-ons are packaged with each Firefox application update (full updates at least). I use portable FF, so I am not sure about installed version behavior re dot releases. If I try to update and I have deleted some system add-on xpi files, the update fails and FF prompts for a full package instead (although the last update from 55.0->55.0.3 didn’t? Can’t remember! Dot releases may vary.). Anyway, I keep an eye on my system add-ons directory, and I only do updates now by downloading the portableapps.com packages (yes they include all the system add-ons too – both 32+64bit app/dirs)

        One of the reasons system add-ons exist, is so that patches/changes can be pushed without an app update. But the update check and update settings can have an effect. See: https://github.com/ghacksuserjs/ghacks-user.js/issues/172 . The ghacks user.js checks for app updates but lets you decide when to apply them – so until this bug is resolved, you won’t get system add-ons re-added or updated without your knowledge.

        You could also try 0505: “extensions.systemAddon.update.url” – if the system doesn’t know where to go, what can it do :) The only reason this is inactive in the js, is because this mechanism was initially designed to PUSH fixes – I think it was spurred on by a critical vulnerability in pdfjs a few years ago (Yes, pdfjs is a system add-on, but seems to be a special case compared to how/where it is stored)

        1. thanks for the clarification

          in my case, with Linux, no system add-ons were displayed in about:support and, actually, there wasn’t even a section called ‘Firefox Features’ – the ‘follow-on search’ add-on wasn’t present either, though others were

Leave a Reply

Your email address will not be published. Required fields are marked *