# This rule means it applies to all user-agents User-agent: * Disallow: /activity/ Disallow: /admin/ Disallow: /album/ Disallow: /author/ Disallow: /backup/ Disallow: /backup-db/ Disallow: /bpa-cache/ Disallow: /bpa-media/ Disallow: /cache/ Disallow: /cdn/ Disallow: /cgi-bin/ Disallow: /compose/ Disallow: /create/ Disallow: /feed/ Disallow: /forum/ Disallow: /forums/ Disallow: /functions/ Disallow: /gallery/ Disallow: /gifts/ Disallow: /go/ Disallow: /group-documents/ Disallow: /groups/ Disallow: /images/ Disallow: /keywords/ Disallow: /links/ Disallow: /members/ Disallow: /messages/ Disallow: /plugins/ Disallow: /profile/ Disallow: /regiser/ Disallow: /registration/ Disallow: /scripts/ Disallow: /single/ Disallow: /tags/ Disallow: /themes/ Disallow: /trackback/ Disallow: /upgrade/ Disallow: /uploads/ Disallow: /widgets/ Disallow: /wp-admin/ Disallow: /wp-content/backup-db/ Disallow: /wp-content/cache/ Disallow: /wp-content/plugins/ Disallow: /wp-content/themes/ Disallow: /wp-content/upgrade/ Disallow: /wp-includes/ # Disallow all yearly archive pages Disallow: /1969/ Disallow: /1986/ Disallow: /1987/ Disallow: /1989/ Disallow: /1990/ Disallow: /1991/ Disallow: /1992/ Disallow: /1993/ Disallow: /1994/ Disallow: /1995/ Disallow: /1996/ Disallow: /1997/ Disallow: /1998/ Disallow: /1999/ Disallow: /2008/ Disallow: /2000/ Disallow: /2001/ Disallow: /2002/ Disallow: /2003/ Disallow: /2004/ Disallow: /2005/ Disallow: /2006/ Disallow: /2007/ Disallow: /2008/ Disallow: /2009/ Disallow: /1969/ # Disallow all monthly archive pages Disallow: /2007/01 Disallow: /2007/02 Disallow: /2007/03 Disallow: /2007/04 Disallow: /2007/05 Disallow: /2007/06 Disallow: /2007/07 Disallow: /2007/08 Disallow: /2007/09 Disallow: /2007/10 Disallow: /2007/11 Disallow: /2007/12 Disallow: /2008/01 Disallow: /2008/02 Disallow: /2008/03 Disallow: /2008/04 Disallow: /2008/05 Disallow: /2008/06 Disallow: /2008/07 Disallow: /2008/08 Disallow: /2008/09 Disallow: /2008/10 Disallow: /2008/11 Disallow: /2008/12 Disallow: /2009/01 Disallow: /2009/02 Disallow: /2009/03 Disallow: /2009/04 Disallow: /2009/05 Disallow: /2009/06 Disallow: /2009/07 Disallow: /2009/08 Disallow: /2009/09 Disallow: /2009/10 Disallow: /2009/11 Disallow: /2009/12 Disallow: /1969/12 # Disallow all archived monthlies Disallow: /2007/0* Disallow: /2007/1* Disallow: /2007/2* Disallow: /2007/3* Disallow: /2008/0* Disallow: /2008/1* Disallow: /2008/2* Disallow: /2008/3* Disallow: /2009/0* Disallow: /2009/1* Disallow: /2009/2* Disallow: /2009/3* Disallow: /1969/1* Disallow: /1969/2* Disallow: /1969/3* Disallow: /1969/0* Allow: / User-agent: grub-client Disallow: / User-agent: grub Disallow: / User-agent: WebZip Disallow: / User-agent: larbin Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: Copernic Disallow: / User-agent: psbot Disallow: / User-agent: Python-urllib Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: CherryPicker Disallow: / User-agent: WebBandit Disallow: / User-agent: ExtractorPro Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: Crescent Disallow: / User-agent: SiteSnagger Disallow: / User-agent: ProWebWalker Disallow: / User-agent: CheeseBot Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: Alexibot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: MIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: Website Quester Disallow: / User-agent: moget/2.1 Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: WebStripper Disallow: / User-agent: WebSauger Disallow: / User-agent: WebCopier Disallow: / User-agent: NetAnts Disallow: / User-agent: Mister PiX Disallow: / User-agent: WebAuto Disallow: / User-agent: TheNomad Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: RMA Disallow: / User-agent: libWeb/clsHTTP Disallow: / User-agent: asterias Disallow: / User-agent: httplib Disallow: / User-agent: turingos Disallow: / User-agent: spanner Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Microsoft URL Control - 5.01.4511 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: Foobot Disallow: / User-agent: SpankBot Disallow: / User-agent: BotALot Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: lwp-trivial Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Microsoft URL Control - 6.00.8169 Disallow: / User-agent: URLy Warning Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget Disallow: / User-agent: LinkWalker Disallow: / User-agent: cosmos Disallow: / User-agent: moget Disallow: / User-agent: hloader Disallow: / User-agent: humanlinks Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: Mata Hari Disallow: / User-agent: LexiBot Disallow: / User-agent: Web Image Collector Disallow: / User-agent: The Intraformant Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: True_Robot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: ProPowerBot/2.14 Disallow: /