# /robots.txt file for http://www.jewishgen.org # Created 17 December 2001, Modified 16 May 2002, 4 Dec 2003 # See "http://www.robotstxt.org/wc/norobots.html" for format details. User-agent: * Disallow: /family/g19* Disallow: /family/g20* Disallow: /ViewMate/ Disallow: /viewmate/ # modified 21-May-2012 GSandler prevent 80legs.com from crawling User-agent: 008 Disallow: /