ExternalSiteCatalog 1.2.0

ExternalSiteCatalog allows you to index and search external sites in a Plone site.

ExternalSiteCatalog is a web crawler that can index external sites and make them searchable in Plone. You can specify the sites to index in a Plone Configlet, and directly index them from Plone, or let a scheduler do the job. Have a look at some of the screenshot in the doc folder of the product to get a first impression of what it looks like. Searching the external sites is done in a special portlet that is installed with ExternalSiteCatalog. External sites are not searchable in the normal Plone catalog, but are only available in a separate catalog in the portal_externalcatalog tool.

Direct indexing

All external sites are configured in the ExternalSiteCatalog configlet.
If you want to index external sites immediately, you can do so after
defining all the parameters, selecting your site, and clicking on
the "index" button in the configlet.

Crawling external sites regularly

If you want external sites to be crawled regularly every day or month,
youll have to do some extra work. Make sure to install PloneMaintenance
starting from version 1.3. Make sure that you are also regularly calling
PloneMaintenance from cron or one of the Zope schedulers.

Follow the installation instructions in PloneMaintenance!

Have a look at the portal_maintenance tool in the Zope Management
interface! It contains a lot of useful information!

Console indexing utility

The console indexing utility is a bit more complicated than doing everything
from Plone. In many cases you dont have the resources to administrate an
external utility, but if you can afford it, this tool gives you the possibility
to decouple the long running work of fetching external sites from Zope.
Basically, you can be sure that there is no Zope thread blocked with
crawling external sites for a long time. Zope is only called from the external
tool when it should index a page, so there is still some load on the
Zope server!

Please note that this external tool is not making use of the information
entered in the Plone configlet. It is completely independent! It also does
not make use of PloneMaintenance, and it is up to you to configure and
call it with a scheduler like cron.