Archive for 'spider'


Sphider script php per creare motore di ricerca nel proprio sito web dotato di spider

Posted on 25. Jun, 2008 by Administrator.

0

Sphider č un open-source web spider e motore di ricerca. Includes un automated crawler, che puņ seguire i links di un sito e indicizzarli. Scritto in PHP e usa MySQL.

Spidering and indexing

Performs full text indexing.
Can index both static and dynamic pages.
Finds links in href, frame, area and meta tags, and can also follow links given in javascript as strings via window.location and window.open.
Respects robots.txt protocol, and nofollow and noindex tags.
Follows server side redirections.
Allows spidering to be limited by depth (ie maximum number of clicks from the starting page), by (sub)domain or by directory.
Allows spidering only the urls matching (or not matching) certain keywords or regular expressions.
Supports indexing of pdf and…

Continue Reading