Saturday 8 March 2008
Welcome to the SocSciBot3 Blog
Please post any bugs or problems you have with SocSciBot3 here - these will help other users to know of any problems and the problems will eventually be picked up and solved by me. Please also feel free to respond to any other users' problems if you know of a solution.
Subscribe to:
Post Comments (Atom)
2 comments:
Dear Mr. Thelwall:
SocSciBot3 has two restrictions:
Every 100 pages crawled it stops for some time in order to avoid an overload on the server that hosts the web site to crawl. I understood that.
But also It crawls 5,000 pages maximum. What causes this limit?
Thanks,
Joan Gironès
Dear Joan,
The reason for the 5,000 pages maximum restriction is that undergraduate students use SocSciBot so it is important to limit the total number of pages that they can download. Without this they might accidentally cause harm to web sites. If you would like the full version (900,000 pages maximum then please email me your email address.
Best wishes,
Mike
Post a Comment