I had the attached crawler developed and it works well. However, there are 3 things I need to have fixed.
1. Speed of the crawl.
2. Speed of the Index
3. Indexing process locks up on large crawls.
Qualified developers will first go through the code and then tell me what needs to be fixed in order to get a much better functioning crawler.