If I stop crawling will the pages crawled at that point be saved?

YaCy
1

If I stop crawling will the pages crawled at that point be saved? (I mean I could just leave my Pi crawling wikipedia for 10 years but a)

2

I have restarted my peer a bunch of times while having several crawls running and they seem to resume where they left off. Does anybody know how YaCy crawls handle new pages sites add after you’ve already started a crawl?

3

What if I just dont want to crawl anymore? Do the crawled pages be saved?

4

Absolutely. Have a look at the Peer-to-peer Network tab and checkout the Senior and Junior peers. There are some that have been on the network for months or years yet they only have a couple thousand or just tens of thousands of documents.

I’m just speculating but maybe they’ve hit a point where they are happy with their local index and just pull any new results from other peers. Or maybe they’ve found their limit with running a peer and decided they would rather not allocate tons of memory to a JVM. I would love to hear others experiences with this.