Im new to yacy, and testing the crawler im really surprised in how it works, im in love :P.
im getting this error even when lauching the crawler and in the crawl check, in one specific web.
failed. Reason: scraper cannot load URL: REJECTED EMPTY RESPONSE BODY 'HTTP/1.1 405 ’
can anyone tellme what im doing wrong please?
thanks in advance.
in my experience, some CDN deny access, if you’re imitating GoogleBot. then REJECTED EMPTY RESPONSE BODY is returned.
How i can change that, in the advanced crawler?
robot behaviour> user agent
change from googlebot to yacy
that was my case, maybe your problem is different.
Maybe actually an error in the web site.
Just give generally genect feedback