Update to latest YaCy version to migrate Solr index to 8.11.2 (preparation for AI integration)

YaCy YaCy Peer-to-Peer
1

I just commited a migration to the latest solr 8 version in migrated solr from 8.9.0 to 8.11.2; activated also migration script. … · yacy/yacy_search_server@88cd17e · GitHub

I recommend everyone to install that version to be prepared for a migration to Solr 9. I consider to use neural search (optional) using DenseVectorField which is a new feature of Solr 9. This will be done together with “big snippets” that are supposed to be used together with Generative AI, i.e. langchain and ChatGPT-like systems that you can host yourself. I am currently preparing a framework which would then use YaCy as external source for RAG (Retrieval Augmented Generation).

So migration to this latest commit is a long-shot preparation for things to come. Please report all problems regarding the index migration here. If you need help migrating your YaCy instance, please ask here as well.

How can you see that your index was migrated successfully?
Look into the folder DATA/INDEX/freeworld/SEGMENTS/. If the re is a subfolder named solr_8_8_1 then you have still the old index. If there is a folder named solr_8_11_2 then you have the new index.

2

The was some misinterpretation about the purpose of the “AI integration”, as you can also follow along this toot: Michael Christen / YaCy Search: "AI integration with YaCy: Update to latest YaCy v…" - Sigmoid Social

This is not about integration of AI into YaCy but the other way around: YaCy as retrieval tool for AI. Here especially Generative AI in the context of RAG (Retrieval Augmented Generation).

See these documents as reference:

3

Works well on FreeBSD, only thing necessary is to install package javavmwrapper to let freebsd use Java 11.

Compile throws following warning:

[javac] Compiling 838 source files to /usr/home/yacy/yacy-src-klo/yacy_search_server/build/classes/java/main
    [javac] /usr/home/yacy/yacy-src-klo/yacy_search_server/source/net/yacy/http/InetPathAccessHandler.java:105: warning: [deprecation] asPathSpec(String) in PathMappings has been deprecated
    [javac] 				final PathSpec pathSpec = PathMappings.asPathSpec(path);
    [javac] 				                                      ^
    [javac] 1 warning

Do you think there is something on Solr 9, what YaCy search could benefit from as well?

4

Latest Solr has now already version number 9.3.0 and tons of bugfixes and improvements are done. I already did a test-migration of the code and I must say that this will not be an easy task: i’ts not only a change in the version number, the whole code base has changed a lot. As we have a deep integration of Solr (we do not call it over an API, we call the code directly) I have to dig deep into the code changes of solr to find out how to adapt YaCy in a proper way. Just changing the version number causes many compile errors.

But I believe it will be worth it and there is no way staying at the old version number forever.