Search & languages

Choosing the embedding model, non-English content, and changing it later.

Hive vectorizes every neuron with a local embedding model so semantic search works without sending your content to any third-party API. Which model it uses decides how well search works in your language.

Two models ship inside the image. Neither is downloaded at runtime — including when you switch between them, so this works on an air-gapped install.

How the English model fails on other languages

It does not throw an error, which is what makes it worth explaining. Content in German, French or Japanese is accepted, embedded, and returned by search — just less accurately, in two specific ways.

• Accents are stripped. The English model's tokenizer folds schön to schon — a different German word. Umlauts and ß are lost before the text is ever embedded.
• Words fragment. German costs roughly twice as many tokens per character on the English tokenizer as it does on the international one, so long non-English neurons exhaust the model's context window sooner.

The practical rule: if a meaningful share of your neurons is not in English, choose International. A brain that is mostly English with the occasional foreign name or quotation is fine on the default.

Choosing at install

Set the environment variable when you first start the container:

docker run -d \
  -e DATABASE_URL=... \
  -e ADMIN_PASSWORD=... \
  -e EMBEDDING_MODEL=international \
  --memory 4g \
  aiqbee/hive-server:latest

Changing it later

EMBEDDING_MODEL is read only when a Hive is first set up. Once neurons have been embedded it is ignored, and editing it in your deployment config does nothing except raise a notice on the Configuration page.

That is deliberate: changing the model means re-processing every neuron in every brain, and a redeploy must never silently start doing that. Instead, change it from Configuration → Embedding model in the admin UI. You will see how many neurons are affected and an estimated duration before confirming.

What happens during the change

• Neurons are re-processed in the background. Search stays correct throughout — neurons that have not been re-processed yet are found by keyword search only, so results are temporarily narrower, never wrong.
• Progress is shown on the Configuration page.
• It is safe to restart the container; re-processing resumes automatically.
• You can switch back at any time. That re-processes as well, so it is not free — but nothing is lost.

Keyword search and non-English text

The embedding model handles the semantic half of search. The keyword half matches on the text itself and works with accented characters in any of Hive's supported languages — a search for Müller matches Müller.

One limitation worth knowing for German: keyword matching finds a compound noun when you search for its ending (Versicherung matches Rechtsschutzversicherung) but not the reverse — searching for the full compound will not match a neuron that only mentions its parts separately. The semantic half is what bridges that, which is another reason to run International on a German corpus.