Mirror an index

Purpose

Mirroring supports to configure a silo such that Enterprise Search starts copying all documents in the silo index to a mirror index. In addition ES starts keeping the mirror index up-to-date along with updates of the silo index.

The mirror index may reside in the Elastic Search instance as currently managed, or may reside in an Elastic Search instance elsewhere.

Mirroring uses the document text and document metadata as stored in the silo index (for information: using the Elastic Search _source field). Mirroring does not require revisiting web sites, databases, file systems and other sources originally visited.

Purposes of mirroring include:

  • When upgrading the version of Elastic Search it is possible to mirror documents to the higher version index, and switch search applications to the new index with zero downtime. Elastic Search also offers upgrade paths, some without limitations, some that are increasingly limited depending on whether a minor or major version upgrade is performed. Mirroring works in more cases and for example also supports a version downgrade.
  • When changing the document field mapping it is required to recreate an index. Changes of the document field mapping include adding or removing fields, changing the type of fields (text, keyword, datatype) and more. Recreating an index requires to repopulate the index, in turn requiring to revisit the original web sites and other sources. Mirroring allows to create and populate a new index using in house data only.
  • In an OTAP setup (ontwikkel - test - acceptatie - productie) it may be useful to mirror a silo in one of the O, T, A or P systems to any of the other systems. Note that this also requires to mirror databases data, in particular documents table esDocuments. Support to mirror or reconstruct table esDocument is not yet available.

Configuration in Web.config

Mirroring requires an Elastic Search endpoint address to send mirrored documents to. File Web.config specifies endpoint addresses, for example:

XML CopyCode image Copy Code
<configuration>
    <appSettings>
        <add key="enterprisesearch.elastic.endpoint" value="http://localhost:9200"/>
        <add key="enterprisesearch.elastic.endpoint.mirror" value="http://localhost:9202"/>
    </appSettings>
</configuration>

  • Key enterprisesearch.elastic.endpoint specifies the regular endpoint address for the currently managed Elastic Search instance.
  • Key enterprisesearch.elastic.endpoint.mirror specifies and endpoint address named mirror.

The endpoint address cannot be entered in the Smartsite manager, for security reasons. Instead the endpoint name mirror is used.

Preparation of a mirror index

Mirroring does not include automatic creation of the mirror index. The silo index and mirror index may differ and may for example use different document field mappings. Or the mirror index may no longer use deprecated index settings and may have introduced new index settings.

The mirror index must exist when mirroring start. Use the manager to create a mirror silo and index for the currently managed ES instance, or use a manager elsewhere to create a mirror silo and index for an ES instance elsewhere.

A mirror silo must not have associated sources. A mirror silo is populated by means of mirroring. Also populating by means of sources would interfere and would yield unexpected results. A switch silo transfers sources from the silo to the mirror silo; see below.

Configuration at silo level

Configuration is completed at silo level. Adjust the silo configuration xml:

XML CopyCode image Copy Code
<data>
    <entry>
        <mirror_indexname>new_index</mirror_indexname>
        <mirror_endpointname>mirror</mirror_endpointname>
    </entry>
</data>
  • Element mirror_indexname specifies the name of the mirror index. This element enables mirroring for the silo.
  • Element mirror_endpointname specifies the above endpoint name. The regular endpoint is used if omitted.

Saving the silo configuration causes mirroring to start.

  • Enterprise Search starts reading documents from the silo index, sending these documents to the mirror index. This can take time.
  • Enterprise search starts mirroring document operations. Whenever a document is added to the silo index, updated in the silo index, or removed from the silo index the operation is also applied to the mirror index.

Stop mirroring, resume mirroring

Mirroring can be stopped by editing the silo configuration. Remove or comment element mirror_indexname.

Mirroring stops upon the save. The mirror index continues to exist and can be deleted manually.

Mirroring can be resumed by restoring element mirror_indexname. A resume starts a fresh check whether documents are present in the silo index and/or mirror index.

  • A document present in the silo index and not present in the mirror index is sent to the mirror index.
  • A document not present in the silo index and present in the mirror index is removed from the mirror index.
  • A document present in the silo index and present in the mirror index is skipped. Updates of documents that occurred when mirroring was temporarily stopped are missed. New updates once resumed are sent to the the mirror index.

Switch silo and stop mirroring

Switching from the silo to the mirror silo is possible if the mirror silo resides in the currently managed ES instance. The Switch silo manager button is enabled if this is the case and if a few other preconditions hold:

  • The silo has a mirror silo
  • The silo has sources
  • The mirror silo has no sources

A switch performs the following:

  • It modifies the silo configuration such that it no longer specifies a mirror silo. Involved xml elements are reworked to comments.
  • It stops mirroring.
  • It transfers all sources of the silo to the mirror silo.
  • It reloads Enterprise Search.

The original silo continues to exist. The original index is no longer updated because the silo lacks sources. The silo and index can be deleted manually.

Note that search applications may still search the original index. Search application should be reconfigured for the new index. Alternatively search applications should search an alias that is stable over time, and the alias should be remapped from the original index to the mirror index. This can be done in the manager. Reconfiguring search applications is not required in that case.