First installation

Smartsite 7.9 - ...

Purpose

The steps below are for a first installation.

Version

Use Elastic Search v7. Several versions are tested; however it is recommended to use the particular version which has been tested in combination with other software components such as the Java runtime environment or the Enterprise Search indexer. See the version table also.

v7.17.3

Elastic Search is not 32 bit or 64 bit specific; this is handled by the Java runtime environment.

Media

The installation media includes folder R12.1\Elastic Search, containing:

elasticsearch-7.17.3-windows-x86_64.zip

Prerequisites

The following prerequisites apply to the Elastic Search server.

The Java run time environment must be available. The Java as bundled with Elastic Search will be used. A separate installation of the Java run time environment is not required.

Installation

This is a new installation. Instructions for an update will be added separately.

Typically install on multiple servers, forming an Elastic Search cluster. For example install on three servers.

Unpack the zip.

Unpack the zip in a temporary folder, for example E:\Temp\elasticsearch-7.17.3-windows-x86_64.
Move folder elasticsearch-7.17.3 to a folder and subfolder under Program Files, for example E:\Program Files\ElasticSearch\7.17.3. Folder 7.17.3 should contain subfolders such as bin and config after moving.

Configure config\elasticsearch.yml. Uncomment and adjust entries:

Set cluster.name. For example: es-cluster.
Set node.name. For example: es-node-1. This is a name for use by Elastic Search, as opposed to a network resolvable server name.
Set path.data with the folder that will contain the Elastic Search data, which can be substantial. For example: E:\ElasticSearch\Data. It is not required to create this folder; this is done by Elastic Search.
Set path.logs with the folder that will contain the Elastic Search logs. For example: E:\ElasticSearch\Logs. It is not required to create this folder; this is done by Elastic Search.
Set the IP address or server name ES will listen to, entry network.host. For example use host-123.example.com. This name should be resolvable on the network. This name is also relevant when securing the cluster, and will for example return as *.example.com in certificates used for https communication.
Consider setting the port; entry http.port. Port 9200 will be used as default if not set.
For a cluster: list the hosts in the cluster, entry discovery.seed_hosts. Use the IP addresses or names as used in entry network.host. For example: ["host-123.example.com", "host-124.example.com", "host-125.example.com"]. The same complete list can be used on each server; the list can include the address of the server itself.
For a cluster: list the master-eligible nodes, entry cluster.initial_master_nodes. Use the node names, not the server addresses. For example: ["es-node-1", "es-node-2", "es-node-3"]. The same complete list can be used on each server; the list can include the name of the node itself.
Add the following comment and setting:

# Do not automatically create a new index because this can result in default mappings.
# The Enterprise Search solution should create an index and should specify appropriate mappings.
action.auto_create_index: .watches,.triggered_watches,.watcher-history-*

If Elastic Search serves on a non-local address allow incoming traffic.

Add a firewall rule. Use an inbound port rule, specifying 9200 or specifying the selected port, for traffic over TCP. Name the rule, for example Elastic-search-7.17.3. Describe the rule, for example Elastic Search index- and search operations.

If the node is part of a cluster allow internode communication.

Add a firewall rule. Use an inbound port rule, specifying 9300, for traffic over TCP. Name the rule, for example Elastic-search-cluster-7.17.3. Describe the rule, for example Elastic Search cluster operations.

Additional configuration is required to secure the cluster, described below. First test interactively:

Start a command box, as administrator.
Go to the bin folder, for example E:\Program Files\elasticsearch\7.17.3\bin.
Run elasticsearch.bat.
Inspect the feedback. Expect a final message like Cluster health status changed from [RED] to [YELLOW]. One reason for a yellow rather than a green status is if the cluster consists of one node only. Or expect a final message like [es-node-1] started.
Visit Elastic Search, using a browser on the server. Issue http://localhost:9200/ for the standard port, or use the selected port. Expect cluster information.
In te command box stop Elastic Search with Ctrl-C.

Secure the cluster

Optionally secure the cluster at these levels:

Enable Transport Layer Security (TLS) for secure communication between nodes of the cluster.
Use TLS / HTTPS for secure communication between clients and the cluster. Clients include the Smartsite Manager, search applications, and Kibana.
Require credentials to access the cluster.

Internode communication

Create a certificate authority (CA) for use in the cluster. Each node will be instructed to trust this CA.

On a node of the cluster: start a command box, as administrator.
Go to the folder above the bin, for example E:\Program Files\elasticsearch\7.17.3.
Run: bin\elasticsearch-certutil.bat ca
Accept: elastic-stack-ca.p12
Create and enter strong password #1.
This generates a CA certificate elastic-stack-ca.p12. Create a folder config\certs and move the certificate to folder certs. For information: there is no need to install this certicate, it is found by means of xpack configuration below.

Generate a certificate and private key for use on each node of the cluster. It is possible to use a distinct certificate per node. Here the same certificate will be used for all nodes.

Continue working in the folder above the bin.
Run: bin\elasticsearch-certutil.bat cert --ca config\certs\elastic-stack-ca.p12
Enter the above password #1 for the CA.
Accept: elastic-certificates.p12
Create and enter another strong password #2.
This generates a certificate elastic-certificates.p12. Move the certificate to folder certs. For information: there is no need to install this certicate, it is found by means of xpack configuration below.

Do not directly add the above passwords to the Elastic Search configuration file elasticsearch.yml. Instead use a keystore (file elasticsearch.keystore, containing passwords, as opposed to certificate elastic-certificates.p12 which contains a private key and which is also called a keystore).

Go to the bin.
Run: elasticsearch-keystore.bat add xpack.security.transport.ssl.truststore.secure_password
Enter the truststore / CA password #1
Run: elasticsearch-keystore.bat add xpack.security.transport.ssl.keystore.secure_password
Enter the node certificate password #2.

This will create or update file config\elasticsearch.keystore.

Enable internode security in config\elasticsearch.yml. Add:

# Cluster security.
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.truststore.path: certs\elastic-stack-ca.p12
xpack.security.transport.ssl.keystore.path: certs\elastic-certificates.p12

Test interactively, as done earlier.

Client communication

The stepts below result in a certificate http.p12 for secure communication between clients and the cluster. The cluster will present the certificate to clients such as browsers, the Smartsite manager, Kibana and curl.

On a node of the cluster: start a command box, as administrator.
Go to the folder above the bin, for example E:\Program Files\elasticsearch\7.17.3.
Run: bin\elasticsearch-certutil.bat http
Generate a CSR? [y/N] => N, the known CA will be used.
Use an existing CA? [y/N] => y
CA Path: certs\elastic-stack-ca.p12. For information: given the current folder one would expect config\certs which however results in an error because config\config\certs is searched. Use certs.
Password for elastic-stack-ca.p12: enter the CA password #1.
For how long should your certificate be valid? [5y] => consider raising this, for example to 30y.
Generate a certificate per node? [y/N] => N, one certificate will be used for all nodes.
Enter all the hostnames that you need, one per line. => *.example.com, the suffix that resolves to the nodes of the cluster.
Enter all the IP addresses that you need, one per line. => enter, no IP addresses
A summary is presented. Do you wish to change any of these options? [y/N] => N
Provide a password for the "http.p12" file: Create and enter another strong password #3.
Repeat password to confirm.
What filename should be used for the output zip file? [C:\Program Files\elasticsearch\7.17.3\elasticsearch-ssl-http.zip] => accept.
Move elasticsearch-ssl-http.zip to config\certs.

The zip contains instructions how to apply the certificate.

Unpack in a temporary folder.
Locate http.p12 and copy this certificate to config\certs.
Enable client security in config\elasticsearch.yml. Add:

# This turns on SSL for the HTTP (Rest) interface
xpack.security.http.ssl.enabled: true
#
# This configures the keystore to use for SSL on HTTP
# Because your keystore has a password, you will also need to add the password to the Elasticsearch keystore
# elasticsearch-keystore add "xpack.security.http.ssl.keystore.secure_password"
xpack.security.http.ssl.keystore.path: certs\http.p12

Also add the password to the keystore:

Go to the bin.
Run: elasticsearch-keystore.bat add xpack.security.http.ssl.keystore.secure_password
Enter the client certificate password #3

Test:

Test interactively, as done earlier.

Require credentials

Require that clienst supply credentials:

In config\elasticsearch.yml add:

# Enable security and require credentials.
xpack.security.enabled: true

Elastic Search uses internal password for its system users, until password are configured. Once configured internal passwords no longer apply. Create and set passwords for all system users.

Make sure Elastic Search is running. Use the above interactive test steps and do not stop with Ctrl-C.
Run (adjust the url): bin\elasticsearch-setup-passwords.bat interactive --url "https://host-123.example.com:9200/"
In turn create and set a password for each system user. For information: this includes [elastic] with password #4 and this includes [kibana_system] with password #5.

Test with a browser.

Restart Elastic Search interactively.
On a remote computer install the CA certificate elastic-stack-ca.p12 in the Trusted Root Certification Authorities store. The result is a certificate installed under a name such as Elastic Certificate Tool Autogenerated CA.
Browse to https://host-123.example.com:9200/_cluster/health?pretty. Expect no warning about an untrusted certificate. Expect a login request. Perform a login using user elastic and using password #4.
Stop Elastic Search with Ctrl-C.

Install as a Windows service

Install and start Elastic Search as a Windows service.

Go to folder bin.
Run elasticsearch-service.bat install.
Run elasticsearch-service.bat start.
Start the services management console and locate the service with a name such as Elasticsearch 7.17.3 (elasticsearch-service-x64).
Set the startup type to Automatic.
Set the recovery mode. First failure: restart the service. Second failure: restart the service. Subsequent failures: take no action.

Test using a browser.

If Elastic Search runs on localhost: visit http://localhost:9200/_cluster/health?pretty. Expect a cluster status summary.
If Elastic Search runs for a server url: perform the test from a browser on a remote system.

Server configuration

Elastic Search heavily uses files for its storage and processing. The following general recommendations apply to the involved folder, for example E:\ElasticSearch\Data.

A faster disk results in better performance, both indexing time and search time.
Disable Microsoft Search and/or other search services, or at least ensure that the ES data folder is excluded from indexing.
Consider excluding the ES data folder for monitoring by virus scanners. The folder does not contain executable files, script files and other files with active content. Technically it is possible to store files in ES, using a field of type binary, which results in the file on disk. The Enterprise Search solution however does not use this support and it can be assumed that files in the data folder are all safe files in terms of virus threats.

Add nodes to the cluster

Add nodes to the cluster, for performance and reliability. Typically add a node by adding a modest server and running one node on that server.

Add a server and perform the above installation steps.
Restart the Elastic Search service on the node and perform the above cluster health sanity check. Expect number_of_nodes to specify the number of nodes in the cluster.

Installer