While previously it was already possible to use Elasticsearch with your Hypernode by connecting to an external search provider, we have now made it possible to use Elasticsearch for search in your Magento shop out of the box on Hypernode without requiring any external service or configuration. In this article we’ll explain a bit more about Elasticsearch and how to enable and configure it.

Please note that due to the relatively heavy resource requirements for Elasticsearch this feature can only be enabled for Magento Professional M plans or larger. If you are on a smaller plan and previously already depended on an external paid Elasticsearch provider now might be a good time to consider simplifying your setup.

What is Elasticsearch?

Elasticsearch is a powerful search engine built around Apache Lucene which was later merged with Solr. A long time ago we added Solr as an opt-in feature for Hypernode, but back in 2017 we again deprecated that feature on Hypernode in favor of Sphinx as Solr integrations were rarely used, and if they were an external solution would often be more fitting. Now in 2019 the landscape has shifted and Elasticsearch is the big winner in the world of search.

The implementation of Elasticsearch on Hypernode is built for this primary use-case of search in a Magento or Shopware shop. Because of this there are some constraints that you need to be aware of. If you intend to use Elasticsearch in your Magento/Shopware shop as the search engine like described in the Magento/Shopware docs or in one of the well known third party modules like Smile ElasticSuite then Elasticsearch on Hypernode will give you a plug-and-play installation ready to go. But if you intend to do more than that like building a full ELK Stack around your shop with custom integrations, then using an external search provider might still be a better fit, both for flexibility and for guarantees regarding persistent data.

If you do want to treat Elasticsearch as a primary database, you should not use the Elasticsearch instance on the Hypernode itself but arrange something externally instead. While we will keep the data between plan upgrades and downgrades, you should regard the built-in Elasticsearch on Hypernode as a cache (akin to how we treat Redis on Hypernode), and not as a consistent database. It is possible that due to future upgrade paths or as a result of a backup restore scenario the data stored in the Elasticsearch will be lost.

In our release note you’ll find a more detailed description of Elasticsearch.

Enabling Elasticsearch on Hypernode

Elasticsearch can be enabled on Hypernodes that have a Magento Professional M plan or larger. To enable Elasticsearch, you can talk to the API or use the hypernode-systemctl command-line tool. First configure the Elasticsearch version you wish to use. The default will be the officially supported 5.2.

To switch to 6.6 , you can change this value by issuing the following command:

At this point, Elasticsearch is not yet enabled. You can verify this by checking if the process is running and by looking at the configured value according to the API:

When you choose to enable Elasticsearch on a production node, keep in mind that Elasticsearch will use a lot of resources. We allocate up to 10% of the system memory as the JVM heap, with a maximum of 10GB or RAM. Note that the service can have a big impact on the CPU and IO utilization of your Hypernode. If you notice adverse effects, you can disable the service again or consider upgrading (especially if you already have other opt-in features like RabbitMQ enabled as well).

To enable Elasticsearch you can run the following:

If you are on a plan that does not support it, you will see the following message:

Once you have enabled Elasticsearch you use hypernode-log or livelog to keep track of the update_node job that will deploy the software. Once that has completed you should see the new process on the node:

Note that we only bind the Elasticsearch ports on the local host:

If you wish to access them directly from your computer (instead of only having the shop talk to the instance), you can perform a local forward to tunnel the port locally:

If you ever need to switch between 5.2 or 6.x or back after you have already been using it, note that when you change versions all Elasticsearch data will be deleted. When you are using the default Magento implementation that should be OK since you can just re-index the data.

Configuring Magento to use Elasticsearch

Since Magento 2.3, Elasticsearch can be used as a default search provider. Here we will provide a short walk-through of how to configure using this default integration on your Hypernode. Once you’ve made sure you have enabled Elasticsearch and set the version to 5.2 (this is currently the default version on Hypernodes), you can configure Magento to use it as the search backend. This example will use a fresh Magento 2.3 shop with sample data.

First, log in to the Magento backend with an admin account. In my case with the sample shop I first had to create that:

Then, just follow the first step from this guide. Navigate your way through Stores > Settings > Configuration > Catalog > Catalog > Catalog Search. On that page you can configure the settings as such:

You can press Test Connection to test the connection. If all is well you should see Successful! Test again?. Once that works remember to press Save Config. The next step is to re-index all indexers. In the command-line run the following:

If you get something like:

It is possible that you accidentally saved Elasticsearch as a Search Engine instead of ElasticSearch 5.0+ on the catalog configuration page from the screenshot above.

You should now be able to list the created index using the _cat API:

Note that the status is yellow instead of green. This is because the index was created with an expected number of replicas. Because on Hypernode there is no Elasticsearch cluster but there is just a single instance, this is expected. If you need guarantees about the replication of your Elasticsearch data we recommend that you use another solution. For some examples check this list of external Elasticsearch providers that we recommend.

You can configure your created index to expect an amount of zero replicas. That will turn the cluster status back to green. Since Elasticsearch 5.x it is not possible to set index level settings in the node configuration, but you can do so during runtime with this command:

This will turn the status back to green, which might be useful if you want to use this endpoint for any kind of monitoring:

Improved search

Once Elasticsearch has been configured in the Magento installation you will be able to see its effects on the search. Here are three examples of search results in the yoga-themed Magento 2.3 sample shop. The first screenshot is the default behavior without Elasticsearch when searching for “yogaa” instead of “yoga”.

This next screenshot is what you would see if you have Elasticsearch configured according to the explanation given in the previous section. This is the built-in out of the box Magento integration. It will now give you some suggestions about what you might have meant to search for.

Finally we also have a screenshot of what the search might look like with the third-party Smile ElasticSuite, using the new Elasticsearch 6.6. Here you can see that the search result for the “yogaa” typo will not only show you the term you might have meant, it will also immediately show you the products that you were really looking for.