Shinydocs is Migrating to OpenSearch

Shinydocs has decided to make the move from Elasticsearch to OpenSearch. An official date for the migration has not yet been set, however, we plan to release Cognitive Suite with OpenSearch by the end of September 2023. We will ensure that you remain informed and provide individual assessments to determine the optimal migration approach for your situation.

To mitigate security and business risk resulting from the Elasticsearch decision to move away from Apache 2.0 in v7.11, Shinydocs has decided to migrate to OpenSearch for both search and visualization functionality:

Elasticsearch → OpenSearch

Kibana → OpenSearch Dashboard

Impacted products

All Shinydocs products that are currently built on the Elastic Stack are being converted to OpenSearch, including:

• Cognitive Toolkit

• Enterprise Search

• Review

• Visualizer

OpenSearch Features

OpenSearch is as scalable, flexible, and extensible as Elasticsearch for search, analytics, and observability applications. Additionally, OpenSearch includes various security features for authentication, authorization, and access control that allow for centralizing user management and audit/compliance logging. 

Among the key motivations for making the switch to OpenSearch is unlocking the various security features it provides out-of-the-box and potentially integrating them into Shinydocs products in the near future, post-migration to OpenSearch.

OpenSearch includes the following features out-of-the-box:

Security out of the box – OpenSearch comes with various security features, such as access control with centralized user account management tools, enabled TLS/SSL by default, as well as various authentication backend connections (for example, LDAP and OpenID). This allows administrators to control who can login/access the API and what data each user can view and change using an existing authentication system.

Cross-cluster replication – The cross-cluster replication plugin lets administrators replicate indexes, mappings, and metadata from one OpenSearch cluster to another. Cross-cluster replication has the following benefits:

• Replicating indexes makes it possible to continue to handle search requests if there’s an outage

• Replicating data across geographically distant data centers minimizes the distance between the data and the application server. This reduces expensive latencies

• Replicating data from multiple smaller clusters to a centralized reporting cluster is useful when it’s inefficient to query across a large network

• Disaster recovery in case a primary cluster fails. A secondary cluster can serve as a hot backup

Configurable Retention Period – OpenSearch allows administrators to set and update the retention period of some indices, such as those keeping system logs, and circulate them.

What to expect

With the first release of Shinydocs products built on OpenSearch, this is what you can expect:

• Our products will connect to either Elasticsearch or OpenSearch until such time as Shinydocs stops supporting Elasticsearch (TBD)

• Continuation of the Apache 2.0 Licenses to decrease business risk

• We’ll provide guidance on:

  • how to install and configure OpenSearch and OpenSearch Dashboard in different environments

  • how to migrate Elasticsearch indices over to OpenSearch

• Various performance and functional improvements in the back end will be implemented:

  • Improved search and classification

  • Thorough (automated) test coverage of search engine-related functionalities