Adding a path validation and removing items

When data on an index is moved or changed an Index Refresh/Path Validation is needed in order to identify the invalid items. Once items have been marked as invalid, you can run a Command in Dev Tools or a Power Shell script to remove them from the index.

Path Validation

1. Open up the Command Prompt

2. Enter the following command in the Cognitive tool:

CognitiveToolkit.exe AddPathValidation -u http://localhost:9200 
-i INDEXNAME -q "Resources\QueryFiles\query-validPath.json"

Update the INDEXNAME and verify the path for the query file

Valid Path Query

1. Copy the below script into Notepad ++

2. Save file to Resources\QueryFiles

3. Be sure to name the file: query-validpath.json

{
	"bool": {
		"must": [
			{"exists": {"field": "path"}}
		],
		"must_not": [
			{"match_phrase": {"path-valid": "false"}}
		]
	}
}

Remove Items that No Longer Exist from Index

1. Open the Shinydocs Analytics/Dashboard

2. Navigate to the Dev Tools Section

3. Copy the script and paste into the Dev Tools. Change INDEXNAME to the name of the index that the path validation was completed on

   CODE

   POST INDEX_NAME/_delete_by_query
   {
   	"query": {
   		"bool": {
   		"must": {
   			"match": {"path-valid": "false": "true"}
   				}
   			}
   		}
   	}

4. Click the play button on the script