CLI: Removing incomplete items from the index

Occasionally, there will be tasks that do not complete fully and therefore remain in the index incomplete. This can be caused by:

System shut down while process is running
Application crash
Terminating the process before completion
Connection to the index breaks

To successfully remove incomplete items from the index you will need:

the Task IDs (indexed as _id) of the entries to remove them (example ID: 5f0d242f-eafe-487d-982c-9250bbed4ecf).
a query to tell the index what to delete

Remove ID query

The following query can be used to indicate to the index which items should be deleted.

{
  "query": {
    "ids": {
      "values": [
        "<_id_here>"
      ]
    }
  }
}

Replace <_id_here> with the ID(s) you want to remove.

Remove_ID_query.json

RemoveItems operation

CognitiveToolkit.exe RemoveItems --index-name shinydocs-jobs --index-server-url http://<index_url:port> --query <path_to_Remove_ID_query.json>

Replace http://<index_url:port> with your index URL and port

Replace <path_to_Remove_ID_query.json>with the path to the Remove ID query

After Cognitive Toolkit has gathered the items to remove from the index, you will be prompted in the terminal to confirm the removal of these items. If you do not want to be prompted, use the --force option in the command.