Testing PII and ROT rules | Shinydocs Pro

This article explains how to test the query (“RawQuery”) from custom PII and ROT rule files.

Review the below two articles for more information on PII and ROT.

Review the following article for more information on custom PII and ROT.

Adding custom PII and ROT rules to Shinydocs Pro Control Center

Before applying new custom rules or editing existing rules in Control Center, you may want to test the query portion of the rule file for the following reasons:

Verify that the syntax of the “RawQuery” is correct
Verify that your query will return the expected number of results against your data source(s)
Edit the query until the results against your data source(s) are satisfactory

Examples of existing rules:

PII (potential_pii-Email_address.json)

{
"Name": "Email address",
"Description": "",
"FieldName": "potential_pii",
"FieldValue": "email_address",
"Sensitivity": 0,
"RegexPattern": "(\\b[A-Za-z0-9.!#$%^&'*+/=?^_`{|}~-]+@[A-Za-z0-9-]+(?:\\.[A-Za-z0-9-]+)\\b)",
"LuhnValidate": "false",
"RawQuery": {
"bool": {
"must": [
{
"exists": {
"field": "fullText"
}
}
],
"must_not": [],
"should": [
{
"match_phrase": {
"fullText": "e-mail"
}
},
{
"match_phrase": {
"fullText": "email"
}
},
{
"match_phrase": {
"fullText": "emailaddress"
}
},
{
"match_phrase": {
"fullText": "user"
}
},
{
"match_phrase": {
"fullText": "login"
}
},
{
"match_phrase": {
"fullText": "username"
}
},
{
"match_phrase": {
"fullText": "contact"
}
}
],
"minimum_should_match": 1,
"filter": []
}
}
}

ROT (rot_obsolete-log.json)

{
"Name":"Log",
"Description":"Old log files",
"FieldName":"rot_obsolete",
"FieldValue":"log",
"RawQuery": {
"bool": {
"must": [
{
"terms": {
"extension": [
"log",
"LOG"
]
}
}
],
"filter": {
"range": {
"lastWriteTimeUtc": {
"lte": "now-6M/d"
}
}
},
"must_not": [
{
"term": {
"rot_obsolete": "log"
}
}
]
}
}
}

Steps to test query:

In both examples above, we can test the query portion of rules to see what the results will be against your data source(s).

Open the PII or ROT rule (json file) you want to test the query for
- you can open this in a text editing applications like Notepad, Notepad++, Visual Studio Code, etc.
Copy the text from the “RawQuery” portion of the file
- starting from the first brace (curly bracket) after "RawQuery":, to the second last brace (curly bracket) of the text
- example below
Login to OpenSearch Dashboards. The Default URL is http://localhost:5601/
Select the hamburger menu icon (from the top-left corner of the page), then select “Discover”
From the drop-down menu in the top-left corner of the page, select the data source you want to test your query against
Select “Add Filter” and “Edit as Query DSL”
Remove the existing default braces (curly brackets)
Paste the text you copied from the “RawQuery” portion of the file (in Step #2), then select “Save”
This should show you the results of your query against whatever data source you selected

Review the results of your query to see if these align with your expectations.

Additional Options:

Adding columns

You can add the specific metadata fields as columns that you would like to see for your results, which may make it easier to review

On the left hand side of the page, select the “+” icon next to each of the metadata field names that you would like to display for your query results
In the example below, we added the “name”, “parent” and “extension” metadata fields as columns
Since our example query above is only looking at “extension” = “log”, all of our results are .log files

Things to check

A filter or syntax error. Make sure you copied only the RawQuery object, from its opening brace to its matching closing brace, and that every brace and bracket is closed. Field names are case-sensitive: fullText, not fulltext.
0 results. Check that the data source you picked is the one you expect, that the source has run, and that the fields your query uses exist on its documents (for example, fullText only exists once Reading file contents has run). Extensions are stored in lowercase without the dot, so use log, not .log. Check date ranges such as now-6M/d.
More results than expected for a PII rule. Discover shows every document the query finds. When the rule runs, Shinydocs Pro also checks each one against the rule's RegexPattern (and the Luhn check, if it is on), so fewer documents are tagged.
Results you expected are missing. A must_not on the rule's own field leaves out documents the rule has already tagged, and the exclusion clause leaves out files tagged Exclude-from-PII-detection or Exclude-from-ROT-detection. Remove those clauses while you test if you want to see every match.