Cognitive Suite 2.9.1.0 (January 2023)

Information on feature enhancements and fixed issues that are part of the Cognitive Suite 2.9.1.0 (January 2023) release.

Features

AddHashAndExtractedText: add --text-timeout configuration in exe.config file

Description

You can now modify the default value for the --text-timeout option for AddHashAndExtractedText.

For example, if there are files that take longer than the default value of 60 seconds to process through Optical Character Recognition (OCR), the full text extraction is not completed for that file.

How to enable

To ensure that the OCR process is completed on each file, modify the --text-timeout value higher than the default setting of 60 seconds in the exe.config file.

CrawlBox: retrieving folder collaborators that have access to the folders and files

Description

When using the CrawlBox operation, the --crawl-collaborators option will retrieve folder collaborators added for a group.

How to enable

To crawl and capture Box groups and their users, include the --crawl-collaborators option when running the CrawlBox operation.

CrawlSharePointOnline: SharePoint Property Bag

Description

SharePoint items contain additional properties that are not captured in the index.

How to enable

To capture additional SharePoint properties, enter a comma-delimited list of properties to pull from SharePoint using the key SharePointPropertyKeys in the config file.

Fixed Issues

AddHashAndExtractedText: false error messages when OCR text extracting (timeout)

Description

Previously, while performing a text extraction to test the text timeout feature in the config file for OCR documents, false error messages were populated in the log files. False error messages are no longer populated.

AddHashAndExtractedText: IronOCR Errors logging document-id in message

Description

Previously, OCR errors were logging the document-id in the message ID field rather than in the in the document-id field column. The ID is now populating into the shinydocs-jobs index document-id field column when there is an error processing files via OCR.
Note: The ID should not be removed from the log file message, as the message field is populated from the log file message, which contains the ID.

AddPathValidation: when the file is detected as in use, mark as valid

Description

When performing a filesystem PathValidation command, errors were displayed for files that were currently open with MS Excel or MS Word. There is no longer an error encountered when running the operation with open files.

CacheFileSystemPermission: cached-owner field is not populating with a value

Description

When running the CacheFileSystemPermission command, the cached-owner field was not populated with a value and no errors were logged in the log file. Now, the cached-owner field is populating as expected.

CLI: email address fields added by operations need to be consistent

Description

CrawlExchange and CrawlFileSystem with --crawl-pst (and text extraction) use different fields to store email addresses. Any CLI tool that stores email addresses in an index will use the same fields. The names were standardized and we use the following names for CrawlExchange and for CrawlFileSystem (--crawl pst).

• receivedBy

• fromAddress

• ccRecipients

CrawlExchange: attachments use "size" field rather than "length"

Description

When using CrawlExchange, the “size” of attachments was recorded using a size field. All other sources use length to record the size of the file. For consistency, CrawlExchange has been changed to use length to record the size of the file.

CrawlExchange: crawling public folders should work on Exchange On-Premise 2019

Description

The exchange source setting json file must contain public-folder-access-email in order to crawl public folders while using CrawlExchange to crawl Exchange On Premise 2019.

Permissions

We strongly recommend creating a new role group in Exchange for managing Shinydocs software with the recommended permissions.

CrawlSharepoint: index UTC times are not correct for SharePoint

Description

SharePoint TimeZones are not transferable to system TimeZoneInfos.

CrawlSharePointOnline: SaveValue does not work for encrypting SharePoint Online secret key

Description

The SaveValue option was previously missing the call to SwapSavedParameters which meant that the option would not work for encrypting SharePoint Online secret key. SwapSavedParameters has been added and the operation is now working as expected. Encrypted IDs can now be successfully added to the saved-parameters.yaml file and the CrawlSharePointOnline tool can be run with a source setting file containing encrypted keys.

CrawlSharePointOnlineSites: provide status update when crawling sharepoint online using site-index

Description

Previously, if you deleted a previously crawled site in SharePoint Online, it would still be in the index and CrawlSharePointOnlineSites would not remove it. Now, a new field called deleted with a value of true is created in the index when a SharePoint site has been deleted and you rerun the CrawlSharePointOnlineSites tool.

CacheFileSystemPermissions: only processing 100 items and getting stack trace

Description

When running the CacheFileSystemPermissions, the progress bar stopped and produced an error message after processing only 100 items. The operation now performs as expected.

ExtractAndCrawlPst: items duplicated when rerun on a pst that was already extracted

Description

When adding a pst to an existing index containing a pst already extracted, the items in the first pst file were duplicated in the file on the fileshare and in the index. To prevent this duplication, a new option is added to the ExtractAndCrawlPst operation: --create-duplicates . Utilizing this command line parameter allows or disallows duplicates to be created when the ExtractAndCrawlPst operation is rerun on a pst file. By default the --create-duplicates value is set to false. That means that if an existing msg file is found, it will be skipped.

RemoveField: setting threads to more than 2 will bypass the provided query and remove all instances of the field in the specified index

Description

Rather than removing the field from the items listed in the query, setting the threads to higher than 2 actually resulted in removing all instances of the field from everything in the index. This has been successfully updated and now RemoveField only removes the field from what was contained in the query.

RestorePermissions - updated error message when permissions cannot be restored

Description

If a document does not have a cached-permissions field (for example, CacheFileSystemPermissions command was not run prior to running RestoreCachedFileSystemPermissions), the permissions for the file will not be reset. This is the intended behaviour; however, the status message in the console suggested that the document was processed successfully. This contradicted the log files, where you will find an error was thrown while processing the document.
The console now displays the correct message while trying to restore the permissions for a file without cached-permissions.

SetFileSystemPermission: allow Security Groups with --identity parameter

Description

Running the SetFileSystemPermissions operation with a security group was previously returning an error. Now, the --identity option can be used with security groups for ease of administration. For example, if there is a change of user permissions related to legal hold documents, user accounts can be updated in the security group instead of running the command each time.

TagDuplicate: change the status of documents that are no longer duplicates

Description

When using the TagDuplicate operation, items that are no longer duplicates can be removed using the --overwrite option.

How to enable

To remove the duplicate value, run RemoveField, and then include the --overwrite option when running the TagDuplicate command. This process will change the value of the remaining file to reflect that it is no longer a duplicate.