Over the past few months a remarkable collaboration has taken place between three parties: Pingar NZ, Synercon Australia and our client Auckland Transport.

We each brought existing knowledge and toolsets, which we combined to build an Intelligent Automation platform for records and information management.  We used Synercon’s a.k.a.® software for development of the taxonomies, retention schedules and appraisal models, and Pingar’s DiscoveryOne software for auto-classification, auto-appraisal and date extraction. We generated metadata through text analytics of content and capture of linked metadata. Then we built a workflow engine to manage the records disposal process, using the metadata we had generated. All of this was done within the Auckland Transport hybrid SharePoint environment.
The results have been compelling.

What’s different about our approach?

Contemporary content management systems (CMS) offer some level of auto-classification for records management. Typically, they auto-classify records using machine learning to locate documents within a file plan or into library/folders mapped to records disposal schedules. What this means for business users is that they must work within a conforming information architecture.

Typically, this is based on a hierarchical Business Classification Scheme (BCS) which describes the functions and activities of an organisation. Over 20 years, the BCS has been the primary form of classification. It is the established records management standard. But it constrains the way in which users can find, analyse and organise content, whereas faceted classification enables a more agile and extensible approach.

Definition: Faceted classification allows the assignment of multiple classifications to an object, allowing searching and browsing of related information through several classes. Elements may include activity, subject, geographical, temporal and form of an item.

What was needed was a solution that delivers benefits to all stakeholders. And this is what we achieved in the Auckland Transport project.

Phase One: Establishing the foundations

Firstly, we established faceted classification schemes for auto-tagging content with rich metadata.

We created an extensive metadata set to describe the records in detail. At Auckland Transport we tagged with ‘Asset Names’, ‘Asset Types’, ‘Activities’, ‘Document Categories’, ‘Events’, ‘Locations’, ‘Mandates’, ‘Projects’, ‘Services’, ‘Stakeholders’ and ‘Stakeholder Types’, etc, according to the division we were working with.

The initial tagging delivered a foundation of rich metadata that enabled all of the subsequent information management processing – and provided us with the ability to drive multiple solutions off the same data set.

Immediate payback

The first benefit from this process was an immediate improvement in findability. By using faceted classification, we enabled all of the faceted search features that SharePoint provides, such as Views, Filters and Refiners.

The second benefit was the improved visibility of the records, viewed directly through Sharepoint or indirectly through analytic tools. Using Microsoft’s Power BI we sliced and diced the data to reveal insights about how and where the business was actually managing its records.

Phase Two: Automating records appraisal

Once the records were classified, we had the metadata we needed to embark on the second phase of our project – automating the records appraisal process.
Definition: Records appraisal is the process of determining the value of records, what to keep, how long to keep it.  These decisions are expressed in records authorities (or disposal schedules).

Appraisal is an intensive and repetitive task requiring specialised knowledge. Given the increasing volumes of records being generated the task is beyond humanly possible. It’s a job that well suited for intelligent automation.

Most auto-classification engines can handle simple appraisal tasks like determine specific document types. But they struggle to manage more complex determinations where the context of the record is based on multiple factors.

Our solution was to unpick the process of appraisal and build the knowledge that an archivist would apply into our data models.  We created contextual pathways within our model and mapped these pathways to the disposal schedule. We then tasked the auto-classification engine with finding the metadata combinations and tagging with disposal rules. We can the use the disposal tags to map the documents against the BCS.

We also set the auto-classification engine to extract relevant dates from within documents.
Once we had tagged the records with the disposal class and the trigger date, we were able to capture additional metadata from the retention schedule and calculate the appropriate disposal dates for processing. We now had all the metadata needed to be able to dispose of the records.

Phase Three: Managing records disposal

The final phase of our project was to process the records through disposal.
The process of records disposal includes both destruction and transfer (to an archive or repository for long term storage). For most organisations’ disposal is authorized by laws and regulations, and the process of disposal must be documented to ensure that disposal is authorised through a disposal authority and by the organisations’ senior management.

We build a simple engine to manage the disposal process, managing approvals through workflows, capturing the requisite metadata in SharePoint, and transferring the permanent records into a customised SharePoint library prior to their transfer to the Auckland Council archives.

All of the processing was undertaken in the background without any noticeable impact on the user or the SharePoint information architecture.

To make an omelette you need to break a few eggs

Metadata has been the key to our approach. We pooled our knowledge of taxonomy and ontology standards, data modelling, semantics, natural language processing, and machine learning. We unscrambled the BCS and disposal schedules to build the metadata models that fed the auto-classification and auto-appraisal engines. Through auto-classification we generated rich metadata describing the records and providing the foundation for further automated processing.

It’s an approach that really works.

But you also need a platform, like SharePoint, which delivers an agile information architecture and supports a faceted classification solution. Which is a problem for content management systems wedded to their file plans and legacy architectures.