What's New in 8.1 - New Release!

Hi all. We recognize that there hasn't been much activity on the blog recently, this is because the team has been heads-down, working hard on a brand new release over the last 12 months.

Don't let the 8.1 moniker fool you, this is a brand new integration, re-developed from the ground up, and using the new SharePoint 2013 app model. The new architecture has given us the opportunity to apply what we've learnt over the last 5 years, and to optimize experience and performance across the board.

We are very proud of what we've built, and excited about the possibilities the new platform and architecture bring. This is very much a solid basis for us to add a lot of new ideas and features over the coming years.

Anyway, on to the headline features and changes in 8.1. More posts will follow with insights and greater depth on each area:

Licensing

HP Records Manager for SharePoint 8.1 is now part of the core HP Records Manager license! So, if you have a current HP Records Manager support and maintenance agreement, you can install and use the integration for free!

SharePoint 2013 app model

The 8.1 release signifies a move from the SharePoint solution model, to the new app model. This means that the 8.1 app release supports SharePoint 2013 only. SharePoint 2010 is still supported with the HP Records Manager 8.1 release, but is just being maintained, there are no new features coming to SharePoint 2010.

The concept of a SharePoint app is that using only a small footprint on the SharePoint farm, it is able to configure UI components such as ribbon buttons. The actual processing provided by an app is performed on an external server, not on the SharePoint server.
This type of SharePoint app is known as a "provider hosted app"

HP Records Manager for SharePoint includes a SharePoint app. This app uses pages and event handlers that are installed on one or more HP Records Manager workgroup servers.
HP Records Manager app architecture

Once the app has been uploaded to the SharePoint corporate app catalog (Note the app is not available on the Microsoft public store, this may come in the future), the app can be added to Site Collections and Sites.

All configuration options are now accessed through what is called an app start page. This, along with all configuration pages, are now hosted on an external server, even though they look like an intrinsic part of SharePoint, neat hey?
HP Records Manager app start page

All new Lifetime Management engine

The current Lifetime Management Policy engine is based on SharePoint workflow, and as a result poses some significant challenges:

  • Very technical to define new policies
  • Inconsistent workflow response and behavior

The upshot being that I don't think many customers successfully adopted a fully implemented policy-based approach to SharePoint governance.

For 8.1, we took an entirely different approach, and have built our own 'event-driven' engine. This does not rely on SharePoint workflow at all, but could be combined with existing workflows, just by monitoring changes to metadata on items.

Lifetime Management Policies(LMPs) are created, managed and applied entirely in the user interface, no more code required!

So, what does a policy look like? A policy is made up of Rule and Action pairs. Multiple rules can be defined, and multiple actions to perform when the rule/s are satisfied. There will be further posts that explore LMPs in greater depth, for now, take a look at this simple example:

First, go to the HP Lifetime Management Policy Gallery, accessed from the new app start page:
Note there are some 20 example policies shipped right out-of-the-box. You can use these to gain an understanding of the types of policies you can deploy, or save copies and modify them to suit your own purposes.

To create a new policy, click on the Create New button, you are presented with the new policy dialog:

In this example I'm creating a policy that will apply to the Document Content Type. As well as basing rules on item metadata, you can also create policies that look at site or list metadata (e.g. Archive this list when nothing has been modified on it for the last 6 months AND it has over 20,000 items).

To add a stage, click on the Add new stage link, this opens a new Lifecycle Stage dialog. In this example I'm adding a rule that says when a document has the word "contract" in the title, carry out the Manage action.

And then I'm adding a second stage with a rule that says when this document hasn't been modified for 3 months carry out the Finalize action.

You can see both stages with their Rule and Action pairs summarized on the new policy dialog.

This policy can now be saved and then applied to relevant locations. LMPs applied at a higher level are automatically inherited down through sub sites and lists, but this can be configured in those cases where you might not want a particular policy to be applied.

Deployment and configuration improvements

Deployment

Deployment and configuration have changed significantly with the move to the app architecture. All configuration data is now stored in a SQL Server database. This does mean access to a SQL Server instance is required. There are a number of supported architectures:
Single Workgroup Server and SQL Server
Single Workgroup Server and SQL Server>

Workgroup Server with separate SQL Server
Workgroup Server with separate SQL Server

Multiple Workgroup Servers with separate SQL Server
Multiple Workgroup Servers with separate SQL Server

Note that centralized configuration also makes scaling much easier. Scenarios with multiple SharePoint Servers integrated with multiple Workgroup Servers are supported through the use of a load balanced URL in front of the Workgroup Servers. All servers share the same configuration database.

This change in architecture paves the way for a whole collection of centralized management ideas that we have, the move to a configuration database is just the beginning. Exciting times ahead!

Configuration

To aid setup and initial configuration, the integration now includes a dedicated configuration tool as part of the installation package. This includes:

  • Initial establishment of common configuration
  • Management of a farm of HPRM workgroup servers
  • Tools
    • Termsets
    • Column management

This tool is run on one of the Workgroup Servers, and stores all resulting configuration in the configuration database. It can be run at any time to retrieve configuration information, or to make changes when required.

Default Site Collection

The integration now includes the ability to define a default site collection. This could be an existing content site, or a dedicated SharePoint 2013 Content Hub. Settings can be configured on the default site collection, and consumed by other site collections. Default configuration includes:

The beauty of this approach, is that new Site Collections can be governed automatically, just by adding the app. No additional configuration is required, as all default behaviors in terms of record types to use, column mappings, and policies are picked up from the central configuration database. Optionally, you can choose to only use the defaults for specific areas of configuration, and override others on the local site collection.

Or consider the scenario where you already have 1000s of site collections, but most of them can be governed in the same way. Nominate a default site collection and perform configuration of policies and mappings. Then deploy the app to all other relevant site collections using a PowerShell script. Job done!

Document Management improvements - Single Document instancing!

Document management has changed in the 8.1 app model. In the past, when managing or finalizing documents in SharePoint, the integration copied the document into HP Records Manager and kept both instances of the document in sync. While this worked, we have always wanted to reduce the storage footprint when managing content.

Now, when you manage or finalize documents in SharePoint, the integration manages the SharePoint document in-place. A metadata-only record is created in HP Records Manager to represent the SharePoint document. HP Records Manager users can still view the document, it is retrieved from SharePoint for viewing, however edits can only be performed from SharePoint.
As soon as the document is either relocated, archived, or deleted, it is transferred into HP Records Manager for long-term storage.
This approach still delivers full in-place management. Once a document is managed by HP Records Manager, all events are tracked and authorized by HPRM as the record authority.

Versioning is also supported, the Default Integration Settings page includes an option to Capture all versions. This determines what happens on Relocate/Archive/Delete, when there are multiple document versions.
When Capture all versions is not enabled, only the latest version of the document is captured.
When Capture all versions is enabled, all versions of the document are captured.

Redesigned job queue

The job queue has been re-designed, now pretty much everything the integration does is performed asynchronously, through the job engine. This includes all management tasks, application of LMPs, mappings and maintenance. The job queue now runs on the Workgroup Servers, and no longer uses SharePoint timer jobs. This results in much quicker response to newly added jobs, and much more consistent performance. There are no longer any instances of saving configuration somewhere and having to watch a progress indicator until configuration has been applied. The user experience is much more fluid as a result.

The job monitoring page has been rebuilt, and now provides multiple pages, including a dashboard overview:

The job queue now processes events from HP Records Manager, as well as SharePoint events, resulting in greater robustness and performance.

Again, the job queue is an area we are looking to build on, particularly in relation to monitoring and troubleshooting.

SharePoint Online(Office 365) support!

The new app architecture and minimal SharePoint footprint has allowed us to deliver something we have been working on for many years, SharePoint Online(Office 365) support! Not only can we manage SharePoint Online content, using the same app, but we can also support HP Records Manager operating in the cloud. In fact, we have tested and developed the online aspects using Windows Azure. This is a very exciting area, as ultimately it may allow us to provide trials of the integration using an Azure environment paired with SharePoint Online trials.

Windows Azure also provides the ability to scale out in a mixed environment, where some of your Workgroup servers are on-premise, and some are running on the Azure platform, all managing the same SharePoint farm.

What didn't make it

Unfortunately there are a few things that didn't make it into this release, purely because of resource and time constraints. However, I can sure you that these are at the top of the teams' list, in terms of development priorities. Please feel free to use the comments to campaign for the features you feel are a priority for your organization/customers. We are listening.

- Exposure
- Security
- Template Defaults

Summary

8.1 is a major release for the integration team. Here's a headline list of the new features:

- New app architecture - SharePoint 2013
- All new Lifetime Management Policy engine
- Big changes to deployment and configuration
- Centralized configuration - Default Site Collection
- Document Management improvments - manage in-place
- Redesigned job queue, and performance improvements
- Support for SharePoint Online(Office 365) and Windows Azure

Please subscribe to the blog to receive regular updates, tips and tricks, and more in-depth articles on the new features. Feel free to feedback to the team using comments or the direct contact form.

General availability for 8.1 will be announced very soon, watch your normal HP Software channels for details on how to grab the 8.1 release.