Sitecore and GDPR in practice - Smart Sitecore Blog

This article is a supplement to my sitecore.stackexchange response, to “Sitecore, personal data and General Data Protection Regulation (GDPR)” question, please check it before reading.

I won’t cover here what GDPR is in general. There are already many great articles about it from Sitecore: GDPR Sitecore 6, 7, 8, GDPR for Sitecore 9, or external: http://www.sphammad.com/blog/gdpr-all-you-you-need-to-know-with-templates or http://www.gdpr-legislation.co.uk/ Instead I will focus on GDPR related issues you may find in your Sitecore XP implementation.

Personal Data Gathered by Sitecore Experience Platform

If you think about Sitecore and personal data, most probably you have in mind Experience Profile contact’s details:

The data displayed in Experience Profile is loaded directly from Analytics Database (MongoDb in Sitecore 8) from:

Contacts collection:

Identifiers collection (identifier may be for example email address):

Basic personal data is also available in Experience Profile contacts list:

The data for it is loaded from sitecore_analytics_index in Sitecore 8:

How personal data may appear in Experience Profile?

when anonymous visitor is identified:

visitor is identified by custom code, e.g. while registering, login, reading geolocation data from IP, etc. Example Sitecore 8 code to identify contact:

// get the personal facet for current contact
var contact = Sitecore.Analytics.Tracker.Current.Session.Contact;
var contactPersonalInfo = contact.GetFacet<Sitecore.Analytics.Model.Entities.IContactPersonalInfo>("Personal");

//  set the name
contactPersonalInfo.FirstName = "Tomek";
contactPersonalInfo.Surname = "Testing";

// set the email
var contactEmail = contact.GetFacet<Sitecore.Analytics.Model.Entities.IContactEmailAddresses>("Emails");
if (!contactEmail.Entries.Contains("Home"))
{
    contactEmail.Entries.Create("Home");
}
var email = contactEmail.Entries["Home"];
email.SmtpAddress = "tomek@smartsitecore.com";
contactEmail.Preferred = "Home";

// get the personal facet for current contact

var contact = Sitecore.Analytics.Tracker.Current.Session.Contact;

var contactPersonalInfo = contact.GetFacet<Sitecore.Analytics.Model.Entities.IContactPersonalInfo>("Personal");

// set the name

contactPersonalInfo.FirstName = "Tomek";

contactPersonalInfo.Surname = "Testing";

// set the email

var contactEmail = contact.GetFacet<Sitecore.Analytics.Model.Entities.IContactEmailAddresses>("Emails");

if (!contactEmail.Entries.Contains("Home"))

{

contactEmail.Entries.Create("Home");

}

var email = contactEmail.Entries["Home"];

email.SmtpAddress = "tomek@smartsitecore.com";

contactEmail.Preferred = "Home";

visitor is identified by WFFM “Update contact details” save action
visitor is identified by custom connector

when contacts are uploaded in List Manager app, eg for EXM:

Data used in this app in Sitecore 8 is stored in:

sitecore_analytics_index:
Analytics Database in Contacts collection
SQL Reporting database:

when Sitecore gathers specific page events or query string parameters from visitor interactions:

Personal data may appear in Analytics database in Interactions collection (and sitecore_analytics_index in Sitecore 8) with your custom goals, page events or query string parameters related with visit, so you should check what additional data you are saving there (e.g. you may have there full history of the user’s search, or parameters you pass to fill WFFM form, etc).

Personal data gathered by XP for contacts enrolled into marketing automation plan can be also displayed in Marketing Automation app:

Hints:

Sitecore Contact’s Id is persisted in SC_ANALYTICS_GLOBAL_COOKIE in visitor’s browser:This value is saved to databases (analytics and reporting) and sitecore_analytics_index.
To have same values of Contact Id in Mongo Analytics database and in the cookie, you should use .Net encoding. For example with this option in Robo:
In sitecore_analytics_index contact id is usually (but not always) stored without dashes.
Sitecore 9 won’t index personal data in by default. It also won’t index your custom facets data if you mark it [PIISensitive] attribute.

Personal Data in Sitecore User Manager

You may also have user’s personal data stored in Sitecore User Manager:

Data source for this, is typically ASP.Net Membership database tables (which are by default in Sitecore core database), but in can be also implemented in custom way (e.g. with Active Directory connector). Profile data can be also manipulated from code with:

var profile = Sitecore.Context.User.Profile;
profile.Email = "tomek@smartsitecore.com";
profile.Name = "Tomek";
profile.ProfileItemId = "{C56A4180-65AA-42EC-A945-5FD21DEC0538}";
profile.SetPropertyValue("Phone", "555-555-555");
profile.Save();

var profile = Sitecore.Context.User.Profile;

profile.Email = "tomek@smartsitecore.com";

profile.Name = "Tomek";

profile.ProfileItemId = "{C56A4180-65AA-42EC-A945-5FD21DEC0538}";

profile.SetPropertyValue("Phone", "555-555-555");

profile.Save();

Restrict Access to Personal Data in Sitecore Back-office

Because user has the right to be informed what is going on with his personal data (you should inform him about his rights on your website, e.g. in pop-up window), you need to have full control who has access to this data. Otherwise you cannot guarantee how personal data gathered by you is used. In Sitecore back-office you can achieve this with limiting access to admin account and setting security rights on following core db items:

SPEAK applications under:

/sitecore/client/Applications

1

/sitecore/client/Applications
Launchpad buttons under:

/sitecore/client/Applications/Launchpad/PageSettings/Buttons

1

/sitecore/client/Applications/Launchpad/PageSettings/Buttons
For WFFM reports, you will need to change security settings for ribbon button as well:

/sitecore/content/Applications/Content Editor/Ribbons/Contextual Ribbons/Forms/Form/Forms/Form Reports

1

/sitecore/content/Applications/Content Editor/Ribbons/Contextual Ribbons/Forms/Form/Forms/Form Reports

Right to be Forgotten

Sitecore 8 prior 8.2 update 7: Sitecore doesn’t have documented API to remove data from MongoDB, but you can rather easily delete the data using standard MongoDB .Net provider shipped with the platform.

var connectionString = ConfigurationManager.ConnectionStrings["analytics"].ConnectionString;
var url = new MongoUrl(connectionString);
var database = new MongoClient(url).GetServer().GetDatabase(url.DatabaseName);

var update = new UpdateBuilder();
update.Unset("Identifiers");
update.Unset("Personal");
update.Unset("Emails");
update.Unset("Tags.Entries.ContactLists");
var queryContact = Query.EQ("_id", Guid.Parse("4c6220e2-ac9f-43f0-8348-4297eead3d38"));
var status = _repository.GetCollection("Contacts").Update(queryContact, update);

var connectionString = ConfigurationManager.ConnectionStrings["analytics"].ConnectionString;

var url = new MongoUrl(connectionString);

var database = new MongoClient(url).GetServer().GetDatabase(url.DatabaseName);

var update = new UpdateBuilder();

update.Unset("Identifiers");

update.Unset("Personal");

update.Unset("Emails");

update.Unset("Tags.Entries.ContactLists");

var queryContact = Query.EQ("_id", Guid.Parse("4c6220e2-ac9f-43f0-8348-4297eead3d38"));

var status = _repository.GetCollection("Contacts").Update(queryContact, update);

For updating Analytics Index you can use https://github.com/vhil/helpfulcore-analytics-index-builder, it’s very simple actually, e.g. to update index for contact entity:

IAnalyticsIndexBuilder analyticsIndexBuilder = (IAnalyticsIndexBuilder)Factory.CreateObject("helpfulcore/analytics.index.builder/analyticsIndexBuilder", true);
analyticsIndexBuilder.RebuildContactIndexableTypes(new List<Guid> { Guid.Parse("4c6220e2-ac9f-43f0-8348-4297eead3d38") });

1 2	IAnalyticsIndexBuilder analyticsIndexBuilder = (IAnalyticsIndexBuilder)Factory.CreateObject("helpfulcore/analytics.index.builder/analyticsIndexBuilder", true); analyticsIndexBuilder.RebuildContactIndexableTypes(new List<Guid> { Guid.Parse("4c6220e2-ac9f-43f0-8348-4297eead3d38") });

For some of entities you would need to use Sitecore.ContentSearch API to search and remove documents from the index (for example entries gathered by EXM can’t be removed with the module from Github, cause they don’t connect with Contact entity directly (they connect to interactions instead):

var interactionIds = new List<Guid> { Guid.Parse("599a1964-714e-4118-8c72-133f4cc22305") };
var interactionIdsIndexed = interactionIds.Select(x => x.ToString().ToLower().Replace("-", ""));

//search for indexed documents
List<SearchResultItem> interactions;
using (var context = ContentSearchManager.GetAnalyticsIndex().CreateSearchContext())
{
    interactions = context.GetQueryable<SearchResultItem>()
        .Where(x => interactionIdsIndexed.Contains(x["visit.interactionid_s"]))
        .ToList();
}

var uniqueIds = new List<IIndexableUniqueId>();
foreach (var interaction in interactions)
{
    //optionally filter by e.g. page event name
    uniqueIds.Add(new IndexableUniqueId<string>(interaction["_uniqueid"]));
}

//remove indexed document
using (var context = ContentSearchManager.GetAnalyticsIndex().CreateDeleteContext())
{
    foreach (var indexableId in uniqueIds)
    {
        context.Delete(indexableId);
    }
    context.Commit();
}

using (var context = ContentSearchManager.GetAnalyticsIndex().CreateDeleteContext())
{
    foreach (var indexableId in uniqueIds)
    {
        context.Delete(indexableId);
    }
    context.Commit();
}

var interactionIds = new List<Guid> { Guid.Parse("599a1964-714e-4118-8c72-133f4cc22305") };

var interactionIdsIndexed = interactionIds.Select(x => x.ToString().ToLower().Replace("-", ""));

//search for indexed documents

List<SearchResultItem> interactions;

using (var context = ContentSearchManager.GetAnalyticsIndex().CreateSearchContext())

{

interactions = context.GetQueryable<SearchResultItem>()

.Where(x => interactionIdsIndexed.Contains(x["visit.interactionid_s"]))

.ToList();

}

var uniqueIds = new List<IIndexableUniqueId>();

foreach (var interaction in interactions)

{

//optionally filter by e.g. page event name

uniqueIds.Add(new IndexableUniqueId<string>(interaction["_uniqueid"]));

}

//remove indexed document

using (var context = ContentSearchManager.GetAnalyticsIndex().CreateDeleteContext())

{

foreach (var indexableId in uniqueIds)

{

context.Delete(indexableId);

}

context.Commit();

}

using (var context = ContentSearchManager.GetAnalyticsIndex().CreateDeleteContext())

{

foreach (var indexableId in uniqueIds)

{

context.Delete(indexableId);

}

context.Commit();

}

If you removed user from list manager (there’s no point to keep anonymous users without email address in the list), you will also need to update recipients count in list item using ListManager<TContactList, TContactData> class to keep your data consistent.
Additionally you need to update Contacts table in SQL Reporting database for ContactId equals _id from Contacts collection:

var command = new SqlCommand
{
    CommandText = "UPDATE dbo.Contacts SET ExternalUser = @ExternalUser, ContactTags = @ContactTags, IntegrationLevel = @IntegrationLevel WHERE ContactId = @ContactId"
};
command.Parameters.AddWithValue("@ExternalUser", "");
command.Parameters.AddWithValue("@ContactTags", "<tags/>");
command.Parameters.AddWithValue("@IntegrationLevel", 0);
command.Parameters.AddWithValue("@ContactId", "4c6220e2-ac9f-43f0-8348-4297eead3d38".ToUpper());
using (SqlConnection sql = new SqlConnection(ConfigurationManager.ConnectionStrings["reporting"].ConnectionString))
{
    sql.Open();
    command.Connection = sql;
    command.ExecuteNonQuery();
}

var command = new SqlCommand

{

CommandText = "UPDATE dbo.Contacts SET ExternalUser = @ExternalUser, ContactTags = @ContactTags, IntegrationLevel = @IntegrationLevel WHERE ContactId = @ContactId"

};

command.Parameters.AddWithValue("@ExternalUser", "");

command.Parameters.AddWithValue("@ContactTags", "<tags/>");

command.Parameters.AddWithValue("@IntegrationLevel", 0);

command.Parameters.AddWithValue("@ContactId", "4c6220e2-ac9f-43f0-8348-4297eead3d38".ToUpper());

using (SqlConnection sql = new SqlConnection(ConfigurationManager.ConnectionStrings["reporting"].ConnectionString))

{

sql.Open();

command.Connection = sql;

command.ExecuteNonQuery();

}

Sitecore 8.2 update 7: you call new pipeline, responsible for removing personal data:

var args = new Sitecore.Analytics.Pipelines.RemoveContactPiiSensitiveData.RemoveContactPiiSensitiveDataArgs(contactId);
Sitecore.Pipelines.CorePipeline.Run("removeContactPiiSensitiveData", args);

1 2	var args = new Sitecore.Analytics.Pipelines.RemoveContactPiiSensitiveData.RemoveContactPiiSensitiveDataArgs(contactId); Sitecore.Pipelines.CorePipeline.Run("removeContactPiiSensitiveData", args);

You can look at this pipeline configuration in Sitecore.Analytics.config and add your custom facets you want to remove. By default it removes: Addresses, Emails, Personal, Phone Numbers and Picture facets.

Sitecore 9: you can call ExecuteRightToBeForgotten method from XConnectClient class: https://doc.sitecore.net/developers/xp/xconnect/xconnect-client-api/contacts/right-to-be-forgotten.html

Exporting Personal Data (Right to Data Portability)

Sitecore 8 prior 8.2 update 7: You can extract all the data form xDB Contacts and Interactions collections using Mongo.Net driver and export it to Json. For example for interactions:

var connectionString = ConfigurationManager.ConnectionStrings["analytics"].ConnectionString;
var url = new MongoUrl(connectionString);
var database = new MongoClient(url).GetServer().GetDatabase(url.DatabaseName);

var queryContact = Query.EQ("ContactId", Guid.Parse("4c6220e2-ac9f-43f0-8348-4297eead3d38"));
var interactions = database.GetCollection("Interactions").FindAs<BsonDocument>(queryContact)?.ToList();

var connectionString = ConfigurationManager.ConnectionStrings["analytics"].ConnectionString;

var url = new MongoUrl(connectionString);

var database = new MongoClient(url).GetServer().GetDatabase(url.DatabaseName);

var queryContact = Query.EQ("ContactId", Guid.Parse("4c6220e2-ac9f-43f0-8348-4297eead3d38"));

var interactions = database.GetCollection("Interactions").FindAs<BsonDocument>(queryContact)?.ToList();

Sitecore 8.2 update 7: You can call new method in ContactRepositoryBase to export visits:

ar contactRepository = Sitecore.Configuration.Factory.CreateObject("contactRepository", true) as Sitecore.Analytics.Data.ContactRepositoryBase;
var history = contactRepository.GetInteractionCursor(contactId, visitToLoadPerBatch, maximumSaveDate);

1 2	ar contactRepository = Sitecore.Configuration.Factory.CreateObject("contactRepository", true) as Sitecore.Analytics.Data.ContactRepositoryBase; var history = contactRepository.GetInteractionCursor(contactId, visitToLoadPerBatch, maximumSaveDate);

Sitecore 9: You can use xConnect API: https://doc.sitecore.net/developers/xp/xconnect/xconnect-client-api/contacts/export-all-contact-data.html

Modifying Personal Data (Right to Data Rectification)

You should allow the user to change personal facets in Tracker.Current.Contact.
Alternatively in Sitecore 8 you will need to directly call ContactManager to lock and update contact:

var contactManager = Factory.CreateObject("tracking/contactManager", true) as ContactManager;
var lockAttempt = contactManager.TryLoadContact(inputContact.ContactId, 1);
if (lockAttempt.Status != LockAttemptStatus.Success)
    throw new Exception($"Failed to get lock. Status: {lockAttempt.Status}");

inputContact = lockAttempt.Object;
inputContact.ContactSaveMode = ContactSaveMode.AlwaysSave;

//update facets
var personal = inputContact.GetFacet<IContactPersonalInfo>("Personal");
personal.FirstName = "Tomek";
var emails = inputContact.GetFacet<IContactEmailAddresses>("Emails");
emails.Entries[emails.Preferred].SmtpAddress = "tomek@smartsitecore.com";
emails.Entries[emails.Preferred].BounceCount = 0;

//save to shared session:
contactManager.SaveAndReleaseContact(inputContact);

//or change will be saved to xDB straight away:
contactManager.SaveAndReleaseContactToXdb(inputContact);

var contactManager = Factory.CreateObject("tracking/contactManager", true) as ContactManager;

var lockAttempt = contactManager.TryLoadContact(inputContact.ContactId, 1);

if (lockAttempt.Status != LockAttemptStatus.Success)

throw new Exception($"Failed to get lock. Status: {lockAttempt.Status}");

inputContact = lockAttempt.Object;

inputContact.ContactSaveMode = ContactSaveMode.AlwaysSave;

//update facets

var personal = inputContact.GetFacet<IContactPersonalInfo>("Personal");

personal.FirstName = "Tomek";

var emails = inputContact.GetFacet<IContactEmailAddresses>("Emails");

emails.Entries[emails.Preferred].SmtpAddress = "tomek@smartsitecore.com";

emails.Entries[emails.Preferred].BounceCount = 0;

//save to shared session:

contactManager.SaveAndReleaseContact(inputContact);

//or change will be saved to xDB straight away:

contactManager.SaveAndReleaseContactToXdb(inputContact);

In Sitecore 9 you can use xConnect API to modify contact data: https://doc.sitecore.net/developers/xp/xconnect/xconnect-client-api/contacts/set-contact-facet.html

Right to be Informed

For every version you need to implement custom message box where you inform the users about their rights and your privacy policy.

Sitecore 8 prior 8.2 Update 7: you need to implement custom facet to store audit trail of when the contact acknowledged the organization’s privacy policy.

Sitecore 8.2 Update 7: you can use built-in GdprStatus facet to store privacy policy acknowledged info:

var contact = Sitecore.Analytics.Tracker.Current.Contact;
var gdprStatus = contact.GetFacet<Sitecore.Analytics.Model.Entities.IGdprStatus>("GdprStatus");

1 2	var contact = Sitecore.Analytics.Tracker.Current.Contact; var gdprStatus = contact.GetFacet<Sitecore.Analytics.Model.Entities.IGdprStatus>("GdprStatus");

Sitecore 9: ConsentInformation facet will be the good place to store privacy policy acknowledged info. You can access it with xConnect client:

using (XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
{
    var facet = Sitecore.XConnect.Collection.Model.ConsentInformation.DefaultFacetKey;
    var reference = new ContactReference(contactId);
    var contact = client.Get(reference, new ContactExpandOptions(facet));
    var consentInfo = contact.GetFacet<ConsentInformation>(facet);
}

using (XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())

{

var facet = Sitecore.XConnect.Collection.Model.ConsentInformation.DefaultFacetKey;

var reference = new ContactReference(contactId);

var contact = client.Get(reference, new ContactExpandOptions(facet));

var consentInfo = contact.GetFacet<ConsentInformation>(facet);

}

Check next part of this article where I cover GDPR in WFFM/Forms and EXM.

Oliver Raduner

Hi Tomasz,
this is a great post, thanks a lot for compiling the information & adding all the details from various sources!

I have one question:
you state “Sitecore 9 won’t index personal data in by default.” – can you elaborate further on this point? I understood it, that this relates to Sitecore 9 won’t create the “SC_ANALYTICS_GLOBAL_COOKIE” by default; but which seems not to be the case (and required programming) as described in the StackExchange article here: https://sitecore.stackexchange.com/questions/1454/disable-xdb-tracking-for-single-requests

So what exactly are you referring to, when you say “Sitecore 9 won’t index personal data in by default.” ?

Thanks a lot,
Oliver
- whuu
  
  Hi Oliver, thanks for your feedback.
  
  With this statement I meant that Sitecore 9, by default won’t add personal facets (marked by [PIISensitive] attribute) to the sitecore_xdb search index (which replaced sitecore_analytics index from 8).
  In practise, it means this information won’t be stored in the index in Solr/Azure Search, but only in xDB. This resolves issue with protecting and anonymizing the data in Solr/Azure Search part of your system (cause it’s not there :). The drawback is that you won’t be able to search it using the index.
  
  To change this behaviour you can edit the config:
  https://doc.sitecore.net/developers/xp/xconnect/xconnect-search-indexer/enable-pii-indexing.html
  
  You can check for example “Sitecore.XConnect.Collection.Model, 9.0.json” file in xConnect directory, to find which OOTB facets are marked with PIISensitive attribute.
  
  I don’t think that “SC_ANALYTICS_GLOBAL_COOKIE” behaviour changed in 9. Anyway, it only stores visitor id, which you don’t need to worry about in terms of GDPR, as far as you keep your analytics data anonymous.
  - Oliver Raduner
    
    Thanks for the follow-up & clarifications, Tomasz.
    
    As a side note, you do have to care about the SC_ANALYTICS_GLOBAL_COOKIE tough – because a visitor should be able to specify his/her cookie settings on your site in a way, to be not tracked at all; which means, you must programatically ensure that this cookie is not written at all.