Efficient Product Importer in Sitecore – Part 2

In previous part of the tutorial we created basic product importer which periodically checks products from external Product Information Management (PIM) system and stores them as Sitecore items. The implementation could be further improved by the usage of Sitecore item buckets and search indexes which improves the performance for large number of data.

To better reflects the structure of products from PIM, we use custom Sitecore item bucket, with following structure of product repository:

Active products
- A (all products with name starting with A)
- B (all products with name starting with B)
- …
Inactive products
- A (all products with name starting with A)
- B (all products with name starting with B)
- …

Extend Data Model

We assume that product status (active vs inactive) is a flag imported from PIM system. Let’s start with extending our data model, created in previous part, by adding new checkbox field for the status:

We add corresponding property to our model class:

public class ImportedProduct
{
    public string Code { get; set; }
        
    public string Name { get; set; }
        
    public DateTime Timestamp { get; set; } 

    public bool Active { get; set; }
}

public class ImportedProduct

{

public string Code { get; set; }

public string Name { get; set; }

public DateTime Timestamp { get; set; }

public bool Active { get; set; }

}

Change Product Repository to Item Bucket

Now let’s select our “Product” template’s standard value item and make it “bucketable”, by checking following fields:

Setting “Bucketable” means that all “Product” items can be stored in item bucket. We checked also “Lock child relationship” which we’ll be use later.

By default Sitecore stores items in buckets using structure based on item’s create date. We want to change this, so it will mirror the structure of our products from PIM system. This can be done by creating custom bucketing rule action item under /sitecore/system/Settings/Rules/Definitions/Elements/Bucketing node. We fill “Text” field with content editor’s friendly message and “Type” field, where we point to our custom class in our assembly.

Now we need to create custom RuleAction class. The class must have “Apply” method which defines each level of item bucket structure. For first level we based on “Active” field. Second bucket level is generated using first letter of product name:

namespace Sitecore.Feature.Products.Rules
{
    public class ProductBucketRule<T> : RuleAction<T> where T : BucketingRuleContext
    {
        public const string ActiveFolderName = "Active";

        public const string InactiveFolderName = "Inactive";

        public override void Apply(T ruleContext)
        {
            string[] path = new string[2];
            var item = ruleContext.Database.GetItem(ruleContext.NewItemId);
            if (item != null && item.TemplateID == Templates.Product.ID)
            {
                var isActive = ((CheckboxField)item.Fields[Templates.Product.Fields.Active]).Checked;
                path[0] = isActive ? ActiveFolderName : InactiveFolderName;

                var productName = item.Fields[Templates.Product.Fields.InternalName].Value;
                path[1] = string.IsNullOrEmpty(productName) ? "#" : productName[0].ToString();

                ruleContext.ResolvedPath = string.Join(Buckets.Util.Constants.ContentPathSeperator, path);
            }
        }
    }
}

namespace Sitecore.Feature.Products.Rules

{

public class ProductBucketRule<T> : RuleAction<T> where T : BucketingRuleContext

{

public const string ActiveFolderName = "Active";

public const string InactiveFolderName = "Inactive";

public override void Apply(T ruleContext)

{

string[] path = new string[2];

var item = ruleContext.Database.GetItem(ruleContext.NewItemId);

if (item != null && item.TemplateID == Templates.Product.ID)

{

var isActive = ((CheckboxField)item.Fields[Templates.Product.Fields.Active]).Checked;

path[0] = isActive ? ActiveFolderName : InactiveFolderName;

var productName = item.Fields[Templates.Product.Fields.InternalName].Value;

path[1] = string.IsNullOrEmpty(productName) ? "#" : productName[0].ToString();

ruleContext.ResolvedPath = string.Join(Buckets.Util.Constants.ContentPathSeperator, path);

}

To join the rule action with our “Product” items we need to edit /sitecore/system/Settings/Buckets/Item Bucket Settings node. We select rule “where the new bucketable item is based on the … template”, choose “Product” template and assign the action created in previous step:

Now we can transform “Product Repository” into item bucket. This is done in “Configure” tab by clicking “Bucket” button, while the repository item is selected in the content editor tree. If we already have some products in the repository, we may need to click “Sync” button, so our bucket will have proper structure.

Change Product Importer to Use Buckets

Assuming we will have lot of products, we want also to improve import mechanism. We can do it by replacing existing products search done directly on database, with search indexes. We take the index of bucket’s item, build predicate and perform the search.

private List<Item> SearchBucket(Item bucketRootItem, ID templateID)
{
    if (bucketRootItem == null || !BucketManager.IsBucket(bucketRootItem))
        return null;

    var indexableItem = new SitecoreIndexableItem(bucketRootItem);
    using (var searchContext = ContentSearchManager.GetIndex(indexableItem).CreateSearchContext())
    {
        var predicate = PredicateBuilder.Create<SearchResultItem>(
            i => i.TemplateId == templateID && i.Path.StartsWith(bucketRootItem.Paths.Path));
        var queryable = searchContext.GetQueryable<SearchResultItem>().Where(predicate);
        return queryable.Select(i => i.GetItem()).ToList();
    }
}

private List<Item> SearchBucket(Item bucketRootItem, ID templateID)

{

if (bucketRootItem == null || !BucketManager.IsBucket(bucketRootItem))

return null;

var indexableItem = new SitecoreIndexableItem(bucketRootItem);

using (var searchContext = ContentSearchManager.GetIndex(indexableItem).CreateSearchContext())

{

var predicate = PredicateBuilder.Create<SearchResultItem>(

i => i.TemplateId == templateID && i.Path.StartsWith(bucketRootItem.Paths.Path));

var queryable = searchContext.GetQueryable<SearchResultItem>().Where(predicate);

return queryable.Select(i => i.GetItem()).ToList();

}

We also need a method, which will synchronize the bucket, so every new item will be correctly placed in our item bucket custom structure:

private void SyncBucket(Item item)
{
    if (item != null && BucketManager.IsBucket(item))
    {
        BucketManager.Sync(item);
    }
}

private void SyncBucket(Item item)

{

if (item != null && BucketManager.IsBucket(item))

{

BucketManager.Sync(item);

}

Now let’s use those methods in our importer code and replace searching for existing products with “SearchBucket” method and add “SyncBucket” at the end of the importing. Mind that we do this only if there are any new or modified products.

protected void RunImporter()
{
    var watch = System.Diagnostics.Stopwatch.StartNew();
    var repositoryRoot = Context.Item;
    Log.Info($"Product importer started in {repositoryRoot.Name}", this);
    var changedItems = new List<Item>();

    try
    {
        var existingProducts = SearchBucket(repositoryRoot, Templates.Product.ID);
        var productService = new ProductSerice();
        var importedProducts = productService.GetAll();
            
        IndexCustodian.PauseIndexing();
        var counter = 0;
        foreach (var newProduct in importedProducts)
        {
            var sitecoreProduct = existingProducts.FirstOrDefault(x => x.Fields[Templates.Product.Fields.Code].Value == newProduct.Code);
            if (sitecoreProduct == null || newProduct.Timestamp > ((DateField)sitecoreProduct.Fields[Templates.Product.Fields.Timestamp]).DateTime)
            {
                Map(newProduct, ref sitecoreProduct, repositoryRoot);
                changedItems.Add(sitecoreProduct);
            }
            Log.Info($"Product importer processed {++counter} of {importedProducts.Count()} products", this);
        }
                
    }
    catch (Exception ex)
    {
        Log.Error("Product importer failed", ex, this);
    }
    finally
    {
        if (changedItems.Any())
        {
            SyncBucket(repositoryRoot);
        }        
        IndexCustodian.ResumeIndexing();
        if (changedItems.Any())
        {
            foreach (var index in ContentSearchManager.Indexes.Where(x => x.Name.Contains("master")))
            {
                var changes = changedItems.Select(change => new SitecoreItemUniqueId(change.Uri));
                IndexCustodian.IncrementalUpdate(index, changes);
            }
        }
        watch.Stop();
        Log.Info($"Product importer completed, took {watch.ElapsedMilliseconds}  ms", this);
    }
}

protected void RunImporter()

{

var watch = System.Diagnostics.Stopwatch.StartNew();

var repositoryRoot = Context.Item;

Log.Info($"Product importer started in {repositoryRoot.Name}", this);

var changedItems = new List<Item>();

try

{

var existingProducts = SearchBucket(repositoryRoot, Templates.Product.ID);

var productService = new ProductSerice();

var importedProducts = productService.GetAll();

IndexCustodian.PauseIndexing();

var counter = 0;

foreach (var newProduct in importedProducts)

{

var sitecoreProduct = existingProducts.FirstOrDefault(x => x.Fields[Templates.Product.Fields.Code].Value == newProduct.Code);

if (sitecoreProduct == null || newProduct.Timestamp > ((DateField)sitecoreProduct.Fields[Templates.Product.Fields.Timestamp]).DateTime)

{

Map(newProduct, ref sitecoreProduct, repositoryRoot);

changedItems.Add(sitecoreProduct);

}

Log.Info($"Product importer processed {++counter} of {importedProducts.Count()} products", this);

}

catch (Exception ex)

{

Log.Error("Product importer failed", ex, this);

}

finally

{

if (changedItems.Any())

{

SyncBucket(repositoryRoot);

}

IndexCustodian.ResumeIndexing();

if (changedItems.Any())

{

foreach (var index in ContentSearchManager.Indexes.Where(x => x.Name.Contains("master")))

{

var changes = changedItems.Select(change => new SitecoreItemUniqueId(change.Uri));

IndexCustodian.IncrementalUpdate(index, changes);

}

watch.Stop();

Log.Info($"Product importer completed, took {watch.ElapsedMilliseconds} ms", this);

}

We need also modify Map method a little bit, to include product status flag:

protected void Map(ImportedProduct newProduct, ref Item sitecoreProduct, Item repositoryRoot)
{
    using (new SecurityDisabler())
    {
        if (sitecoreProduct == null)
        {
            var name = ItemUtil.ProposeValidItemName(newProduct.Code);
            sitecoreProduct = repositoryRoot.Add(name, new TemplateID(Templates.Product.ID));
            Log.Info($"--Product importer add new {name}", this);
        }

        try
        {
            sitecoreProduct.Editing.BeginEdit();
            sitecoreProduct[Templates.Product.Fields.Code] = newProduct.Code;
            sitecoreProduct[Templates.Product.Fields.InternalName] = newProduct.Name;
            sitecoreProduct[Templates.Product.Fields.Timestamp] = DateUtil.ToIsoDate(newProduct.Timestamp);
            ((CheckboxField)sitecoreProduct.Fields[Templates.Product.Fields.Active]).Checked = newProduct.Active;
            sitecoreProduct.Appearance.DisplayName = $"{newProduct.Code} - {newProduct.Name}";
            sitecoreProduct.Editing.EndEdit();
        }catch (Exception ex)
        {
            sitecoreProduct.Editing.CancelEdit();
            throw ex;
        }
    }
}

protected void Map(ImportedProduct newProduct, ref Item sitecoreProduct, Item repositoryRoot)

{

using (new SecurityDisabler())

{

if (sitecoreProduct == null)

{

var name = ItemUtil.ProposeValidItemName(newProduct.Code);

sitecoreProduct = repositoryRoot.Add(name, new TemplateID(Templates.Product.ID));

Log.Info($"--Product importer add new {name}", this);

}

try

{

sitecoreProduct.Editing.BeginEdit();

sitecoreProduct[Templates.Product.Fields.Code] = newProduct.Code;

sitecoreProduct[Templates.Product.Fields.InternalName] = newProduct.Name;

sitecoreProduct[Templates.Product.Fields.Timestamp] = DateUtil.ToIsoDate(newProduct.Timestamp);

((CheckboxField)sitecoreProduct.Fields[Templates.Product.Fields.Active]).Checked = newProduct.Active;

sitecoreProduct.Appearance.DisplayName = $"{newProduct.Code} - {newProduct.Name}";

sitecoreProduct.Editing.EndEdit();

}catch (Exception ex)

{

sitecoreProduct.Editing.CancelEdit();

throw ex;

}

After the scheduled task completed we should see our products in new bucket structure:

In previous step, we checked “Lock child relationship” field in “Product” template’s standard values. This allows us to create subitems under the items which are “bucketable”, so for example our products could have variants:

Mind that you don’t need to check “Bucketable”, or “Lock child relationship” fields in subitems.