Quantcast
Channel: sitecore – Bas Lijten
Viewing all 55 articles
Browse latest View live

How to use Sitecore Rocks with Visual Studio 2015 preview

$
0
0

If you want to use the recently released Visual Studio 2015 preview together with Sitecore rocks for sitecore development, you won’t be able to install this plugin from the Visual Studio Extensions gallery: you won’t even find installable plugin while searching for the plugin. This blogpost describes how to get this plugin to work.

In fact, it’s the same trick as was used for the Visual Studio 2013 preview as described in this blogpost on the sitecore site, although there is a small caveat.

First, Download the sitecore rocks visual studio extension from the gallery. Open this package with an archiving tool, in my case, I used 7-zip. Find the file named “extension.vsixmanifest”. This file describes on what Visual studio versions the extension can be installed. Open the file inside the archive and find the following section:

It would be to easy to add the same xml snippet for Visual Studio Version 13.0: There is no spoon ;). As the number “13” is an unlucky number, Microsoft tends to skip all these versions, like they did for the Office and SharePoint products as well.

Just add the xml snippet below, save your manifest and install the extension manually: you will now see that the extension gets installed

<VisualStudio Version="14.0">
        <Edition>Ultimate</Edition>
        <Edition>Premium</Edition>
        <Edition>Pro</Edition>
</VisualStudio>

et voila: your extension can be installed!


How to add support for Federated Authentication and claims to Sitecore using OWIN

$
0
0

Out of the box, Sitecore only offers their own forms-based authentication provider, which requires to add every user to the Sitecore membership database. At Achmea, we had the requirement to facilitate login via ADFS, as we are using our user accounts amongst different systems, web applications and apps. In addition to the absence of this functionality, it’s not possible to work with claims as well.

This blogpost describes how to add and use the Federated Authentication middleware using OWIN in combination with Sitecore and how to access the claims that are provided using the federated login. The solution supports a multi-site scenario, which can handle different identity providers and multiple realms. This opens up possibilities to use external identity providers, for example via ADFS or Windows Azure Active Directory.

The source code for federated login component can be found on github. Please feel free to contact me via twitter/mail/github if there are any questions! A special thanksto Kern Herskind Nightingale of Sitecore: We discussed a lot on the integration patterns for Federation and Sitecore.

Why not to use the ADFS Authenticator Marketplace module?

During my quest on integrating Federated Authentication with Sitecore, I found this module. I had some issues to get it to work in Sitecore 8 build 5, (although I managed to get it to work), but there were some drawbacks why I decided not to use this module:

  • Microsoft is putting their bets on OWIN. As I expect that Sitecore will go that direction in the future, I want to write software that can be easily migrated to future products. (That’s why we don’t create webforms solutions anymore as well)
  • Some extra pipelines were added for User resolving and token requesters.
  • I had to add a few extra HttpModules
  • It didn’t support multiple sites and multiple realms in a secure way

The Sitecore architecture

Basically, the default user management implementation for Sitecore, is a custom Forms Authentication Provider, which makes use of the default ASP.Net Forms Authentication implementation. Under the hood, these users are partially managed in a standard Asp.Net Membership database. For example, information like roles, passwords, and user statuses are all managed in this membership database.

The Sitecore implementation lies around the FormsAuthenticationProvider and FormsAuthenticationHelper, which both exist in the Sitecore.Security.Authentication namespace in the Sitecore.Kernel assembly.

Upon login, there is an Authentication manager which has all login and user management logic abstracted away. The FormsAuthentication Manager, which has been registered in the web.config, is injected in the Authentication Manager as an Authentication Provider. Under the hood, the following actions happen:

  • Username and password are being validated
  • A new Sitecore User object is created
  • The Sitecore user object will be assigned to the HttpContext.Current.User and Thread.CurrentPrincipal

Challenges with OWIN and Sitecore

Adding the OWIN Federated Authentication middleware isn’t too hard (more on that matter later). There are a number of challenges, which can be found in the combination of the federated authentication and Sitecore.

Authentication Challenge

In Sitecore, the AuthenticationManager.Login(username, password) is being used. As stated before, the used Provider is configurable within the web.config. This requires a custom Authentication Provider implementation and a custom Authentication Helper implementation. As this is a serious job that has to be done, I was a bit reluctant to use this. It replaces some out of the box functionality, something I want to prevent as much as possible.

Lifecycle challenge

It can be quite complex to determine when the Claims principal is available, complete and how to map it on the Sitecore user objects.

Claims challenge

Due to the fact that the Thread.CurrentPrincipal and the HttpContext.Current.User object are both being replaced with the Sitecore User object, the provided claims are not available anymore. Solving this in the Sitecore pipeline is not possible, as the claims property is not available on the User class. Replacing the Sitecore User object with another User object would seriously break Sitecore. Writing custom logic to create a custom Sitecore ClaimsUser object would be a serious effort and I don’t know whether or not that would even be possible, due to the internal usage of the Sitecore User object.

Multiple site challenge

The WsFederation Authentication Middleware does not support multi-tenancy: configuring a single instance of authentication middleware with different hostnames and realms that need to be accepted is not possible.

Luckily, all of these challenges can be encountered! As I wrote in some of my previous blogposts, adding OWIN Federation middleware is quite easy. But for the sake of completeness in my first serious Sitecore blogpost, I’ll describe this process later on in this blogpost.

Configuring OWIN in Sitecore

We’ll start with a simple, plain OWIN configuration, which injects the Cookie Authentication module and the WsFederation Authentication Module. The WsFederation Authentication module handles the initial authentication challenge and redirects the user to, in this case, my own STS. When the RST has been returned, the WsFederation Authentication module handles and verifies this token, while the Cookie Authentication module creates a “.AspNet.Cookies” cookie (often referred to the claims cookie), which contains all the user information.

Both middlewares can have several configuration options and events attached: we’ll get into some of those later on.

Federated login for Sitecore – the login flow

When a page is requiring a login, the pipeline could handle the login challenge. I chose to redirect the user to a login page. On this page, there is a controller rendering, whose action is decorated with the [Authorize] Attribute. This attribute does not cause a Sitecore Forms authentication challenge, but a plain ASP.Net authentication challenge, the one that has been configured with OWIN.

The OWIN middleware handles the RST token and sets the claimcookie and sets the current identity on Thread.CurrentPrincipal and HttpContext.Current.User. After handling this token, the Controller logic can be executed and the user will be redirected to the original, requested page.

At this point, there is still no Sitecore user identity. The result: The user gets redirected back to the login page, the authentication challenge will not be triggered, as the claims cookie is available.

Right now we are are missing logic to do an actual “Sitecore user login”. There are bootstrap options to do this:

  • On the SecurityTokenValidated event of the WsFederationAuthentication middleware
  • In the Sitecore pipeline
  • In the login controller action

But before we can do the actual bootstrap, another problem has to be solved.

Supress Form validation when ADFS posts the RST to Sitecore

The RST that is posted to Sitecore by ADFS, needs to be handled. At the moment of writing, there is a PreProcessRequest processor, which handles form posts to Sitecore, the SuppressFormValidation processor.

This processor throws an exception if an unsafe form post was found, but adds some exceptions to Sitecore: unsafe form posts to “/Sitecore/shell” and “/Sitecore/admin” are allowed. Unfortunately, these paths are not configurable, thus I replaced that processor by this implementation:

public override void Process(PreprocessRequestArgs args)
{
    Assert.ArgumentNotNull(args, "args");
    try
    {
        new SuppressFormValidation().Process(args);
    }
    catch (HttpRequestValidationException)
    {
        string rawUrl = args.Context.Request.RawUrl;
        if (!rawUrl.Contains("login"))
        {
            throw;
        }
    }
}

These solution respects the original processor outcome, catches the exception, but adds a path that should accept an unsafe formpost as well. In any other situation: rethrow the same exception, this causes Sitecore to behave exactly the same as it did before.

The bootstrap moments

SecurityTokenValidated

This event seems the most logic place to login the Sitecore user, but it has a major drawback. At the moment that the RST has been validated, a Claimcookie hasn’t been created yet by the Cookie Authentication middleware. The Authentication Ticket, however, is available. When a virtual user is being created, the actual claims from the ticket can be mapped to this user (for example to map some Sitecore user roles), but at the moment that AuthenticationManager.Login() has been called, the HttpContext.Current.User and Thread.CurrentPrincipal properties are being overwritten with Sitecore user. All of your claims, that weren’t mapped to the Sitecore user, are lost.

If there is no need to use claims in your custom code, or the use of the Sitecore roles is sufficient, this is the best place to do the user login, however, if you are in need of using claims, this moment cannot be used as a bootstrap moment.

Sitecore pipeline

As the Sitecore pipeline is highly extensible, this might be a good solution as well. But I wanted to keep the login logic as far away from Sitecore as possible, as it might introduce unwanted complexity, so I didn’t investigate this option further. If anyone has some good reasons to put the logic in a processor, please share these reasons with me!

Controller action

I chose the controller action as bootstrap moment: After being returned from ADFS, the Claim cookie has already been generated. In the controller action logic, the claim cookie is accessible, while the user hasn’t been logged in to Sitecore yet. This is the moment do the Sitecore login and execute some additional actions.

Logging a Sitecore user and mapping claims to Sitecoreroles

As stated before, at the moment that the controller action is being executed, the user can be logged in to Sitecore. I have reused the code that was written by Vasiliy Fomichev. He created a login helper as part of the ADFS Authenticator solution, which is available on github. This loginhelper compares all roleclaims to the Sitecore groups. For every positive result, that Sitecore group is being added to the virtual Sitecore user. After the group assignment has been finished, the virtual user is logged in to Sitecore. The implementation of the loginhelper can be found here.

Mutiple sites configuration

With this OWIN configuration, the multi site requirement hasn’t been fulfilled yet. As the WsFederationAuthentication middleware does not support multi-tenancy, another solution was needed. This solution could be achieved by making use of the pipeline-branching options of the OWIN pipeline.

Pipeline-branching

OWIN supports pipeline branching. app.Map or app.MapWhen can be used to inject some middleware to a specific path or to a specific situation. In my case, I could use the app.MapWhen option:

app.MapWhen(ctx => ctx.Request.Get("host").Equals("multisite.local"), site => { Console.WriteLine("branch 1"); });

I used this technique to register all my sites together with its specific middleware. This can be hardcoded, but it’s better to provide the configuration in a separate configuration file, as it doesn’t require a redeployment when a Sitecore site has been added. An addition of a new option, however, does require an application pool recycle in IIS

The claims challenge

The claims challenge was a harder one to tackle. In a normal Asp.Net webapplication, we can retrieve our claims from the Claimsprincipal that is assigned to the HttpContext.User property. But as Sitecore overwrites this property, we can’t retrieve those claims.

Because of the choice I made for the bootstrap moment, I have access to the .AspNet.Cookies cookie, in which the claims identity is stored. On every request, this cookie is being decrypted and deserialized by the OWIN middleware, to provide the identity. And again, after that moment, Sitecore is overwriting that identity with its Sitecore user.

Retrieving the AuthenticationTicket

The cookie value can easily be retrieved, but it’s encrypted. When adding the CookieAuthenticationOptions to the CookieAuthenticationMiddleware, the TicketDataFormat is being set. This is a property which helps storing the AuthenticationTicket in a cookie. The default implementation even encrypts this data:

if (base.Options.TicketDataFormat == null)
{
    IDataProtector protector = app.CreateDataProtector(new string[]
    {
        typeof(CookieAuthenticationMiddleware).FullName,
        base.Options.AuthenticationType,
        "v1"
    });
    base.Options.TicketDataFormat = new TicketDataFormat(protector);
}

As the dataprotector is used internally by the middleware, it was hard for me to decrypt that data in the cookie. By providing an own dataprotector to the TicketDataFormat, it’s easy to decrypt the cookiedata and return the AuthenticationTickets by decrypting the cookies:

internal class MachineKeyProtector : IDataProtector
{
    private readonly string[] _purpose =
    {
     typeof(CookieAuthenticationMiddleware).FullName,
        CookieAuthenticationDefaults.AuthenticationType,
        "v1"
    };

    public byte[] Protect(byte[] userData)
    {
        return System.Web.Security.MachineKey.Protect(userData, _purpose);
    }

    public byte[] Unprotect(byte[] protectedData)
    {
        return System.Web.Security.MachineKey.Unprotect(protectedData, _purpose);
    }
}

This function can be used to get the AuthenticationTicket from the cookie:

private static AuthenticationTicket GetAuthenticationKeyTicket()
{
AuthenticationTicket ticket = null;

var ctx = HttpContext.Current.Request;
if (ctx.Cookies != null && ctx.Cookies[“.AspNet.Cookies”] != null)
{
var cookie = ctx.Cookies[“.AspNet.Cookies”];
var secureDataFormat = new TicketDataFormat(new MachineKeyProtector());
ticket = secureDataFormat.Unprotect(cookie.Value);
}
return ticket;
}

[/csharp]

And within that Ticket, the ClaimsIdentity can be found:

// Summary:
//     Contains user identity information as well as additional authentication state.
public class AuthenticationTicket
{
//
// Summary:
//     Initializes a new instance of the Microsoft.Owin.Security.AuthenticationTicket
//     class
//
// Parameters:
//   identity:
//
//   properties:
public AuthenticationTicket(ClaimsIdentity identity, AuthenticationProperties properties);

//
// Summary:
//     Gets the authenticated user identity.
public ClaimsIdentity Identity { get; }
//
// Summary:
//     Additional state values for the authentication session.
public AuthenticationProperties Properties { get; }

SessionStore to prevent replay attacks and keep the cookies small

In addition to this TicketDataFormat, I decided to implement the SessionStore property as well. When this property is not null, the OWIN middleware doesn’t store the complete claimsidentity, with all it’s claims, into the cookie, but just a reference to it: this keeps the cookies small. For an explanation see this blogpost on reference mode by Vittorio Bertocci. I also used his code.

A great and safe side effect, is that there is a server side storage which can be used verify if identities are still valid.

In normal FormsAuthentication scenario’s (like Sitecore has), a user can logout. All that happens, is that the cookie gets deleted. When someone intercepts that cookie, for example on a public machine, that person could restart the website, add that cookie and he is logged in again.

When using this SessionStore technique, just the reference to the cookie is being stored and the real AuthenticationTicket can be deleted when a user logs out. When a malicious person is adding the stolen cookie to a new browser session, he won’t be able to steal your identity anymore. I’ll write more on this subject in a future blogpost.

The lifecycle challenge

Last, but not least, I had to cope with the lifecycle challenge. As we are working with two identities, they have to aligned which each other: The Sitecore identity (represented by the .aspxauth cookie) and the OWIN identity (represented by the .AspNet.Cookies cookie and the session store). I put the OWIN identity as leading Identity; when this identity is not valid, available, expired, or whatsoever, then the Sitecore identity should be invalidated as well. In all other cases, the identities should match or not be available at all, to represent a valid request.

After the user resolver processor in the HttpRequestBegin pipeline, I added a new processor, which checks the authentication status. I created the following table for it:

  fedAuth cookie Claims Principal Sitecore User Status Beschrijving Handled by
1 0 0 0 OK No identification options available: anonymous request Pipeline
2 1 0 0 NOK Cookie not valid: delete and redirect to login page Handled by OWIN
3 0 1 0 OK No identification possible. Token is automatically deleted by cleanup job. Anonymous request Pipeline
4 1 1 0 NOK No corresponding Sitecore ID – delete cookie and token. Sitecore handles redirect to login if needed Pipeline and Sitecore
5 0 0 1 NOK Logout, redirect to login page Pipeline
6 1 0 1 NOK Cookie not valid: delete, logout and redirect to loginpage Pipeline
7 0 1 1 NOK No identification possible, same situation as 5: logout Sitecore user and redirect to loginpage Pipeline
8 1 1 1 OK Valid situation. Validate both identities, if they correspond with each other, valid request. Otherwise: logout and redirect to loginpage Pipeline

Basically, it comes down to 3 valid situations, of which 2 reside in valid anonymous request and only the last one leads to a valid authenticated request.

Summary

Adding Federated authentication to Sitecore using OWIN is possible. In the end, the solution wasn’t too complex and makes use of standard Sitecore where possible, without intervening in it’s core logic. Using ASP.Net for authentication on top of Sitecore as a kind of passthrough authentication layer, keeps us safe and it can easily be removed.

Third party libraries used by Sitecore

$
0
0

At Achmea, we made the strategic decision to use Sitecore as platform for all of our websites, we’ve got hundreds of them. To do this at enterprise level, on such a scale, it’s important that we have guidance on infrastructure, development, deployment, security, content, DMS, just to be sure that the stable environment that we deliver, will stay stable. We all know that it’s important to work with the right Sitecore assemblies (correct version) and not to overwrite them. But Sitecore doesn’t only deliver Sitecore assemblies, but 3rd party assemblies as well, for example: Microsoft assemblies, TweetSharp, Facebook API, Google API’s, et cetera. This blogpost tells us what 3rd party components Sitecore ships with, what license is tied to it and what version is delivered. It currently only includes Sitecore 8.0 update 5 and Sitecore 8.1 update 1.

And please try to guess the answer to now: “What is the year of the eldest component that ships with Sitecore?”. You will be surprised 😉

Why is this list important?

All of these components have a history in it’s development lifecycle, may have security issues in older versions and always have a license tied to it. If there’s no license, the components fall under the author’s copyright, which means that you’re not allowed to use that third party component in your software.

Aside from the license issues, make sure that you use the correct version. Sitecore may replace or update the 3rd party components in a future version, which means your software can (and believe me, someday it will) break. There is no guarantee that Sitecore will ship these component in next versions, thus it’s better to be prepared on what to do when things change. Some commercial products are included in the product as well. Questions that I have regarding those products:

  • Do we, as a company, need to have a license as well to use these products, or does the Sitecore license cover this? (Probably it does, but I am not 100% sure)
  • If Sitecore may use the product, for example, Telerik, may we make use of that product in our custom made components as well? For example, when extending content editor functionality? Or do we need to have separate development licenses?

I don’t have answers on those questions yet, but we already asked the question at Sitecore.

The one list

Well, this wall of text finally lead us to the list. I must admit: it was just a desktop research, but I think we are quite complete regarding the versioning. The version info is the “Assembly Info” that I got via ILSpy. I got the most recent versions by looking into github and codeplex repositories, so for some commercial products I really don’t know what the latest version is.

My main concern is that I see some really ancient builds in this list. The eldest component is from 2006! Maybe this list can help Sitecore to upgrade these components to the latest version, for the sake of its ALM 😉

 

Library Sitecore 8.0
update 5
year Sitecore 8.1 update 1 year Current version year License type  
Telerik.Web.UI.Skins 2012.2.607..35 2012 2015.1.401.45 ?? 2015 License? Telerik UI lib
TweetSharp 2.0.0.0 < 2013 2.0.0.0 3.0.0.1 2015 ?? Twitter library
WebGrease 1.6.5135.21930 2014 1.6.5135.21930 1.6 2014 ??  Optimizing javascript
Yahoo.Yui.Compressor 2.1.1.0 2012 2.1.1.0 2.7 2014 BSD-2 Compression library
Componentart 2010 2010 2010 2012 2012 ?? Visualization controls
CsQuery 1.3.3.249  ?? 1.3.3.249  ?? 1.3.5.200 MIT CsQuery is a CSS selector engine and jQuery port for .NET 4 and C#.
DotNetOpenAuth 4.0.0.11165 2011/2012 4.0.0.11165 4.3 2013 Ms-Pl
Ecmascript.net 1.0.1.0 2012 1.0.1.0 1.0.1 2012 MPL 1.1 (Mozilla Public License) EcmaScript.NET is an open-source implementation of EcmaScript based on Rhino (JavaScript for Java) written entirely in C#.
Facebook C# SDK 5.4.1.0 < 2012 5.4.1.0 6.0.10  <2012 Apache License Facebook API
Facebook API 1.0.0.0  ?? 1.0.0.0 ?? ?? Ms-Pl (Microsoft Public License) Facebook API – not sure about the source…
GoogleApis.Authentication.OAuth2 1.0 2011? 1.0 1.9.3 2015 Apache 2.0 Google OAuth2 library. Very ancient library. Stackoverflow post: “class not supported anymore”
Google.Apis 1.0.0.30541 2011? 1.0 1.9.3 2015 Apache 2.0 Google API library
Google.Apis.Plus.v1 1.0.0.0 < 2013 1.0 1.9.2 2015 Apache 2.0 Google plus library
Hammock.Clientprofile 1.0.0.0 < 2011 1.0 1.3.1 2013 MIT Rest Wrapper
Html Agility Pack 1.4.6.0 2012 1.4.6.0 1.4.9 2014 Ms-Pl HTML Parser that builds a read/write DOM.
Iesi.Collections 1.0.1.0 2011 1.0.1.0 4.0.1.400 2013 No license Enhanced collectrions for .net
IT Hit WebDAV Server .Net v2 2.1.1.108 2009 2.1.1.108 V4.0.2416 2015 Found here webdav server engine for net
Lucene.net 3.0.3 2015 3.0.3 3.0.3 2015 Apache 2  Search
Mvp.Xml 2.0.2158.1055 2006 2.0.2158.1055 2.3 2007 BSD License From the time before the dinosaurs even didn’t exist
Netbiscuits.OnPremise 1.1.0.0 ??
Newtonsoft.Json 6.0.5 2014 6.0.8.18111 2014 7.0.1 2015 MIT JSON (de)serializer
OAuthLinkedIn 1.0.0.0 ?? 1.0.0.0 ?? ?? ?? ?? Looks like it’s taken from a github source, can’t find the original source
Protobuf-net 2.0.0.668 2013 2.0.0.668 2013 2.0.0.668 2013 Apache 2 Protocol Buffers library for idiomatic .NET
Telerik RadEditor.net2 7.2.0.0 ?? 7.2.0.0 ?? ?? ?? ??
Stimulsoft Base 2013.1.1600.0 2013 2013.1.1600.0 2013 2015.3 2015 License Reporting technology
Stimulsoft Database 2013.1.1600.0 2013 2013.1.1600.0 2013 2015.3 2015 License Database helper
Stimulsoft Report 2013.1.1600.0 2013 2013.1.1600.0 2013 2015.3 2015 License Reporting technology
Stimulsoft Report Web 2013.1.1600.0 2013 2013.1.1600.0 2013 2015.3 2015 License Reporting technology
Stimulsoft Report Web Design 2013.1.1600.0 2013 2013.1.1600.0 2013 2015.3 2015 License Reporting technology
Telerik.Web.UI 2012.2.607.35 2013 2015.1.401.45 2015 2015 ?? Reporting technology
Ninject 3.2.0.0 2015 3.2.0.0 2015 3.2 2015 Ms-Pl Lightweight dependency injection for .NET
ASP.Net MVC 5.1.0 2014 5.2.3 2015 5.2.3 2015
System.Web.Webpages 3.0 2013 3.0 2013 3.2.3 2015
System.Net.Formatting 4.0 ? 5.2.3 2015 5.2.3 2015

Conclusion

Quite some third party components are delivered with Sitecore, some are up to date, and some are really ancient. This list can help you to decide whether or not to allow a component to be used in your custom code and whether or not the license tied to the component will have impact on your company from a legal perspective.

Multiple Site Manager error: The requested document was not found

$
0
0

Recently our admins faced an error in one of our Sitecore environments that we couldn’t explain. Sitecore tried to serve a site that it shouldn’t serve, which caused a nasty error. Digging through the logs didn’t help us and due to the fact that the logs didn’t tell us anything, our admins had problems pinpointing the problem.

Our Sitecore installation

Our Sitecore installation was a vanilla Sitecore 8.0 release 5 installation with just one site active, This site was created under the default “website” site node in Sitecore configuration and is used for our initial performance baseline. On a set interval, we measure its performance and compare it with the initial performance figures: this way, we can see how our Sitecore environment behaves. Aside from the vanilla Sitecore 8.0 installation, we only had the multiple sites manager module installed, as we are (normally) hosting multiple sites in a single Sitecore instance.

Sitecore trying to serve the wrong website

The first thing I noticed, was that Sitecore was trying to serve the wrong website. As it was a vanilla installation, the hostname should be tied to the “website” node, but watching closely to the error message, I saw that Sitecore was trying to serve the rootnode of the “scheduler” site, a default site that is provisioned by Sitecore:

Multiple site manager caused the problem

As the only customization on the Sitecore instance was the Multiple Sites manager module, I decided to disable this module, which fixed the problem immediately. I haven’t experienced any problems with the Multiple site manager before, this had to be a configuration issue. Knowing that the default “website” node resides above the “scheduler” node, I suspected some issue with ordering in the module.

When comparing it to the actual order of the Multiple Sites manager configuration, I saw a discrepancy:

The website node in the Multiple sites manager configuration didn’t have the same position as the website node in the sitecore.config file.

Only moving the website node above the scheduler node, however, didn’t fix the problem. The configuration is kept in memory and moving nodes around doesn’t trigger a flush, I tried an applicationpool recycle. Et voilá: problem solved.

Why is this a nasty problem?

Although a small hint on the order of site nodes is given in the documentation, it’s not clear why this causes trouble and definitely not everyone knows about the importance of the order. (Quote from the documentation: “The order in which the sites stand under the Sites folder is important for the SiteResolver method. “)

Why is this a nasty bug or problem? When working in a company where multiple people are managing multiple Sitecore environments, everyone has their own way of working and not everyone knows every in and out of every module that has been installed. Some people order information on purpose (to get a clear overview), other people accidentally move an item to another position, not knowing that it can break the environment. Because a flush or application pool recycle is needed to trigger the problem, somebody making the wrong change will not necessarily suspect that the caused the problem.

Conclusion

From my point of view, there are two key lessons to be learned:

  1. Be very careful when making changes at system level: Problems may be revealed at a later point in time.
  2. Modules, especially modules that heavily intercept in the regular Sitecore lifecycle, should be monkey proof. You may exactly know how all the modules work, but don’t expect this from others -> Assumption is the root of all evil 😉

Sitecore ALM and governance part 1: Validate your deployment packages – how to keep your website stable

$
0
0

When writing code for Sitecore, this code should someday be deployed to an existing Sitecore environment. Preferably, this should happen “the first time right”. One of our guidelines to achieve this is: Don’t overwrite Sitecore files. Don’t update existing files of other packages. Don’t upgrade assembly versions. Don’t break your site. It might cause a lot of trouble without knowing where to look. When we were still working with SharePoint, there was an internal mechanism to create and remove deployment packages. Developers had to do their best to overwrite out of the box files, as packaging mechanisms were introduced which explicitly required to select the files that you wanted to deploy. I was (and I am) wondered that Sitecore doesn’t offer this feature (well, not as I expected it), and thus I decided to write a blogpost on what shortcomings we see, how to solve them and how to verify that things will good right.

Source code for the Sitecore Validate Webdeploy packages is provided on github.

Our Sitecore Environments

First of all, a small explanation on our Sitecore environment is needed before we can progress with our observations and solutions.

In our company, we will host hundreds of sites in the near future. To accomplish that, we have multiple Sitecore instances (thus separate Sitecore installations), running multiple versions and each Sitecore instance may host multiple websites, varying from 1 to more than 100 different websites. We try to keep all of our guidelines, deployments and environments as standard as possible: that way, it’s easy to switch sites/deployments between environments, to keep a single deployment strategy, modify that strategy without breaking things and to shift developers over projects as well. It may be possible that multiple different project teams are working simultaneous on different websites that in the end will be hosted on the same Sitecore instance.

We use the same standards for our native Asp.net applications (MVC apps, WebAPI’s), but of course every technology stack may have their own extended rules and guidelines. In our case, we are using the following toolset to build and deploy our software:

  • TDS – Create our content packages (layouts, templates, settings, minimal content)
  • TFS – our code repository
  • Microsoft Release manager – orchestration tool to deploy packages over DTAP
  • Microsoft Web deploy – deploy code packages
  • Sitecore Ship – deploy content

Baselining is key

We decided to create a baseline for every Sitecore version that we host. This baseline consists of several layers:

Layer Description
Sitecore The Sitecore version, with all it’s configuration and all preinstalled Sitecore modules, for example:

·         Sitecore 8.0 update 5

·         SOLR

·         Multiple Sites Manager

Nuget All commonly nuget packages that most developers use when building new solutions, for example:

·         Microsoft.AspNet.MVC

·         Microsoft.Owin

·         Microsoft.AspNet.WebAPI

·         Microsoft.Owin.Security

·         Etc…

Achmea All web deploy packages that we created ourselves which are a pre-requisite for our websites:

·         SEO components

·         Federated Authentication

·         Commonly used practices

·         Logging

·         Etc…

These three layers together are called the Sitecore baseline for Sitecore version x.x and is comparable to, in Sitecore habitat terms, the Foundation. A project is, thus, based on the Foundation and has the freedom to build it’s own project, domain and Framework layer, as long it respects the packages and their versions that are provided with the Baseline.

Projects are not allowed to update the assemblies in this Foundation to a next version, this is centrally managed by the owner of the baseline/platform. In other words: they are not allowed to overwrite any file on the filesystem. All the files that they deploy must follow naming conventions, to be sure that they will not overwrite other projects files as well.

We facilitate developers to get the right version of packages by supplying different nuget feeds: When working on a (new) project, that developer has to connect to a nuget feed for a separate Sitecore baseline version, to include the right assemblies in his project. We have nuget feeds for Sitecore (structured by the idea of Sean Holmesby on the Sitecore community forum), nuget feeds for the allowed third party nuget packages and a nuget feed for our own code.

In addition to these feeds, we also provide a store with all nuget packages, that have to be deployed to the developers Sitecore instance.

Sitecore development – where does it break things?

So far so good. With all these baselines it’s important to be sure that the baseline doesn’t get damaged. With a single team working on a single site, it shouldn’t be too much of a problem when a version gets overwritten, but when working with multiple teams on multiple sites, with multiple code bases and in multiple instances, chances will increase by big numbers that soon (tm, blizzard ;)) “things” will break. And believe me, been there, done that, on the SharePoint platform. We don’t want to get there again anymore. Never.

But how easy is it to break “things”, then? Well: very easy. Assume that you create a new, empty web application project to start some Sitecore development for Sitecore 8.0 update 5. The first thing you do, is adding some Controllers and some views. To use the views “in the Sitecore way”, it’s important to include some Sitecore assemblies, these are copied from the Sitecore directory, a pretty common set are the following assemblies:

  • Sitecore.Kernel
  • Sitecore.Mvc.Analytics
  • Sitecore.Mvc
  • System.Web.Mvc
  • System.Web.WebPages

The next thing that may be needed, are standard Nuget packages, for example, the Microsoft.AspNet.MVC libraries: with a simple instruction, the latest version of these libraries will be installed: install-package Microsoft.AspNet.Mvc

So what did we achieve, with those few actions?

  1. The referenced files are all set to copy local = true. Which means: those files are copied into the webdeploy package on build and will be deployed to Sitecore.
  2. The web.config is also set to copy local = true. This breaks Sitecore, as it has a heavily modified web.config.
  3. Due to the installation of the Microsoft.AspNet.Mvc package, the System.Web.Mvc gets upgraded from version 5.1.1.0 to version 5.2.3.0. Assume that we didn’t copy the web.config: This would have broken Sitecore again, as there should have been created a assembly redirection binding. This can’t be patched into Sitecore, but should have been done with a manual action. The result: an unsupported scenario, as seen in this knowledge base article. We also had official word from Sitecore regarding this subject, although MVC 5.2 works in a lot of scenarios.

So with some minimal actions, we managed to

  1. Overwrite installation files
  2. Break Sitecore
  3. Get into an unsupported state, if we had fixed the web.config. Sitecore 8.0 works with MVC 5.1 and following this Sitecore compatibility Table only 5.1 is supported. We had contact with Sitecore regarding this subject and they confirmed that upgrading to 5.2 was not supported.

How to solve these issues

As all of the above issues are mitigated by our development standards and guidelines, all of the above issues have been mitigated 😉

No, I’m just kidding. Guidelines alone don’t help you. Standards and guidelines help us to think about how software should be written, helps us to maintain and deploy it in a stable way. But without tooling, without help, developers ‘tend’ to take shortcuts or accidentally make mistakes. So we have to help them prevent taking those shortcuts.

Offer a baseline nuget feed

This one I already wrote about. Offer a nuget feed with all packages/assemblies that you allow, this will make it lots easier for the developer to use the correct version

Prevent assemblies from being copied into the deployment packages

There are a few options:

Copy local false

This one is a bit harder. It’s easy to give instructions to the developer to set “copy local” to false. But this may break the intellisense in the Razor views and sometimes breaks the build:

Because of this behavior, this is not a solution for us.

TDS

If you are using TDS, use TDS to exclude those files. See this blogpost from Sean Holmesby on how to do this. This, however, does not work with web deploy packages and is not suitable for us.

Web Deploy and the Web Publish Pipeline (wpp)

If you are using Web deploy packages, like we do, it’s also possible to create a <projectname>.wpp.targets file. This page explains how to use this file. As the file is accepted by msbuild as well, it can be used to create packages without those files

Project templates

Create default project templates with

  • The basic Sitecore assemblies referenced
  • A basic .wpp.targets file with a lot of files already excluded
  • A default web.config that will not be copied

This will help to prevent a lot of commonly made mistakes.

Helping isn’t enough: Seeing is believing!

With above tools in place, it’s still possible to create packages that overwrite system files. Verify every single package that will be installed, before it will be deployed to the different environments. Just compare the contents of the web deploy package to baseline Sitecore installation, to the custom baseline packages (the nuget files and your generic packages) and to the other projects, to prevent conflicts. How did I do that?

  1. I made a baseline directory for a specific Sitecore version and created two different directories in it:
    • Installation
    • Webdeploy packages
  2. I maded an “Instance” folder with a specific “Sitecore instance <name>” folder in it, which is used for all the projects that will be deployed to the same Sitecore instance. In this folder, I have the folder D(evelopment), T(est), A(cceptance) and P(roduction). This leads to the following structure:
    • Instance
      • <Instance name>
        • D
        • T
        • A
        • P
  3. In the Installation folder, I placed a copy of the Sitecore zip file (that can be downloaded from dev.sitecore.net). In addition to that, I created a zip of the Website folders of the Content Management and Content Delivery servers and placed them into this installation folder. These baselines should be equal for all versions.
  4. I placed The web deploy packages that I created to deploy the assemblies from the generic nuget packages in the webdeploy folder, together with my company specific solutions (thus the nuget layer and the achmea layer from the table at the start of this article)
  5. The instance specific web deploy packages are copied into the instance folder. As the Lifecycle of the project specific packages could be low, they are deployed often, and chances are big that the Development version is not the same as the Production version. To be able to see the difference over different environments, these packages should be managed per environment.

The script that I created takes 4 input parameters: baseline directory, instance directory, the package to validate and the environment. It does do a few actions:

  • Validate the webdeploy package against every package in the Sitecore baseline directory. Every duplicate file in the same path, will be reported. If the file is an assembly, the version will be reported as well
  • Validate the webdeploy package against the nuget baseline. Every duplicate file will be reported, If the file is an assembly, the version will be reported as well
  • Validate the webdeploy package against all the instance specific packages, depending on the environment. Every duplicate file will be reported, If the file is an assembly, the version will be reported as well

In the end, all the outcomes are merged and a report is being written to output.txt and the results will be returned; this way, the script can be used in automatic deployment scenario’s. An example output is the following:

In the example above, I am using the application that we just created. What did we learn from the output?

  • I used the Sitecore 8.1 assemblies vs the Sitecore 8.0 baseline (and I didn’t do that on purpose, I really made a mistake ;)) -> break Sitecore
  • I was about to upgrade the System.Web.Mvc version -> Unsupported scenario & break sitecore
  • I was about to overwrite the web.config -> break Sitecore

Well, now we know how easy it is to prevent the world from making mistakes, one last question resides:

How can YOU use that script?

The script is free to used and can be found here. Although there is a readme.md, I’ll give quick instructions:

  1. Copy the scripts to $home\Documents\WindowsPowerShell\Modules\ValidatePackage
  2. Import-Module ValidatePackage
  3. Create the following folder structure:
  4. Place a copy of your (vanilla) Sitecore installation in the right Installation directory, place your generic web deploy packages into the Webdeploy directory and place your project or instance specific packages into the Instances\<instance>\<environment> directory
  5. Import-Module ValidatePackage
  6. Validate-WebDeployPackage -BaselineDirectory “C:\sitecore\validation\baselines\sc 8.0 update 5\” -InstanceSpecificWebDeployPackagesDirectory “c:\Sitecore\validation\Instances\<instance>\” -WebdeployPackageToValidate “C:\sitecore packages\custom package.zip” -Environment “D” -Verbose

Conclusion

Keeping your Sitecore installation clean and healthy can get complex with multiple teams. But with the right mindset, standards, guidelines, tooling and validations, this can easily be achieved. “Never trust other developers on their bright blue eyes” (a nice dutch saying), but verify everything they do. Manual verifications are almost impossible to do (you would make a lot of mistakes yourself, apart from the fact that it is very time consuming), so automation is key here.

Happy validation-time!

I have become a Sitecore MVP!

$
0
0

I am very happy and proud to announce that I have become a Sitecore MVP! For seven years, I have tried to become a SharePoint Server MVP, but last year I thought: screw you SharePoint, Hello Sitecore! Of course, I am just kidding ;). I have worked for years on very cool SharePoint projects, together with Microsoft, Avanade, a bunch of SharePoint MVP’s and some other very high skilled people. I had the chance to meet great people on SharePoint Saturdays, DIWUG, the SharePoint Conferences, Ignite, I shared a lot of the things we learned at Achmea on SharePoint, Security and WCM and learned a lot from the community.

But things change

Microsoft doesn’t invest much anymore in the Web Content Management capabilities of SharePoint, let alone the digital marketing capabilities. The move towards the App model, all the Office 365 API’s and the recent announcement to get rid of WCM in Office 365 was a direct roadblock for a possible migration of our public facing websites to the cloud in the far, far future.

And then came Sitecore…

Last year, we invested a lot, together with Microsoft, Avanade and Sitecore in this platform and in our Sitecore capabilities. We believe that the Sitecore platform can help us further and help us to reach our commercial goals in the near future. Of course, the Sitecore plaform isn’t perfect (no product is), so from the start, we had a lot of good discussions on infrastructure, platform deployment, product features, technical challenges, building Sitecore capabilities, together with our partners, Microsoft, Sitecore and some community members. As I believe it’s important to invest into the community and the fact that I like to share knowledge, I immediately started to participate in the community; visiting the SUGNL, writing blogs, getting in touch with the members, giving feedback to Sitecore on various (important) Sitecore features.

And now I am a MVP!

I was really honored, flabbergasted, thrilled, proud and excited when I got the message that I have become a Sitecore MVP. I really hope that I can meet all the expectations and that I am a worthy MVP. I’ll continue to share my findings and vision on Sitecore from an enterprise perspective, and I really hope that I can inspire you all to do great, and most important, FUN things as well! See you at SUGNL, SUGCON EU, or even maybe some other conferences J

By the way: this doesn’t mean that I leave the SharePoint community; We’re still doing great things with SharePoint WCM and collaboration, I just have less focus on it. If there some cool things to share, I’ll blog about it.

Another look at URL redirects in Sitecore

$
0
0

Redirection of urls, it’s a very common action, it’s important to maintain your SEO-value when URL’s move around and to provide friendly, short URLs. The only thing that you have to do is to create a permanent or temporary redirect, right? There are some solutions which add redirect functionality to Sitecore, for example the great Url Rewrite module by Andy Cohen, which is based on the IIS Url Rewrite 2.0 module by Microsoft. But there are several scenario’s when you can solve several redirects in other parts of the infrastructure, or with other products. This may, for example, be the case in in larger companies, hosting multiple Sitecore instances with multiple sites, where configuring certain types of redirects in different parts of the infrastructure can prevent a lot of other configuration in those same layers, reduce complexity or prevent issues on the permissions to configure redirects.

This blogpost explains why we chose to handle redirects in different parts of our infrastructure, from a technical and a functional perspective.

A small look at the infrastructure and the way it gets managed

If we look into a typical infrastructure to serve websites, this could be visualized as follows. For every website that needs to be accessed via the internet, configuration is needed (not on every layer of the infrastructure, but on several layers). In small organizations, some layers are not present, this may be done by a single team, or even a single person, but in larger organizations, those tasks are usually split amongst several teams, with different workloads, procedures, schedules and priorities. Those tasks vary from configuring DNS, Virtual IP’s, create/order new certificates for connections over SSL, to block incoming requests to specific paths/locations. All these layers are in place to keep the entire infrastructure as safe as possible. Imagine what happens when you are hosting not a single site, but tens, or even hundreds of sites. When those sites should all be accessible over http/https and the sites should be available via a www- and non-www entry, the workload of all the teams grows and with a lot of sites, the traffic will grow as well. Think about what happens to the infrastructure when every request would reach the webservers: every layer would get a lot of traffic. Those are just a few reasons which can contribute to the choice of aborting/redirecting requests as early as possible in the chain.

Note: an example of a path that should be blocked over the internet (thus, the Content Delivery servers), are the /Sitecore/ and /App_config paths. They should be blocked for anonymous users on IIS as well, but defense in depth (multiple layers of security) is always a good practice.

Redirect requirements

When only taking into account the technical aspects of a solution, all the redirects would be handled by the Reverse proxy. This wouldn’t be a suitable solution for the content editors, as they have the need to create new redirects on the fly. In our situation they don’t have access to the reverse proxy, so the only possibility that they would have, was to provide a task to the team which administrates the reverse proxy, with a request to configure those redirects. That would create serious delay in providing new redirects. Imagine what would happen when they make some errors. That would take ages ;).

That’s why we split the redirects into 4 categories:

Redirect all http requests to https

Serving sites over http is not safe for the end user. Google is even using https as a ranking signal; thus boosting sites that use https. This kind of redirect should be handled as early as possible in the infrastructure and it’s a one-time configuration which has become part part of our standard operation procedures when configuring new sites. This configuration takes place on the reverse proxy.

Redirecting early in the chain, prevents a lot of configuration that was needed on other layers otherwise. Think about configuring firewalls, load balancers and webservers, they all should support the http and https traffic, the www- and non-www url’s. At Sitecore level, extra configuration should be needed to serve both sites and it would introduce complex scenario’s when posting data over the internet. When do you choose to use http, when to use https. What would it mean for developers? What kind of security flaws would you introduce?

Note: in the end you want to get rid of non-encrypted traffic. By implementing Http Strict Transport Security, it’s possible to protect against downgrade attacks and it’s easier to protect against cookie hijacking. Read this excellent article by Troy hunt on this technique.

Redirect www to non-www, or vice-versa

All of our sites should be accessible via www- or via the non-www variant, this is a single configuration that should happen once during a lifecycle for a domain. Customers tend to access sites as “domain.com” in the browser, which lacks the “https” protocol and the “www” subdomain. As for the http/https redirect, the same reasons apply: it prevents a lot of configuration. An extra benefit is that it’s easier to configure and maintain canonical urls for your content if you just have one endpoint for one site on the IIS-servers, which is very important for the SEO-value of your site.

Note: the http/https and www/non-www redirects could be included in one redirection rule: this might prevent one additional redirect in the process of redirecting.

Complex redirects

A complex redirect is a redirect which uses regular expressions or other conditions; those kind of redirects are not configured often, but often happen during migrations, after rebuilding, or after restructuring a site. As those regular expressions should be tested thoroughly (preferably over the DTAP), as they can break the sites, we prefer that content managers are not able to configure them.

An example of redirecting urls using regular expressions is the following. An old asp.net site was presenting articles using the following url: http://www.domain.com/article.aspx?id=2&title=some-cool-title. When migrating to Sitecore, a more friendly url would be: https://www.domain.com/article/2/some-cool-title. This can be achieved by using regular expressions. These are the places where these regular expressions can be configured:

  • Solve the complex redirects in the Reverse proxy
  • Solve the complex redirects in Sitecore
  • Solve the complex redirects in IIS using the URL Rewrite 2.0 module

Solve complex redirects in the reverse proxy

When it would be possible to configure those redirects by the build-team, this would be a superb solution, but often, this is not possible. This means:

  • larger throughput time,
  • changes for DTAP would take a lot of time
  • very application specific changes
  • Often the reverse proxies are not configured in such a way, that they can redirect the paths

This is, on our case, not really a viable option.

Solve the complex redirects in Sitecore

Solving complex redirects in Sitecore could be an option. It causes pretty much and complex, tailor-made code, while the same solution could be implemented using out of the box IIS functionality. The URL Rewrite module by Andy could help here, it offers some of the functionality as the IIS URL Rewrite 2.0 Module, but it has, for us, some drawbacks:

  • It’s custom code. IIS and the reverse proxy offer functionality that is, we assume, thoroughly tested. Is it optimized for performance?
  • Although the code has a high quality level, it causes a dependency on an open source project that is not very actively maintained. (if we decide that we are going to use it, we will actively contribute to the project)
  • It doesn’t provide testing of patterns, which the IIS URL Rewrite 2.0 module does offer. (however it does have a neat page to test url’s, to see what rules are being hit).
  • More complexity on denying access to content editors for certain redirect rules.

Although this option could work in other scenario’s, for other customers, we didn’t choose for it (yet).

Solve the complex redirects in IIS using the URL Rewrite 2.0 module

This module is an officially supported module and created by Microsoft. It even fits in a cloud scenario, as the azure web applications can make use of this module as well. However, not all the capabilities of this module are supported: It’s possible to write a custom provider for this module. This means that this module can be pointed to other sources, for example databases, files or even Sitecore, to get it’s configuration. Those providers need to be deployed to the Global Assembly Cache, which isn’t possible on Azure.

An advantage of the redirect module is that it has an complete interface with a lot of options. Complex (very complex) rules can be configured and it’s possible to include conditions, for example to match the HTTP_HOST server to the hostname of the site. Another advcantage of the URL Rewrite 2.0 module is that it will kick in before the Sitecore pipeline, so no additional processing power is needed and no additional database queries are made when a request meets the rule conditions.

The output of testing the pattern results in the following screen:

These results can be used to concatenate the new url (article/{R:1}/{R:2}

Adding configuration changes can be done easily by editing the web.config. In a later blogpost, I will explain on how to do this easily using web.config transormations during deployments; no manual changes will be necessary, they look as follows and can easily be templated:

This means that, the condition match for the hostname, the regular expression and the action type easily can be injected using MRM tokenization or the facilities in octopus-deploy, thus, deployment-time. This will help in the continuous delivery or even continuous deployment process when having one of those mechanisms in place.

As administrators have access to those environments and changes can be automated using the web.config transformations, this seems to be a very viable solution: preventing content editors to make complex changes, while having the needed velocity to make changes.

Simple redirects

The last category is “simple redirects”. An example is redirecting https://www.domain.com/actionable-url to https://www.domain.com/path/to/page, thus an incoming path to any other URL, internal or external. The content editor must be able to set the friendly url to redirect it to an existing page. Sitecore offers the “alias” feature, but this feature doesn’t cut it: it doesn’t redirect but rewrite and it’s only viable when working with single sites and it doesn’t work with external url’s. Aliases with the same name for different sites cannot be configured. This is the part where some tailor-made code will do it’s job. There are two options in this one:

  • Writing an IIS URL Rewrite 2.0 provider which interacts with Sitecore
  • Write a custom pipeline for Sitecore

Writing an IIS URL Rewrite 2.0 provider which interacts with Sitecore

As stated before, this provider does not work in Azure, as it has to be deployed in the Global Assembly Cache. A great advantage could be that Sitecore can act as a provider. Content editors would be able to configure simple redirects, which are handled by the provider. My gut feeling is that Microsoft doesn’t actively put any effort in this provider-pattern anymore, as it’s hard to find the SDK’s for these providers.

Writing a custom pipeline for Sitecore

This solution can be very viable. Content editors would only have to configure the from-path and the to-url, (exactly the same as in the IIS URL Rewrite 2.0 provider), but the redirects would be handled by a custom pipeline extension. A simple caching mechanism could be used here, to improve performance.

This module should handle the real Sitecore scenario’s: multi-language, configurable for multi-site solutions, easy access for editors, and reporting on redirects using the analytics database.

When a plan comes together

When those 4 kind of redirects are being combined, the following scenario’s are handled as followed:

http://domain.com/action-path -> https://www.domain.com/the/real/path/for/sitecore

  1. Reverse proxy redirects http://domain.com/action-path to https://www.domain.com/action-path using a 301-redirect
  2. IIS Url Rewrite – no regular expression has been configured, so this url doesn’t match, no redirect takes place
  3. The path /action-path is being redirected to /the/real/path/for/sitecore , so https://www.domain.com/action-path is being redirected to https://www.domain.com/the/real/path/for/sitecore using a 301 redirect.

http://domain.com/article.aspx?id=2&title=some-cool-title -> https://www.domain.com/articles/2/some-cool-title

  1. Reverse proxy redirects http://domain.com/article.aspx?id=2&title=some-cool-title to https://www.domain.com/article.aspx?id=2&title=some-cool-title using a 301-redirect
  2. The IIS URL-rewrite module kicks in: the article.aspx?id=2&title=some-cool-title matches the regular expression “^article.aspx\?id=([0-9]+)&amp;title=([_0-9a-z-]+)” and 301-redirects this page to /article/2/some-cool-title
  3. This page is available in Sitecore and is being served. This page could have been redirected on it’s turn by a content editor to another location.

As seen in the two scenario’s above, most of the time, the amount of redirects is limited to two. Theoretically, there could be a 3rd redirect, but this shouldn’t happen too often. Another advantage of this mechanism, is that paths, which were coming from the different https/http and www/non-www configurations, can be handled in a single configuration item, instead of 4 different mappings.

Using these 4 kind of redirects, all page requests that will reach the IIS-webservers, are having the https://www.domain.com structure, which greatly reduces complexity. All redirects that are more application specific, can be handled by IIS. Complex and simple redirect only have to be configured for a single URL, instead of 4:

As seen in the picture above, a single domain on https has to be configured throughout the food-chain, while the application specific redirects can be handled by IIS

Conclusion

When taking another approach on redirects, and by configuring redirects on other parts in the infrastructure, this can greatly reduce complexity, traffic throughput and it might increase security. This can reduce the need for custom code in your Sitecore modules and greatly reduce the complexity of this code. Sitecore is not the centre of the Universe. (well, not always ;))

Sitecore Security #1: How to replace the password hashing algorithm

$
0
0

Let’s face it: It’s a business nowadays to hack sites, retrieve personal information and sell them on the black markets, think of usernames, passwords, credit card details and-so-on. Often, this data is stolen using SQL injection attacks, which may be possible to your Sitecore site as well, thus, it’s better to be safe than sorry. As Sitecore ships with an old hashing algorithm to handle Sitecore users login, it’s time to replace the hashing algorithm as well. When having a fresh installation, this isn’t much of an issue, but for existing installations, you will face the challenge on upgrading your existing users, because the password hashing algorithm will be changed. This blogpost will show how to upgrade the hashing algorithm, describe those challenges, and tell you how to increase your Sitecore security.

Find the sources on https://github.com/BasLijten/SitecoreDefaultMembershipProvider for use on your own Sitecore environment!

The default Membership configuration

The default SQL Membership provider is configured out of the box as follows:

Sitecore is configured to use the provider “Sitecore”, which redirects to the “real provider” called “sql”, which is, in fact, the old membership provider from 2008.

When we look closely at the configuration, we notice a few things:

  • Hashing algorithm SHA1 is used (which was standard back in those days)
  • The default password policy is pretty weak

The hashing algorithm

Hashing is a form of obfuscating a value that is irreversible, which makes it impossible to get the original value of this hash without brute forcing it, which basically means trying to hash every possible value until you get a value which is equal to the original hash. The longer the hashing function takes to execute, the safer it is to save passwords with this hash function.

Changing the hashing algorithm

To use an alternative kind of all that is needed, is to replace the “hashAlgorithmType” to another, supported hashing algorithm. From a security perspective, there are few recommended algorithms, like for example “bcrypt” and “PBKDF2”, but they are not part of the Microsoft .Net framework. The SQLMembership provider that is being in use by Sitecore, only supports SHA1 and SHA256. Sitecore uses the SHA1 version, probably because of backwards compatibility issues.

Option 1: Changing the membership providers

First,  the possibility to change the Sqlmembership provider into a newer one was explored; the DefaultMembershipProvider or the SimpleMembershipprovider. Although it’s not too much work to change this membership provider, this is not recommended

  • The new membershipproviders require entity framework to run. This requires a change to the connectionstrings.config and the deployment of this framework, which might conflict which your current assemblies
  • Those providers use new tables in the core database and a migration of the old tables to the new tables. This may cause issues when upgrading to a new Sitecore version
  • The DefaultMembershipProvider does not support bcrypt or PKBDF2
  • The SimpleMembershipProvider uses PKBDF2 (with 1000 iterations), but it may be possible that this one does not fully work with Sitecore (didn’t test this out)

Option 2: Adding new hashing algorithms to Sitecore

Another option was to deploy the new hashing algorithms to Sitecore. It turned out that this was the most secure and easy way to increase the Security level.

The company “Zetetic” created the .net versions of these algorithms which can be used within your application. Before this algorithm can be used, these have to be registered first. I chose to do this in the initialization pipeline of Sitecore with just a single line of code:

This line of code will make the hashing algorithm available for use in the membership configuration section of Sitecore. Change the “SHA1” hashAlgorithmType to “pbkdf2_local” and you’re good to go. Except for one small issue: due to the change of the algorithm, no one will be able to login anymore.

Resetting the admin password

Step one would be to reset the administrator password. I create a small console app to do this, which can be found on github; it contains just a few lines of code:

If your installation is brand new, you are good to go, otherwise, the next step would be to reset the passwords for all other users as well. As the hashing values are irreversible, the only option would be to reset the passwords of all of your users, so that they would have choose a new one. Another option is to create a custom Membership provider to gradually change the passwords when users logon. Kam Figy wrote a blogpost on this subject as well and has created a prototype membershipprovider.

Increasing the password policy

Another part is to increase the password policy, as the out of the box is pretty weak. This page describes the parameters that can be configured. I would recommend to set at least the following parameters:

  • maxInvalidPasswordAttempts:             5
  • passwordAttemptWindow:             10
  • minRequiredPasswordLength             12
  • minRequiredNonAlphanumericCharacters 2

 

A better approach would be to use the following regular expression (passwordStrengthRegularExpression) which forces the passwords to have at least an upper case, lower case, one number and a special sign in your password:

/^(?=.*\d)(?=.*[a-z])(?=.*[A-Z])(?=.[\W]).{8,}$/

Summary

Increasing the password security is not hard and I really suggest that you will do this right NOW! Although it might bring some inconvenience for your customers, you will really increase your security policy by a lot. If you don’t believe me? Watch this short youtube movie:


Realtime personalization monitoring with Sitecore and google analytics

$
0
0

Some of our bigger sites, which don’t run on Sitecore yet, use google analytics to realtime monitor events that happen on a website, think about forms that are submitted and personalizations that are shown to a specific user. Most of the time, external (javascript) tooling is used to inject those personalizations and an event needs to be implemented which will be send to google analytics to register that event. In Sitecore, we can implement those google analytics events by including a javascript in our razor views, but, how can we tell whether or not the component that was shown was part of a personalization flow? Was a custom datasource selected, was the completed component rendered as a personalization? This blogpost series learns you on how to determine what kind of personalizations where exposed to a user and how to tell external systems about those events. It turned out that a (beautiful) pattern can be used that Sitecore itself already introduced themselves a while ago.

All sourcecode can be found here on github

The use case – sending real-time events to google

A user visits a site for the first and sees a few blocks of information on the site. As it’s the users first time, no profile has been built yet and all the default renderings are shown to the user. One of these blocks is a call out for a travel insurance. The user appears to be the owner of a small company and navigates through the “business” area of the site. Due to his behavior, the user gets the “Business user” profile applied. At the moment that he returns to the front page, the “travel insurance” callout gets replaced by a “Business travel insurance” call out, nothing special so far. Our marketers expect that for every personalization, a custom event will be send to google analytics with the name of that personalization, that tells google analytics (or any other analytics tool) that a personalization event has been taken place. And this is exactly the part where things become harder.

The Challenge

What’s so hard then? The requirement to only send an event when a personalization has taken place is the hard part. A developer could include for every view that he creates the JavaScript to send an event to google analytics:

ga(‘send’, ‘event’, ‘Personalization’, ‘Name of personalization’);

But how could you possibly know if a component was injected as a conditional rendering? Or that only the datasource was personalized? What was the name of the Rule that was applied? What if the developer forgot to inject the JavaScript? Or what if the rendering wasn’t a personalized rendering? The short answer is: This is not possible (although, not in an easy, scalable way) and chances are big that if it was an easy solution, every razor view would contain logic to a) determine if the JavaScript should be injected and b) inject the JavaScript if this was the case.

Analysis of the challenge

As stated before, there are two major issues:

  • When and how do we know what personalizations have been applied for each component?
  • How can we automatically notify google analytics with the personalization events?

Aside from those two questions, there were a few additional requirements:

  • The solution has to be easily maintainable by a developer
  • The solution has to work in a multi-site setup
  • The solution may not mess up the HTML – we have to come as close to the html as the designers deliver to us.
  • No overhead for our content-editors or marketers – a new personalization should cause any effort

What personalization is applied?

The most important part of the solution, is to find out what personalization are applied and there are a few ways to find out how this is done.

The rendering pipeline

The first solution that I came up, was to intercept the rendering pipeline. Sitecore runs a pipeline for each rendering, which executes a few different steps:

The first part of this getRenderer pipeline is clear: this the part where the customizations are determined. Under the hood, it runs the mvc.customizeRenderingPipeline:

The Personalize processor is where the magic happens: the rules are determined and executed, to find out what rendering should be rendered. A great place to find out what rule was applied. But how could we insert JavaScript to the response, to send the events to google analytics?

As the personalize processor only applies possible the actions that are tied to the rules, for example setting a different datasource or setting a completely different rendering, it does not set any information on what rule was applied. After the Personalize processor, the customization pipeline directly falls back to the GetRenderer Pipeline and tries to render the rendering that was set by the personalization step.

This means, that we cannot directly find out what personalization was applied from the rendering step. Introducing a new processor in this stage, run all the personalization rules again, selecting the name of the applied rule and inject some JavaScript before or after the rendering would get complex and it would mess up the clean HTML structures that we wanted to use.

It became clear that this wasn’t the most beautiful solution to solve the challenge.

Using Analytics

Browsing around in the interactions section of the Xdb set me into the right direction. It appeared that all personalizations are stored into the Xdb!

For every interaction is stored what page was visited and what rules were exposed to the visitor. It appears that the RuleId is the ID of the that was applied on the rendering. Too bad it doesn’t have any information on the rulename or rendering stored as well. This information is also available via the API (late in the pipeline lifecycle, that is! This means that this information is not yet available during the getRenderer pipeline which sounds, logical to me. It sounds a bit like the Chicken Egg paradox), via the following code:

The personalization property contains information on both RuleSet and RuleId. But the meaningful rulename (the information that our marketers are interested in) is not accessible directly via the API.

Getting the meaningful rule names

As you probably saw in the picture above, the RuleId is stored as a Guid. As “Sitecore is build on Sitecore”, the expectation was that the personalizations were stored somewhere in the Sitecore databases, but it turned out that was not the case. A deep dive into the personalization pipeline learned me that those rulesets are stored on the item itself. Below is a snippet with an example of it.

Parsing the xml directly was a consideration, but, this format can change over time and it’s unsure whether or not this xml contains all the information? And what about global rules? Will Sitecore introduce them again? A gut feeling that this wasn’t the most stable solution for the near future. All that was left to do, was to iterate through all the renderings, find the rules on these renderings and correlate them to the correct name of the personalization rule. I also introduced some logic to ignore the default renderings, as they aren’t really real personalizations.

The information on what RuleId was exposed to the visitor is not yet available during the insertRenderings or GetRenderer pipeline, so another moment in time has to be found to use this information.

Create a new processor in the mvc.requestEnd pipeline

This kind of modification belongs in the mvc.requestEnd pipeline. At this moment, all renderings have been customized, inserted, all personalization data is available and it’s the point to before the HTML is being returned to the browser. The perfect place to inject some HTML. But how could the HTML be injected into the Response of the current context?

The answer lies in the HttpContext.Response.Filter. The output of the response is piped through this Filter (if any available) which allows the output to be modified. Sitecore itself uses this pattern to inject, for example, the Experience Explorer Control and the DeviceSimulation controls. A small look at the code put me into the right direction: Sitecore already created a neat PageExtenderResponseFilter class which allows to manipulate the page output. I copied that one and modified it to run a new pipeline, the RenderPageExtender pipeline, in which I put a processor to render the personalization processor (in this case, the google analytics variant). In this processor, the logic is stored to find the personalization names and to render this to the HTML that will be returned to the client. This mechanism can be used to inject other kinds of HTML as well!

Making it usable for developers

The most parts of the puzzle are on its place now, the last part is to make this available in a multi-site solution and to create a solution that causes the least amount of work for developers to inject HTML into the page. For the multi-site solution, I chose to make the pipeline configurable by Site parameters: if the site name of the current context is not specified as a filter, the processor won’t run at all.

The solution to inject HTML was a bit more complex. The first solution was to create a processor per site and insert the JavaScript via .Net code, but that one is quite error prone, with all the escaping characters. I decided that I wanted to make use of razor views and dynamically load them and render the output to the response filter. This solution has a few advantages: it’s easier to make reuse of the processor and customizations can be added very fast, without to much of a hasle. And this solution was also already available in Sitecore: Ever wondered why the ExperienceExplorerView.cshtml was in the /Views/Shared folder? Exactly for this reason, as it’s dynamically inserted as well.

The Experience Explorer view has some information on what rule was applied for the 
current user as well. This solution executes all rules for a 2nd time after the 
page was rendered, I choose not to do this for performance reasons.

The code below shows how a Controller dynamically can be created, inserted and rendered. The partialName at line 1 comes from the processor parameters as well: this makes it possible to specify a separate razor view per site.

This is the razor view, that easily can be maintained and created by any developer:

Summary

Sending real-time personalization events is possible with Sitecore, although it wasn’t easy to find out how this could be done. No information of personalization is stored on the rendered objects itself and injecting HTML into the Response wasn’t as easy as expected as well. It turned out that there is quite a nice pattern that Sitecore uses internally to inject renderings after the rendering pipeline, which can be for rendering normal pages as well.

 

 

 

 

 

Improving the Sitecore Logging and diagnostics experience™ part 1: expose more information using a new Logger

$
0
0

Lately, I have been working on improving he Sitecore Logging Experience™. Sitecore uses the log4net framework to handle all kinds of logs, but, with the standard configuration and implementation, we’re not making the most out of it.  Where Alex Shyba wrote some excellent posts on writing your logs to SQL to make the logs easier accessible, I am going to take the logging capabilities to the next level! In this blogpost I will describe why the out of the box Sitecore logging implementation should be improved, how to do this and eventually I’ll show how to modify the appenders to show some extra trace information. This is all a step-up to my next blogpost, I will explain how all the Sitecore logs can be send to application insights on azure to get even better insights in your application usage!

The out of the box logging experience with Log4Net

The Sitecore logging mechanism is based on an old version of Apache’s Log4Net, which is a very flexible and pluggable logging provider. The log4net library has been included in Sitecore as in, delivered in an own assembly, called Sitecore.Logging. While there is a default log4net logging façade (in the Sitecore.Logging assembly, namespace log4net.spi.LogImpl), Sitecore provided a specific Sitecore logging façade as well. It’s available via the Sitecore.Kernel assembly, in the Sitecore.Diagnostics namespace. When referencing this library, a log can be written by simply entering the line “Log.Info(“message”, this)”. This rule logs the “message” to the specified logger, if available. The specified logger is being specified by “this”, and represents the type of the current class. This page has an explanation on how the log4net can be configured.

Because the log4net implementation is a) outdated and b) being hosted in a Sitecore assembly, it’s not possible to easily use 3rd party solutions for log4net with Sitecore. The 3rd party solutions generally use the newer implementations of the LogEventInformation class (which has been altered over time) and they can’t find the log4net assembly, because it isn’t there.

Log4Net Loggers

A logger is a piece of configuration in which the name of the logger is specified and contains some configuration options:

  • Name: name of the logger. When logging, by specifying the name of the logger or specifiying the type (remember, this?), Log4net checks out if this logger is available. (more info on the documentation page ;))
  • Level- What is the minimum Logging Level that should be handled. For example, if the Level INFO has been specified, all messages of level DEBUG are ignored
  • Appender: target where the logs should be stored. This can be anything, appenders are separately configured

Appenders

In the /Sitecore/log4net/ configuration section, these “appenders” can be found, this is a target to which log4net can write its information. The Sitecore out of the box appender is the SitecoreLogFileAppender, which writes its logs to the /data/logs/log-{date}.txt files. Other examples of appenders are:

  • ADONetAppender
  • ASPNetTraceAppender
  • EventlogAppender

Custom Appenders can be created as well , but that’s not part of this blogpost. More on that in part II – Sending logs to Microsoft Application Insights on Azure. As part of the configuration, Log4net conversion patterns can be used to determine which data will be logged, the out of the box setting is as follows,

“%4t %d{ABSOLUTE} %-5p %m%n”

which results in writing the following logs:

According to the documentation these conversion pattern names mean “ThreadID”, “date”, “Level” and “Message”. This message can be useful for hunting and solving bugs.. As shown in the picture above, I included information about the controller, the classes that were instantiated and the methods that have been used, but this can cause a lot of rework, especially when refactoring your application. It can even be confusing when the logging information is not being updated when the code is being refactored, or when the logging information has some typo’s. According to the documentation, this shouldn’t be needed, as there are a lot more conversion patterns can be used for diagnostics. A lot of people use classnames and methods in their logs, but this information is already part of the log information: “%M and %C” can be used to include this information in the logging message. When changing the config to use this pattern, the following logging will be written:

For every logentry, the type “Sitecore.Diagnostics.Log” and Method “Info” are logged, while I expected that my custom controller and classes would appear in these logs. Something is wrong, but what? And more important: how can this be fixed?

How can this be fixed?

The fix to include the right information can be fairly easy. The Sitecore logging implementation for log4net is a façade that is encapsulating the default log4net façade. Basically, it comes down to the following pattern:

The above example is very, very simplified, but it gets to the point: Our custom class is calling the Sitecore logging implementation, which, on it’s turn, is calling the Log4Net façade. This Log4Net façade executes the function “Assembly.GetCallingAssembly()” method, which returns “Sitecore.Diagnostics.Log”.

The trick is to write an implementation which looks like the Log4Net façade: a Sitecore.Diagnostics.Log which calls the “Assembly.GetCallingAssembly()” and passes that information into the Log method of the ILogger:

When using this implementation, together with the proposed conversion patterns, the logs look like as follows:

Now the correct calling assembly and the correct calling method are being logged, which enables the developers to use shorter, meaningful messages in their code and the logs give much more insights on the the code that was executed. The good thing is: Sitecore can fix this and this fix can be backwards compatible with the current logging experience, while the exposed logging can greatly be improved! It would even be possible to upgrade to the current version of log4net!

On my github page I created a new SitecoreDiagnostics repository with a new Logging implementation. I didn’t test this on production, but it’s an example on how easy this stuff can be fixed. This repository includes a log4net implementation which is able to write the correct information, as well the application insights appender to submit those logs to the Azure Service.

Summary

The default Sitecore implementation of log4net is not optimal, the implementation is outdated and doesn’t support 3rd party add-ons. It’s not hard to improve the current implementation and building a custom logger façade to use the Sitecore log4net implementation and to be able to write into the standard sitecore logs, is quite easy and opens up possibilities. One of these possibilities is append the information to application insights and take the insights to the next level, as every logged item, your custom logs, as well as the Sitecore logs, would can be correlated and be traced See below for a sneak preview. In this overview, we see custom logging, the calling class, method, line number and all other events that have been logged for this request. Even some of the stored procedures for this request are visible in this image, how cool is that?

Improving the Sitecore logging and diagnostics Experience™ part 2: logging to Application Insights

$
0
0

In my previous blogpost I wrote on improving the Sitecore logs, which was a prerequisite for this blogpost, to send all that logging information to Application Insights. This blogpost will explain the steps on how to do this. Application Insights is a tool, hosted on Azure, which helps to get a 360-view on your application. It tracks application health, adoption metrics and crash data. With several interactive tools it’s possible to filter out, segment data and drill down into event instances for details. With a few clicks, it’s possible to view the whole logged call-stack of your application. In this blogpost, I will explain how to send your logs to Application Insights. The great thing is: The is not limited to your custom logs, but the full stack of logs, thus custom and Sitecore logs, will show up in this tool. This platform is not limited to Microsoft, there are a lot of SDK’s available for other technologies.

All source code can be found on my SitecoreDiagnostics repository on github.

How can Application Insights help?

When sending all the data to Application Insights, the service can help to get insights in page requests, views, exceptions and other metrics. What is happening, which dependencies are there, what kind of SQL statements are executed, et cetera. Not only your custom logging is shown, but the Sitecore dependencies as well. In the paragraphs below, I will explain one usecase extensively and will show others by just displaying some cool graphs. In the end, all data can be used in Microsoft Power BI as well! I will explain one use case on narrowing down to some server side events. Other overviews that can be created are overviews of all exceptions, failed requests, page views, user sessions, active users. When set up correctly, the performance counters of a server can be send to Application Insights as well.

Manual tracing

Open up Application Insights, the default view is looking as follows:

Selecting each of the graphs will result in a different follow up-screen. I selected the first graph, as I was interested in the server response times. I added the diagnostic timeline myself, it shows a lot of exceptions: this is due to mongoDB that crashed at my local machine, which is generating a lot of exceptions.

The next screen that opens up after selecting the server response time, is the Server responses page. This is a preconfigured page with diagnostic information on Server response times, the dependency durations and the amount of server requests completed. It’s possible to easy filter a smaller timezone on these graphs, by selecting a timerange and press on the magnifier glass:

After pressing the buttom, all events are filtered. Every row can be selected to drill down to even more information. In my case, I want to know why the Sitecore/Index request is taking a lot of time to execute. The average call takes 2.3 seconds, with a standard deviation of 4.64 seconds!

Some requests take 4 seconds or more to load, while others are being executed in 100-200 milliseconds.  Let’s narrow down to the request that takes 5.8 seconds. A lot of information is being shown. I removed a lot of remote dependencies (all SQL calls), those are automatically measured, and if there is a stored procedure, this one is logged as well. WebAPI dependencies and WCF calls are measured automatically as well!

This overview gives the ability to drill down even further. When selecting “All available telemetry”, all related events, traces, dependencies and exceptions are being shown. And when I say all, I really mean “all”. All related Sitecore generated logs are appearing as well in this overview:

This telemetry can be narrowed down even further, to get all meta information that is available for this log. Remember my previous blogpost, where I explained on how to log the calling class and method to the Sitecore logs? The reason is below, this information (and even more) appears in the meta overview of this call! Classnames, methods, identities, linenumbers, even the sourcecode files is showing up in this overview! And with just a bit of extra work, calling Site, environment or other data can be exposed. In the case below, I selected a custom log, that was created by my own code.

When selecting a Sitecore generated log event, other information is showing up:

Notice the Classname and methodname? They are always defaulting to the Sitecore.Diagnostics.Log. If another telemetry event is selected, for example a telemetry dependency, other information is showing up. In this case, even the SQL command (in this case the stored procedure on the core database).

Application Insights search in visual studio

From the visual studio interface, it’s possible to search through the logging as well:

Using and Configuring Application Insights

In a normal scenario, application insights can be configured when creating a new web application, it’s part of the new project modal dialog. This will create a new project, add a new Application Insights resource to azure, configure the instrumentationkey, will configure the web.config and add some javascript to the _Layout.cshtml to make sure that every pageview will be tracked. These actions will be correlated to the events that are executed serverside. The Trace API can be used directly to write to Application Insights, but the logging framework of choice can be used as well.

Configuring Application Insights for Sitecore

In our case, the Sitecore case, configuring Application Insights is a bit harder. You can follow above practice and just throw away anything that you don’t need, or start off with an empty web application, without the Application Insights instrumentation. In this case, you need to do the following:

  • Create a new application insights resource
  • Add the following nuget package to your solution:
    • ApplicationInsights.Web

This will add some javascript files, make sure that your layout page references these javascriptfiles and make sure to execute the javascript that is provided by application insights on every single page. The custom javascript that needs to be run on your page can be found here:

The web.config has to be modified as well, as some http modules have to be added. I chose to use Xml document transformations for it, to be able to do this in a repetitive way on different environments:

The InstrumentationKey is a key to identify the application and is generated by Application Insights. In normal scenarios, this key can be found in the applicationinsights.config, but I decided to make this one configurable in the Sitecore settings. The patchfile has to be altered for this one.

The last part is adding the Application Insights appender for log4net nuget package. And that’s the part where things go wrong: this package doesn’t work, because of two reasons:

  1. The Sitecore log4net implementation is an outdated implementation. The logEventInformation class, for example, isn’t compatible anymore with the AI-Appender.
  2. The appender looks for the log4net assembly. As Sitecore decided to host it’s own version, this one cannot be found. Adding this log4net assembly will not work, as it won’t pick up configuration and it wouldn’t be possible to send Sitecore data to Application Insights.

That’s why I added a custom ApplicationInsightsAppender in the project on github, which can handle the outdated implementation. A downside is, that I had to use some reflection to be able to send Exceptions to the service.

Last, but not least, the custom appender has to be added to the log4net configuration. I decided to only add the appender to the root-logger, all the others will not log to Application Insights. Changes can be made in the patch-file.

I created a Sitecore specific variant and placed it in this repository. Compile it and deploy the .Web project to the environment and you would be good to go (apart from the web.config changes). The test project contains a layout page and a controller rendering with both the javascript and some logging on it, which can be used to generate some test-data.

Summary

Application Insights can be a really good addition to your toolset to analyze your Sitecore webapplication. Not only gives it insights in your own application, but it’s possible to trace down to the Sitecore logs as well!

Presenting at Sitecore Symposium 2016 – Keeping hackers out

$
0
0

At the upcoming Sitecore Symposium, starting on September 15th, I’ll have the pleasure of presenting a session about Sitecore Security –  Keeping Hackers out: Secure Application Development for Sitecore.

Fix vulnerabilities before the bad guys find them

Your Sitecore installation might be hardenend, but that doesn’t mean jack to a hacker. This session will explain the basics on secure application development for Sitecore, to make your websites safe. Using lots of demos, I will show you common security vulnerabilities, their root causes and how to fix them. You’ll learn why you should invest in security, how easy your site may be to hack, how to fix vulnerabilities, and how to write code that’s secure. You’ll walk with source code goodies, free to use.

This is an altered and improved session of my SUGCON session in Copenhagen this year, with some great reviews:

Come and join my session

Come over and join me on my session on web application Security. Astonishing demos and maybe you’ll see a glimpse of Bobby Hack.

Sitecore Security #2: Secure connections and how to force the browser to use the secure connection

$
0
0

Secure connections? Why would I even bother? It’s expensive, slow, complex and I’ve got a firewall anyway? On the SUGCON in Copenhagen I showed off how easy it is to intercept, modify and redirect unencrypted traffic and what can be done against this threat. This blogpost is all about why you should serve your website always and fully over HTTPS and how the browser can be forced to use this HTTPS connection. This blogpost will not show off how to configure HTTPS and will not tell about all the benefits of https. The technique to achieve this is bu adding a HSTS header for your domain, google recently announced that they will introduce this for the complete www.google.com domain as well!

 

Note: Some other great articles have been written about this subject, but I intentionally wrote this article to reach out the Sitecore (and SharePoint) community!

The configuration is included in the blogpost below, it will also be released as a XDT as part of a bigger security project

It’s not only about the login page

As the Sitecore hardening guide describes:

Make the login page available only to SSL requests

This is not true. This guideline suggests that the login page is the only page taht needs to be exposed via https, but every page should be served over SSL. And when SSL is mentioned, the technique that should be spoken about is TLS, Transport Layer Security. SSL is completely insecure, while TLS is the improved version of SSL and is nowadays on version 1.2.

Why is this not true? When people are connected to insecure Wi-Fi networks data can often easily be sniffed, which means, all traffic can be captured by other devices. This means that all the data that flows over the (virtual) wire, can be captured. When the data is send over HTTP, this data is unencrypted and can by the persons who are capturing this traffic. Personal interests, details, and-so-forth can be captured. Imagine what happens when login credentials would be send over HTTP?

Things get even worse, when someone manages to intercept the connection, which is called a “Man in the Middle”. This person would be able to intercept the traffic, monitor it, transform it or even redirect it to another location, while the visitor wouldn’t have any indication on what’s currently happening.

During my session on the SUGCON in Copenhagen, I brought a small device with me which is called a “rogue access point”, which tries to trick devices to automatically connect to this wireless access point. See what happened in a few minutes, before my session even started and almost no visitors were in the room:

But my credentials are posted over HTTPS, right?

That’s true. That might be true. But at the moment that someone would be able to intercept the initial request for serving the loginpage, which would be over HTTP, that person would be able to alter or redirect that request and prevent the initial request from ever landing on it’s initial destination:

MITM

I could redirect traffic to any site, fishing email, passwords, et cetera.

Forcing the browser to use HTTPS

To prevent this kind of attack, even the initial request, must be send over HTTPS. Most visitors enter the address of the website that they want to visit in their browser:

www.website.com

mention that no protocol has been specified. Browsers initially send this request over HTTP and BAM: you’re vulnerable.

To solve this, the “Strict-Transport-Security” header can be used. When this header has been set, the browser will know that they should never visit that website over HTTP, but always use the HTTPS variant of the address. Pretty much all major browsers support this header.

Trust On First Use (TOFU)

Please note that this solution is a TOFU-solution – Trust-On-First-Use. When you have never, ever visited this website, it’s very likely that the website will be requested over HTTP – At this part, the visitor still vulnerable and the request could be intercepted/altered/redirected.  But let’s assume that this first request is safe.

Setting the Http Strict-Transport-Security header (HSTS)

After this initial request, The website should 301-redirect the request immediately to the https-equivalent, together with the following header: Strict-transport-security: max-age=31536000. The browser will cache this value and now knows that for every request, even if it has been initiated over http, the secure equivalent has to be used.

To show off what exactly happens, I included that images below:

The inital request

The initial request is over http and responds with a 301 -> https://insurco

The page gets redirected

The page that is served securely, has the strict-transport-security header added to the response

The next time the user tries to visit the webpage over http

An internal redirect (status code 307) to the secure page, no network traffic is involved in this step

Cool, I want this, how do I do that?

First, a secure connection is needed. After that one, you can configure your proxy to inject this header on every request, or configure the IIS Rewrite module of Microsoft Internet Information Server to use this one. My advice would be to configure this rule on the the proxy, but I you don’t have any access to this proxy or it takes too long, it’s also possible to configure IIS using the following rule. It forces the website to be visited over https and sends the Strict transport security header along with it.

&lt;rewrite&gt;
		&lt;rules&gt;
			&lt;rule name="Http Redirect to HTTPS" enabled="true" stopProcessing="true"&gt;
				&lt;match url="(.*)" ignoreCase="true" /&gt;
				&lt;conditions logicalGrouping="MatchAny"&gt;
				  &lt;add input="{HTTPS}" pattern="off" ignoreCase="true" /&gt;
				&lt;/conditions&gt;						
				&lt;action type="Redirect" url="https://{HTTP_HOST}/{R:1}" appendQueryString="true" redirectType="Permanent" /&gt;
			&lt;/rule&gt;
		&lt;/rules&gt;
		&lt;outboundRules&gt;
			&lt;rule name="Add Strict-Transport-Security when HTTPS" enabled="true"&gt;
				&lt;match serverVariable="RESPONSE_Strict_Transport_Security" pattern=".*" /&gt;
				&lt;conditions&gt;
					&lt;add input="{HTTPS}" pattern="on" ignoreCase="true" /&gt;
				&lt;/conditions&gt;
				&lt;action type="Rewrite" value="max-age=31536000" /&gt;
			&lt;/rule&gt;
		&lt;/outboundRules&gt;
	&lt;/rewrite&gt;

 

Securing the Initial request

It is possible, for several browsers, to secure the initial request as well. Most major browsers work with a HSTS preload list. A domain can be submitted to this list on https://hstspreload.appspot.com/ which will put into the preload list of all browsers.

Be cautious! Things may break!

This preload list requires a very important setting for the preload list: includeSubdomains. This setting forces that every subdomain should be served over https as well. If any subdomains exist that are only accessible over HTTP, they will break.

Summary

Serving sites over HTTP is not safe. Although you might only serve content, attackers may use unsafe connections to inject malicious forms, redirect requests, phish usernames and passwords. To force browsers (and thus, their users) to connect over HTTPS, the Strict-Transport-Security header should be used.

 

 

 

Sitecore Security #3: Prevent XSS using Content Security Policy

$
0
0

Clientside code is being used more and more on modern websites. Any kind of resources, for example Javascript, css, fonts, complete pages can be loaded dynamically into websites, from the current website or from an external domain. Attackers might be able to pull off an XSS attack by loading different kinds of data or scripts into your site which will run on your client’s browsers. These injections might happen on your own site, or in external services that you make use of (for example, disquss, or ads you are displaying). Applying a content security policy is one of the defenses against this kind of attack. This blogpost shows of scenarios that might happen (some of them tailored to Sitecore) and how the content security policy can help to prevent a successful attack from happening. As regular solutions provided on the internet do not supply the flexibility that a Sitecore solution (and CMS’ses in general) needs I decided to create a content manageable module and added that one to my SitecoreSecurity module.

This is not a write-up on the complete CSP specification, there are other great sources for that on the web, I included them at the end of the article.

The module will be available on the marketplace when it has passed quality control.
Sourcecode is available on: https://github.com/BasLijten/SitecoreSecurity

The danger of XSS attacks – some scenarios

XSS attacks on itself can be quite harmful, but often the lead to even worse attacks. Imagine what could possibly happen when someone is able to inject custom JavaScript into your website. That attacker is able to run code on the clients machine, which could lead to several situations. The least harmful is showing an alert:

From a business perspective, this is a situation that you don’t want to appear on your site. The complete site could even be defaced.

A more harmful situation is that a user might get control over your browser, without even noticing it. They would be able to load an external JavaScript and take over control of your browser. With the current HTML5 capabilities being able to use the camera, this could lead to a situation that the attacker can take videos or pictures remotely:

I even showed this off during my SUGCON presentation – Sitecore might be secure, but your site isn’t:

Worse things happen, when other parts of your security aren’t in place as well, for example, secure cookies (when working over HTTPS and yes, you should always do that), http-only cookies or session management. I showed off a scenario at the SUGCON where session fixation was possible, due to an XSS vulnerability in combination with bad session management. In this case, Bobby Hack (identity: extranet\robbert) was able to view personal details of me, Bas Lijten:

What did these hacks have in common?

All hacks had one thing in common: the attacker was able to inject JavaScript into the page via a query string parameter that got interpreted or to load a malicious external JavaScript file that got executed.

A very simple, yet effective way to prevent reflected XSS attacks, is to apply correct input and output validation. Don’t trust your external actors, never! But when that line of defense is broken, or some external service that you use has a broken security layer, other defensive mechanisms have to come in place. This is also known as “defense in depth”: never rely on a single line of defense.

How can the content security policy help in here?

With the CSP, policies can be defined on what kind of resource is allowed, how is it allowed and what sources are trusted? The resources are split amongst different categories, for example, default (your fallback/main policy), script, images, styling, et cetera. Per resource different options can be applied:

None Prevents loading resources from any source
All Allows all options
Self Allows loading resources from ‘same-origin’
Data Allows loading resources via de the data scheme (base 64 encoded images)
Unsafe-Inline Allows the use of inline-elements; style, onclick, script tags, et cetera
Unsafe-eval Alls unsafe dynamic code evaluation, such as the JavaScript eval() function
Hosts The external hosts that are allowed to load content from

Not every parameter is available in each resource-group, it depends on the type of resources. Due to the categorization for each resource-type, flexible policies can be created to allow javascript to be loaded from *.googleapis.com, disable inline-script and disable the unsafe-eval functions, while the css may only be loaded from the same-origin and Insafe-unline is allowed. A policy like that looks like:

Content-Security-Policy: default-src 'none' ; script-src 'self' 
https://*.googleapis.com; style-src 'self' 'unsafe-inline' ;

Please note that all major browsers support the content security policy header, some in the form of the normal header, some need the X-Content-Security-Policy headers. Some browsers do not support all resources as well.

Creating a CSP

To create a CSP by hand can be very time-consuming. You’d need to know about the exact specifications (a script policy contains other parameters than a CSS policy, while the referrer section looks completely different). Is the option ‘none’ with or without quotes, is the option ‘self’ with or without quotes? How to specify the allowed hostnames? Scott Helme, security specialist, created the website report-uri.io, on which he hosts security tools and one of them is to create those CSP’s. Go play around with it and see how stuff works ;).

For Sitecore, I created a module to do this, as multiple policies may need to be served and maintained per site. These CSP’s are manageable from content without too much effort, more on that later

Testing a CSP

When configuring a CSP, the policy can be created and be applied to the page. The next image contains the out of the box Sitecore landing page, with the following CSP applied.

Content-Security-Policy:default-src 'none' ;script-src 'none' ;style-src 'none' ;
img-src 'none' ;font-src 'none' ;connect-src 'none' ;media-src 'none' ;object-src 'none' ;
child-src 'none' ;frame-ancestors 'none' ;form-action 'self' ;manifest-src 'none' ;

One of the lessons learned: Misconfiguring the CSP can seriously damage your site! The good thing is, that the Console exactly shows what is going wrong:

While this information can be used to test and modify existing policies, it’s not convenient to do this with a broken site. Luckily, there is a solution for this one: The Content-Security-Policy-Report-Only header. When the same policy is used with this header and the original CSP is deleted, the site will still work (unprotected!), while the new policy can be tested:

A mixture of a CSP and CSP-Report-Only will work together: Using this mixture, your site remains protected, while the modifications can be tested. Of course, different pages, with different policies will behave differently on different browsers (and how many times can the word ‘different’ be used in one sentence? I need to differentiate more ;)). See Akshay’s blogpost on the browser differences for more info! This can be pretty hard to test and monitor in different circumstances. And of course, there is a solution to test and monitor these policies.

Reporting CSP violations

To monitor violations (which can be malicious, or might be due to an error) is not too hard: the CSP specification contains a report-uri field which can be used to send to violations to. The website report-io, which I mentioned earlier, can also be used as endpoint to collect the CSP violations. I configured my site to report it to this service:

In this specific example, some enforced policies are shown (a filter on report-only didn’t work, as I didn’t have any report-only errors at the time). The first policy which is shown, shows a violation on the script-src resource policy: because of the use of the unsafe eval function, this violation happened. Another example is the 4th one, where an image violation has taken place. I configured my policy to only allow images from my own domain, while the site tried to load images from an external domain.

Seeing a CSP in action!

In my example I referred to a XSS hack that I showed during the SUGCON. This hack involved loading a malicious, external JavaScript that was injected into a trusted subsystem. This Javascript was loaded on my page, which caused to remotely take a picture of me using my webcam:

When the CSP is applied, this leads to the following behavior: the requests to the domain that is not whitelisted get blocked by the CSP. This mitigation prevented a malicious user from taking a picture of me.

Introducing Convenient Content Security Policy for Sitecore

While most blogposts on the internet learns us how to configure IIS to use CSP (by adding a header in the web.config), this doesn’t work too well for Sitecore. Each page might need another CSP (although I wouldn’t recommend that one) and how would a multi-site setup be configured? Would every CSP have to be created on an external site, copied over, verified, published, et cetera? While this would work for the first times, this wouldn’t work over time.

I created a template which contains all of the CSP 1.0 and 2.0 options (yes my friends, next surprise, there are multiple versions ;). Using this template, separate policies can be created:

Another template, the Content Security Policy link, which contains the CSP Link and the CSP Report Only link, can be used as one of the base templates of a page ( I would prefer to have one single base content page template to which all the generic templates can be assigned). This results in the addition of two extra fields on your pages:

Using this mechanism, CSP’s can be reused easily, be put on standard values, be updated easily, et cetera! This results in the following extra response headers in your request.

This module adds the X-Frame-Options header as well, based on the frame-ancestor setting for your CSP. It should lead to the same result, so it’s no use to configure this one separately and possibly differently. This would lead to strange browser behavior. Especially with the introduction of the X-Frame-Options header in Sitecore 8.1 update 3 – out of the box the X-Frame-Options: SAMEORIGIN header would be send in the response: this would conflict with this CSP policy.

Summary

The Content Security Policy header is an excellent mechanism to defend against XSS and other code injection attacks. While the CSP itself can be quite big, confusing and even break your site, there are several tools and patterns to aid you in the journey to a safe website. With the introduction of the Sitecore Security module, the CSP’s can even be managed from within your content on a per-site or even per-page basis!

 

 

 

 

 

 

 

 

 

 

Sitecore Security #4: Serve your site securely over https with Let’s Encrypt

$
0
0

In a previous blogpost about the Http Strict Transport Security I explained how to force connections to make use of https to encrypt connections. A lot of people think it’s expensive, hard to implement and slow. This blogpost shows off how you can get a free, secure certificate, get your Sitecore site up-and-running in no more than 5 minutes, just by using the Let’s Encrypt service. Source-code can be found here on Github.

Why https and what’s needed for that

Basically, the only thing that you need to serve your site is a certificate which has to be updated once in a while. Of course, your site has to be built in such a way that it can be served fully over https, or you won’t make the most out of it. (for example, no mixed content).

The reasons to switch over to HTTPS are numerous:

  • it’s safe – with a man in the middle attack, your data can’t be sniffed, manipulated or redirected
  • it’s faster – when used over http/2. Windows server 2012 doesn’t support this, but Windows server 2016 In my next blogpost I’ll show off some results
  • It’s better for SEO – When a page is served over https, google increases the page-rank for that page

But why are there so many sites not using https?

Costs – On a lot of different places, those certificates are offered for $10 to $200 dollars per year.

Extra work – Those certificates are send over mail (boohoo), should be downloaded manually or any other manual interaction is required to obtain and install these certificates. I promise you: This *will* be forgotten. One, two, ten certificates will be mangeable, but when you’re about to administer more-and-more certificates, things will be forgotten. Eventually, this will cause your site to break, as your certificates are out-of-date

Hard – Where should I store them? How can I import them to IIS? Who should do that?

Meet Lets Encrypt

Lets encrypt is an online service which has the mission to serve the complete web over https. To achieve this, using https has to be as easy as possible, that’s why they are using the following principles:

  • Free:Anyone who owns a domain name can use Let’s Encrypt to obtain a trusted certificate at zero cost.
  • Automatic:Software running on a web server can interact with Let’s Encrypt to painlessly obtain a certificate, securely configure it for use, and automatically take care of renewal.
  • Secure:Let’s Encrypt will serve as a platform for advancing TLS security best practices, both on the CA side and by helping site operators properly secure their servers.
  • Transparent:All certificates issued or revoked will be publicly recorded and available for anyone to inspect.
  • Open:The automatic issuance and renewal protocol will be published as an open standard that others can adopt.
  • Cooperative:Much like the underlying Internet protocols themselves, Let’s Encrypt is a joint effort to benefit the community, beyond the control of any one organization.

How does this work?

There are a lot of tools to get Let’s Encrypt to work with windows and IIS. Troy Hunt wrote a great blogpost on how to set Let’s Encrypt up with Azure web apps. The short story on how Let’s encrypt works is:

  • Validate the domain – if the website is accessible via the domain that you want to create certificates for it must be your domain.
  • Let’s Encrypt creates a token – that needs to be accessible via your website on that same domain
  • Place that token on your website – Let’s Encrypt must validate that token. If it’s accessible, the certificate can be created
  • Store and bind the certificate – Store the certificate in the certificate store, bind it to your domain in IIS and you’re ready to go!

One of the tools that’s available to get let’s encrypt to work with IIS is Let’s encrypt win simple. It offers an executable which lets you select a host to create a certificate for, install it to IIS and automate that process which is included above.

After the selection of the correct host, the tool tries to validate your domain and tokens:

Because of the protocol, the token is expected to be available via <yourdomain>/.well-known/acme-challenge/xxxxxxxxxxxxxxxxxxxxxx.

Get it to work with Sitecore

Out of the box, this doesn’t work with Sitecore, as the Sitecore (well, MVC) prevents the item from being rendered.

Using a simple web.config in this directory, fixes the issue. In my Security module on github, I included this web.config which automatically get’s deployed. It work’s multi-site, so it’s accessible via any site that you configure, no extra work is needed!

After validation by the let’s encrypt tooling and positive feedback, the certificate is automatically imported, put into the right store and bound to your IIS application. This results in a website that’s being served over https:

Summary

Securing your website has become much more accessible since the introduction of let’s encrypt and their automation tools! Using the my Sitecore Security module on github, the use of let’s encrypt is accessible as for any other .net application.

 

 


Getting started with sitecore: The 101-guide to the community

$
0
0

A few years back, back in the Sitecore 7.x days, I started to work with Sitecore. I originated from the SharePoint community (take note of the capital “P”!), where there are SO many active bloggers. I think this was caused by a bit of the history. “Back in the SharePoint 2007 days” all the SharePoint info we got, came from google, or from reflector, as the documentation wasn’t always “that well written”. It appeared that there were a few persons actively blogging about their findings and through the years, the amount of people actively blogging, writing cool code or helping each other out, exploded, but you had (and still have) to find your ways to find all the information.

I see the same pattern happening in Sitecore. A lot of great functionality, a great product, but not every feature is always documented. As everyone tries to get the most out of the platform, people are seeking the boundaries of the product and finding out how stuff works. A lot of people are looking for help, a lot of people are blogging, but it’s not always that evident to find the sources that you need. “Where is the community?” you might ask. And that’s exactly why I decided to write this blogpost.

A first free lesson: First lesson: SiteCore is written as Sitecore. Please take 
care of this, as most Sitecore community members are a bit sensitive to it ;).

Sitecore and the Community whereabouts

There are a few different places where the community gathers around, so if you are looking for information, try these places!

Your own blog!

Without people like you, a community doesn’t exist! Did you do something cool, noteworthy or do you just want to share your experiences? Write it down! Other people invest (valuable) time in their blogs to share their information with you, so why wouldn’t you do the same? To me, it started out to write things down that I shouldn’t forget. Why not share those notes with the community? Of course it may be hard to start, so check out the places below to get started.

Twitter

Engage actively in the Sitecore community by following this hashtag and account and post to twitter by using the #Sitecore hashtag

  • Follow the #sitecore hashtag in tweetdeck.twitter.com and you’ll read the latest news.
  • Follow @Sitecore – now explanation needed 😉

Sitecore Slack

https://Sitecorechat.slack.com – a very active community, although that the slack isn’t publicly available. Send a DM on twitter or LinkedIn to one of the following members to get access:

  1. Akshay Sura
  2. Michael Ian Reynolds
  3. Kamruz Jaman
  4. Adam Najmanowicz
  5. Nikola Gotsev
  6. Mike Edwards
  7. Johann Baziret
  8. Anis Chohan
  9. Richard Seal
  10. Dan Solovay
  11. Nathanael Mann

More info on Sitecore slack? Visit this page to find out how to join this community!

Community.Sitecore.net

https://Community.Sitecore.net – A forum hosted by Sitecore where community members can ask their questions regarding Sitecore

Sitecore on stackexchange?

Do you really want to start with your first contribution to the Sitecore community? Signup and commit yourself to the Sitecore community. We’ll get our own http://sitecore.stackexchange.com!

Join Conferences and usergroups

Tere are several usergroups througout the world and several yearly sitecore conferences. A few examples:

Usergroups

Noteworthy places to find information

Sitecore blog feed

http://feeds.sitecore.net/Feed/LatestPosts – a sitecore feed managed by Sitecore with interesting topics

Sitecore technical blogs

http://www.sitecore.net/learn/blogs/technical-blogs.aspx  – A list with a lot of (skilled) bloggers, but not complete, as a lot of new technical bloggers have been joined the community the last years

Blogfeed on Slack

https://sitecorechat.slack.com/messages/blogfeed/   – a broad collection of blogs, updated every minute of the day!

Summary

Getting involved into the community isn’t hard, you just need to find the right places. Find us on twitter, find us on slack and find us on the public forums!

Sitecore 8.2 update 1: Azure deployments, ARM, Web Deploy and the Sitecore Azure Toolkit

$
0
0

With the release of 8.2 Update 1, Sitecore also introduced support for Azure Web Apps. This release is, in my opinion, a major step for Sitecore as this update makes it very convenient to deploy to Azure using the Azure Marketplace or the provided PowerShell scripts, that’s why I think that this release is even bigger than Sitecore 8.2 initial. This deployment pattern is an interesting pattern to use on premise as well, although not all of the services can or should be used on premise. This blogpost describes how the Sitecore Azure Toolkit works. My next blogpost will describe how to use this toolkit to create your own custom web deployment packages, both for Azure and your on premise installation, with even older versions than Sitecore 8.2

Note: be careful when deploying to your own Azure subscription: when managed incorrectly, a Sitecore deployment on Azure can cause Azure to provision an extensively scaled environment, which generates many resources. Be careful as the cost of this could be high.

Update: modified the blogpost slightly thanks to excellent feedback from Rob Habraken, Steve McGill and Michael West. Thanks guys!

How Sitecore gets deployed to Azure

There are a few different ways to deploy Sitecore to Azure, but two of them make the most sense

  • Deploy from the Azure Marketplace
  • Deploy using PowerShell

Using the Azure Marketplace, a one-click-deployment is initiated (I won’t go too deep into how to provision this app, as Praveen Kumar Sreeram already blogged about it) :

When deploying this app, just a few parameters are needed which have to be provided during the creation steps. The required parameters are limited to Database settings, resource group name, license file and … the admin password!

 

This app provides a basic Sitecore 8.2 XM installation: no XDB whatsoever. The cool thing about the technique about this provisioning method is that it uses the same techniques as I describe in this blogpost: ARM templates and Web Deploy packages.

Deploying using PowerShell using the Sitecore Azure Toolkit

Another mechanism that can be used to deploy to Azure is PowerShell. Sitecore provided the Sitecore Azure Toolkit, which contains a few beautiful gems, but more on that later. This package contains a few folders:

  • Copyrights
  • Resources
  • Tools

The resources folder contains an 8.2.1 folder (which implies that support for other versions will be added in the future) with 3 folders: CargoPayloads, configs and msdeployxmls.

The tools folder contains a few assemblies and a PowerShell module, which provides three different public Cmdlets:

To deploy a new Sitecore environment to Azure, the Start-SitecoreAzureDeployment can be used, however, it requires a few parameters, with the most important being the ARM Template path and the ARM Parameter file:

ARM-what?

ARM-templates are Azure Resource Management templates that allow you to deploy an application using a declarative template. With a single template, it’s possible to deploy multiple services with their dependencies. It’s a JSON file in a specific structure which describes dependencies of services, their names, usernames, passwords etcetera. I won’t go too deep into the current implementation and how Sitecore used it, as they did a solid job on providing these. If you want to get more knowledge on ARM, you can get a lot of information on here. The templates themselves can be downloaded here.

This template needs to be fed with parameters to be able to do its job:

 

The template above, which is used for the XP1 provisioning, needs the Sitecore admin password, MongoDB ConnectionStrings, SQL Server credentials, the license file and the packageURL of the web deploy packages, in total 4 of them: for Content Management, Content Delivery, Reporting and Processing. . Interesting to note is These packages can be downloaded on dev.sitecore.net. In a classic (on premise) situation we were always forced to create role packages ourselves. How many people created PowerShell, batch files or other smart solutions to generate those role packages for us? Well, Sitecore created tooling for this job and I am pretty positive about it! And the fun part: this tool can be used to create your own packages, which can be used to deploy to Azure and on premise!

How are Web Deploy packages created?

A Web Deploy package 101: this is a package that can be deployed to Azure or IIS using MSDeploy and contains the website files, database files and a lot of other information. In this Web Deploy package, a parameters.xml can be supplied as well, which describes what parameters can be provided during deployment. Using this packaging technique, the same deployment package can be used for different environments, while providing different parameters for every environment. These parameters can be supplied by specifying the values in the msdeploy command, using a setparameters.xml or by using an ARM template.

The Sitecore Azure Toolkit contains a cmdlet to create Sitecore Web Deploy packages. One very important note:

How the Sitecore Azure toolkit creates web deploypackages is NOT a standard way of doing this. Usually, these packages are created using MsBuild during build, but Sitecore had to create this alternative, as it would become way to complex to create web deploy packages and maintain flexible configurations. Cargo payloads, the common and sku config are terms that Sitecore introduced. 

It requires the Sitecore package and some locations in which specific configurations are stored:

  • Cargo Payload
  • Common config
  • SKU Config path
  • MSDeploy XML

The configuration in this location is needed to create the new role-based Web Deploy packages: the standard zip of the webroot (Data/Databases/Website) will be converted to a Web Deploy solution which can be deployed to Azure or your local IIS environment and all the configuration required, per role, as described on docs.sitecore.net and in the installation notes, will automatically be applied. But how?

Basically it works this way:

  • skuConfig path: defines the kind of configuration for which the Web Deploy packages need to be created (in our case, XP1). The contents define which configs to apply.
  • commonConfig: the config that always needs to be applied, in any case. The contents define which configs to apply.
  • Cargo payload folder: the set of actions that need to be applied to add, for example, Application Insights support.
  • Archive and ParameterXML Path: the manifest and parameters.xml that need to be part of the Web Deploy packages.

In a “normal” situation these changes would have to be made manually or using own PowerShell scripts. These actions always consisted of:

  • Enabling/disabling patch files
  • Adding/removing files
  • Changing configuration

And that’s exactly what this PowerShell does for us.

Common and SKU config

Sitecore introduced the common and SKU config to be able to design roles and supply the required archive.xml, required parameters.xml and the cargo payload actions that need to be applied for that role. Those Cargo payloads are specified by the sccpls. sccpl is propably an abbreviation for SiteCoreCargoPayLoad.

Cargo Payload

The cargo payload is something that was introduced by Sitecore, to define transformations for existing web deployment packages and is, thus, not part of the Microsoft msbuild toolkit but is delivered with the Sitecore Azure Toolkit. It defines a set of configuration changes that can be applied to the Sitecore Web Deploy package and are tied to a central theme. Those themes are specified in the common and sku config and may contain one or more of the following themes

When unzipping these files, a few different folders are shown, which are tied to some actions.

  • copy to root contains an action to copy the items to the root of the web deployment package (for example SQL provisioning scripts, I will get back to this later)
  • copy to website contains files which need to be copied to the site root
  • XDTs are XML Document Transform files (configs to transform, for example, the web.config or any other XML.)
  • IoActions: an xml which contains information on patch files: which one needs to be disabled or enabled

These are actions that Sitecore defined which can be used to transform the data in this folder into the Web deploy package. For example, the Sitecore.Cloud.ApplicationInsights.sccpl contains two actions: Copy To Website and XDTS. using this technique, the “vanilla” on premise Sitecore installation does not contain Application insights, while it can be added to the packages that have to be used on azure. The copy To Website contains all patch-files and the Microsoft.ApplicationInsights binaries, while the XDT contains logic to alter the web.config and to add configuration to the connectionstrings.config

Below is an example of the IoAction file:

 

Archive and Parameters

The archive and parameter.xml are two files that need to be copied to the root of the web deploy package. This is an example of the parameter.xml of the CM role:

It contains a lot of parameters which may be different on every deployment and/or every role, that’s why they need to be parameterized ;).

 

Taking a look into the Web Deploy packages

Using the Start-SitecoreAzurePackaging cmdlet, the Web Deploy packages will be created. For the XP1 SKU, this will result in 4 different kind of packages: CD, CM, PRC and REP. The paths to these packages can be supplied to the config file, mentioned earlier (and I will get back into that).  When taking a closer look INSIDE the packages, the following structure can be seen (in this case the CM version). I will highlight a few observations:

Directory structure

As opposed to the default structure (Data, Database, Website), a totally different structure can be seen. Inside the Content folder the actual website can be found, while /data and /databases seem to be missing.

The /database folder is not available anymore: the database will be provisioned using the dacpac files, while the *.sql scripts are being executed after provisioning. References can be found in the parameters.xml that I shared previously. A nice addition is that the SetSitecoreAdminPassword.sql will be executed after installation: this means that the default password “b” will be overwritten!

Parameters.xml

As described previously: this file describes which parameters are required to deploy this package.

Website and the /data folder

The actual website can be found in the /content folder. When taking a closer look we find that the contents of the old /data folder is located in /app_data. The reason for this is probably that only website roots can be deployed to Azure, thus a /data folder wasn’t an option anymore. This means that your license.xml will be deployed to the /app_data as well.

When a plan comes together

In this (lengthy) blogpost I talked about the ARM templates, their parameters, the required packages that needed to be created and how these parameters worked. Some parameters need to be specified in the ARM parameters template, such as SQL username, password and the Web Deploy packages, which can be used to deploy to the correct instance. The ARM template in itself deploys the different Web Deployment packages to the different Azure Web Apps and takes care of all the required parameters in the parameters.xml. This toolkit can be used to create custom configurations and opens up opportunities to include specific customizations in your baseline!

 

Sitecore on Azure: Create custom web deploy packages using the Sitecore Azure Toolkit

$
0
0

In my previous blogpost I described how the Sitecore Azure Toolkit works and how to create web deploy packages. In this this blogpost I’ll explain how to create your own web deploy package configurations which can be used on Azure and on-premises, even with Sitecore versions older than Sitecore 8.2 update 1. You can apply role specific configurations, or add custom modules like Coveo, PowerShell Extensions, Unicorn, or even one you package up. Using these techniques will help you establish a repeatable process with standard tooling leading to decreased deployment time. How cool would it be to have Continuous Delivery and Deployment all the way to production?! I’ll demonstrate in an example, Sitecore PowerShell Extensions, how to work towards a continuous delivery process. As a bonus I’ll package Unicorn as well – future posts will depend on this example so why not tackle them now.

special thanks to Rob Habraken, Michael West and Kam Figy who reviewed this post!

Use Web deploy to add new functionalities

Deploying Web deploy packages is the way to go to deploy your Sitecore files to Azure and, in my opinion, to your on-premises IIS installation. In the future I will shed light on my thoughts about that subject. A cool feature of web deploy is, is that you can parameterize environment-specific variables for your deployments. In the current WDP’s provided by Sitecore, they have parameterized the Sitecore admin password, database connection strings for core, web and master, basically every parameter that differs on each environment.

When looking from a Helix perspective, the following layers are available.

The current Azure toolkit, provided ARM templates, and web deploy packages allows you to deploy Sitecore + its infrastructure; this makes up the Foundation. The Feature and the Project still need to be deployed. One option is to simply FTP the files, however there are some drawbacks to that. Web deploy is surely the way to go. The process to create web deploy packages is fairly straightforward when using the parameters.xml to parameterize your deployments.

There is one “grey” area: Sitecore modules in this Foundational layer.

I truly believe that when you are deploying your projects, you should clear the complete web application site root, redeploy the foundational modules, the features and on top of that the projects. A big challenge with this approach however includes the following:

How should Sitecore module packages be deployed? The use of web deploy packages does not include installation modules such as Sitecore PowerShell Extensions, Coveo, WFFM.

Difference between initial provisioning and continuous delivery

The initial provisioning of these modules still needs to be performed using the Installation Wizard, as web deploy can’t deploy the content of Sitecore packages (yet) and can’t trigger post deployment steps. But when the initial content has been provisioned and the configuration has taken place, the binaries need to be redeployed over-and-over again, there’s no content involved anymore in this process. In the past I saw solutions where people would redeploy these packages using Sitecore ship, which took, in my opinion, too much time; this overhead wasn’t needed at all. But how can we get those files onto the platform without using Sitecore packages? Naturally, Webdeploy is the answer for that 😉

Create webdeploy packages from Sitecore update packages using the Sitecore Azure Toolkit

The Sitecore Azure toolkit offers the ConvertTo-SitecoreWebDeployPackage cmdlet which let you create web deploy packages from Sitecore module packages:

This is a very basic webdeploy package which quickly can be deployed to your (Azure environment) on top of your Sitecore installation. No parameters can be specified, no config files can be altered. It’s all or nothing. Of course it’s possible to open this zip yourself, alter the configuration files and insert the parameters.xml. Who wants to do that? It’s a very interesting option to use if none of those changes are needed.

An alternative is to create a cargo payload and add it to your Sitecore webdeploy packages. With a small investment of your time you will create modules that can be installed over-and-over again on different environments, without having to alter any of the patch-files anymore.

A small recap on cargo payloads and configurations

The Sitecore Azure Toolkit provides a PowerShell command to create web deploy packages from existing Sitecore module packages. This command requires configuration files to determine what kind of web deploy packages need to be created. These configuration files point to cargo payloads: small blocks of functionality which can be added to the web deploy packages. Examples include Application Insights binaries, Azure Search functionality settings, or role specific configuration. Of course it’s possible to create own cargo payloads, that can be used in your own configuration.

How to add your own cargo payload

The cargo payload defines a set of actions/transformations that should be applied on the web deploy package. It’s a file which contains a sccpl extension, “Sitecore Cargo Payload”, and is a standard zip file. The CPL’s that are delivered with the Sitecore Azure Toolkit expose 4 actions:

  • CopyToRoot: copy files to the root of the web deploy package, such as dacpac files and sql scripts. These are typically files that are needed for provisioning
  • CopyToWebsite: copies files to the website root – the files will be deployed to the Sitecore web root using webdeploy
  • IOActions: xml file with actions to enable, disable or delete configuration files.
  • XDTS: folder with Xml Document Transformation files. These transformations can include changes to the web.config or other xml-files like every file in the include folder

In the common.config and sku config files get defined what CPL should be applied to what role. The common.config is used for all packages, while the sku.config defines the role specific transformations.

Creating your first cargo payload

Let’s start with the Sitecore PowerShell Extensions module as an example. It contains everything: Content, configuration, binaries and parameters that may differ per environment. We also want to only install it on the CM. It’s a prerequisite for SXA and let’s face it, every solution should make use of it especially for continuous deployment.

The starting point for these foundational modules, which come in the form of Sitecore module packages, is the previously mentioned ConvertTo-SitecoreWebDeployPackage command from the Sitecore Azure Toolkit. It basically takes all the files from the files folder and converts that one to the most basic possible web deploy package:

This package can easily be converted to a Cargo Payload. Create a new folder with the name “BasLijten.PowerShellExtensions” and create a CopyToWebsite folder in it. Drop all the files from the WDP into this folder. That’s it!

Adding transformations and parameters

The next step would (probably) be to transform it to your needs. It’s important to keep those files as vanilla as possible: this will make future module upgrades easy, Next to that, you need to keep track of your own customizations. By using XDTs (Xml Document transformations) your customizations can be applied on vanilla patchfiles, without having to modify those patchfiles.

The most important question is always: what configuration is generic to my environments and what would be specific to a server/set of servers? In my situation we are hosting multiple Sitecore instances which we want to be configured the same, but have instance specific parameters. To clarify this, I’ll take the security settings for the Sitecore PowerShell Extensions as an example:

I want the the remoting service Enabled and requireSecureConnection to be configurable on each environment (thus specific) and I always want to deny Sitecore\admin permissions on each server. This means that I should add the <add Permission=”Deny” IdentityType=”User” Identity=”sitecore\admin” /> line using XDT (so I am not touching the source) and I should use parameterization for the remoting attribute change.

The XDT

Inside the BasLijten.PowerShellExtensions folder, create a folder called “XDTS”. To apply an XDT to a patch file simply recreate the same folder/file structure to that file. In this case, the patch file resides at the location “App_Config\Include\Cognifide.PowerShell.Config”. This means that the same folder structure in the XDTS folder needs to be created. To be able to apply the XDT transformation to “Cognifide.PowerShell.Config”, the file “Cognifide.PowerShell.Config.xdt” has to be created in that folder. The transformation will be applied automatically after creating the WDP. For the sake of completeness, this is the XDT that I used:

The parameters

To be able to disable or enable remoting deployment time, this needs to be a parameterized variable. Please note: you are not working in the CPL folder anymore, but we are going to add a custom configuration. The addition of this parameter is quite easy:

  • Make a copy of \resources\8.2.1\msdeployxmls\xP0.parameters.xml
  • Rename this copy to MyFirstWDP.parameters.xml

Upon opening this file you will see a lot of parameter entries in this file. These entries are all definitions for the parameters that should be used:

When watching this file closely , there are entries which are used for SQL, text files and XML files. Let’s take the Cloud Search Connection String as an example:

A parameter with the name “Cloud Search Connection String” is defined. The parameterEntry describes how that parameter should be applied. In this case, it’s an XML file.Tthe scope defines the location of this XML file and the match is an XPATH query to locate the right node.

Below is the snippet that I used to make the remotingEnabled and RequireSecureConnection parameterizable:

When a plan comes together – create your web deploy package

Our goal is to apply the CPL’s to the web deploy packages. To make this happen, two steps are needed:

  • Create a sccpl from the CPL folder
  • Create a custom configuration to apply the right CPL’s

Create a sccpl

I first fiddled around by manually zipping that directory, rename it to sccpl and try to see if it worked. Turned out that I made quite some mistakes. Aside from the manual mistakes, it’s boring work, that’s why I created a PowerShell script to automate it. If I had taken a look at the commands that are provided by Sitecore, I wouldn’t have wasted much time. Import-Module Sitecore.Cloud.CmdLets.dll is all you have to do:

This library exposes the New-SCCargoPayload command, which creates the sccpl in a correct way for you.

Create a custom config

In this example, I will modify the config for the XP0 seat, which contains all the roles, but any config can be used

  • navigate to the \resources\8.2.1\config folder
  • copy xp0.packaging.config.json
  • rename the copy to MyFirstWDP.packaging.config.json
  • open it.

You’ll see the following contents:

  • change the parametersXML value to MyFirstWDP.packaging.config.json
  • add your new sccpl to the sccpls array and you’re good to go.

Create and install the web deployment package

The final step is to execute the Start-SitecoreAzurePackaging with the correct parameters:

Start-SitecoreAzurePackaging -sitecorePath ‘C:\shared\Sitecore\repo\Sitecore 8.2 rev. 161115.zip’ -destinationFolderPath C:\sitecorewdps\custom -cargoPayloadFolderPath .\resources\8.2.1\cargopayloads -commonConfigPath .\resources\8.2.1\configs\common.packaging.config.json -skuConfigPath .\resources\8.2.1\configs\custom.packaging.config.json -archiveAndParameterXmlPath .\resources\8.2.1\msdeployxmls

This will create a customdeployable Sitecore web deploy package which can be deployed to azure. When taking a look inside the WDP, we do see that the XDT to add a rule to block remoting access for the Sitecore admin got applied and that the default remoting settings (remotingEnabled=false) are still there:

I will not go into the details on ARM, as Rob Habraken will do that. And as I don’t want to re-provision the apps (and thus creating new databases) using the ARM templates, I just want to redeploy the contents. This can be done using msdeploy. All we need to do is supply the parameters using a set parameters.xml:

It’s basically specifying the parameter-name and the value that you want to provide. In this setparameters.xml I specified the remoting service attributes “enabled” and  “requireSecureConnection” both set to be true.

Below is the batch file to deploy web deploy package locally:

Please note: this will probably not work with your own Azure deployment, as I modified my WDP a bit more, but those changes go too far for this blogpost: I will go deeper into that subject in my next blogpost on provisioning and (continuous) deploying Sitecore.

Conclusion

Sitecore finally created a tool which can be used to configure and assemble role specific packages with own or community modules. As it is Sitecore version agnostic and, although it’s an azure toolkit, it can be used to create packages for on premise, I am definitely going to use this at our own company. To create a custom cargo payload, very little effort is needed, while adding it to a custom configuration is little work as well. I would advise to start using this toolkit right away to create your custom role specific web deploy packages and use web deploy to deploy your Sitecore environments, be it on premises or on azure. Fun fact: I wrote this blogpost without deploying to Azure once, due to lack of credits J

Use the Sitecore Azure toolkit to deploy your on premises environment

$
0
0

Let’s face it: a lot of customers won’t deploy to Azure immediately, but will have a migration to Azure on their roadmap for the next year. It’s wise prepare as much as possible to make the transition smooth. This blogpost shows off how what the differences between the current Azure and classic on-premises are and how to create custom web deploy packages for your on premise environments, to be in line with a possible future upgrade to Azure. It will make your local deployments repeatable while making use of Microsoft standards. Additional advantage: Your (initial) deployments may happen faster!

See the video below where I explained what I did

The biggest difference between classic deployments and Azure

In classic deployments we do recognize the /data, /database and /website folders in the root of IIS. This choice was made in the past to be sure to keep the /data folder not accessible via IIS. In Azure, however, this is not possible: the choice has been made to move this /data folder into the /App_data. This will also be the “new” location of your license.xml. Some 3rd party modules, such as Unicorn, also have their data-storage in /data, these should be moved to /App_data as well.

Preparations: Create dacpac files for your databases

In the past, we had to manually attach the databases to SQL before Sitecore could be enrolled. With Sitecore 8.2.1 Sitecore also delivers dacpac files to deploy databases. A new directory, called DACPAC, has been added to the database folder in the Sitecore site root folder. This DACPAC isn’t used while executing a “classic on-premise” deployment, only when deploying to Azure.

DAC is A data-tier application (DAC), a logical database management entity that defines all of the SQL Server objects – like tables, views, and instance objects, including logins – associated with a user’s database. A DAC is a self-contained unit of SQL Server database deployment that enables data-tier developers and database administrators to package SQL Server objects into a portable artifact called a DAC package, also known as a DACPAC.

This technique can be used on premises as well, even with older versions of Sitecore. The only thing that is missing are the DACPAC files, but they can easily be created:

  • Install a fresh Sitecore instance (with the version that you want to deploy using web deploy)
  • Open up visual studio
  • Connect to the database using SQL Server object explorer
  • Right click on the database that needs to be exported as a DACPAC
  • Make sure to select “Extract Schema and Data”
  • Export
  • Repeat this for all four databases

Create a Web deploy package

The following steps have been extensively explaind in my previous blogpost, so I will go rather quickly through it.

As this will be a web deploy package used for on premise deployments, no cloud cargo payloads are needed. This means that new configurations have to be created. Pre 8.2.1-packages are lacking DACPAC files and this deployment setup needs a datafolder change, thus a new cargo payload has to be created for this setup. As the cherry on the Apple sauce (a dutch saying, which means: “The icing on the cake”) the Sitecore 8.2.1 technique to set the admin password during setup will be included as well!

Create the cargo payload

The first step is to create  a new cargo payload which contains the DACPAC files, the SQL script to set the admin password, the patchfile for the datafolder and the license.xml. With the ARM templates in Azure this license.xml can be provisioned, but I didn’t find a way to dynamically include this license.xml in web deploy packages yet. A cargo payload with the following files and structure has to be created:

  • CopyToRoot
    • Core.dacpac
    • Web.dacpac
    • Master.dacpac
    • Reporting.dacpac
    • sql
  • CoptyToWebsite
    • App_Config
      • Include
        • config (with \App_data as datafolder)
      • App_Data
        • xml

Run the New-SCCargoPayload command to create the sccpl and you’re good to go.

Create the custom config file for your web deploy package

I created two new configuration files: one common.config and one singleRole.config which are used to create the web deploy package. No Sitecore.Cloud references should be left in heren, only the BasLijten.SIngleRole.sccpl should be included. In these files it’s possible to make different configurations for different roles!

Alter the archive.xml and parameters.xml

The archive.xml and parameters.xml will be copied into the web deploy package by the tooling. I copied an archive.xml and removed the entries of some .sql files that aren’t available. The parameters.xml defines what parameters are required to provision the application. An example is the following snippet:

3 parameter entries are defined. Each Parameter entry tells what to look for (ProviderPath, XmlFile, Textfile), what to look for and what to do.

  • Sitecore Admin New password – replace the PlaceHolderForPassword string in the SetSitecoreAdminPassword.sql
  • Core Admin Connection String – make sure that the Sitecore.Core.dacpac gets provisioned to this database and execute SetSitecoreAdminPassword.sql on this database. As the PlaceHJolderForPassword was replaced by the provided password, the new password will be stored in the database. “Bye Bye b” :D.
  • Core Connection String: replace the core connectionstring by the provided value in the XML file “connectionstrings.config”

The complete parameters.xml that I used can be found on Github.

Create the actual package

The last step that needs to be taken is to run the Start-SitecoreAzurePackaging command (as described in my previousblog) which creates the new Web deploy package for Sitecore 8.1.3

Deploy Sitecore using webdeploy

At this point, a Sitecore webdeploy package with the 8.1.3 database dacpac files, the Sitecore root, the license.xml, datafolder patchfile has been created. All you need to do, is to create a setparameters.xml which provides all (required) parameters and start provisioning:

The next step is to deploy using the following batchfile:

Conclusion
Creating a custom web deploy package for an older version of Sitecore isn’t hard, it basically just needs the DACPAC files and a patchfile. This makes deploying to your on premises environments a blast and lets you use the same tooling and techniques for your on premise installation as well for azure.

Zero downtime deployments with Sitecore on Azure

$
0
0

From a business perspective, downtime is not desirable, ever. And if possible, you want to deploy as often as possible, even multiple times a day Maybe even 50 times a day for complex multi-instance  environments. And if there would be any downtime, that should be during nights, as most visitors would be asleep at that time. From a technical perspective, deployments should occur during business hours: all the developers and administrators are working during these hours, thus issues (if any) could be resolved as every engineer would be available.

We all know about this story, but how many organizations really implement this scenario? This blogpost will show what challenges exist when deploying web applications and how easy it is to implement zero downtime for Sitecore on Azure. The move to Azure not only opens up opportunities for automatic scaling (please make sure to watch his video as well!), but also offers possibilities for enhanced continuity! This blog post does not show off how to integrate with Visual Studio Team Services and Microsoft Release Manager, that will probably be a future topic. Don’t want to read? Watch this video!

As this blogposts is quite technical, here are some links of the basic fundamentals

Provisioning vs Deployment

Provisioning usually means that the infrastructure will be set up for a new application, including all dependencies. Deployment is the process of putting a new application, or a new version of an application onto a pre-provisioned environment. How does this compare to Azure Resource Manager (ARM) templates that Sitecore provided? The ARM template provisions the azure infrastructure and in these ARM templates the web deploy packages are referenced to deploy the first version of your application.

During a deployment lifecycle the infrastructure shouldn’t be reprovisioned, only the new application needs to be  deployed to the environment, while not re-deploying the databases, as it would reset all content to the initial state: all user generated content would be deleted.

This blogpost focuses on the part where provisioning stops: application deployment. The starting point is a provisioned Sitecore XP xM1 infrastructure (Content Delivery and Content Management) in Azure, with modified, user generated content:

Deploying an Application – Potential Issues

Deploying an application from Visual Studio is not too hard, see this blogpost by Johannes Zuilstra on this subject. However, a repeatable and stable deployment within your continuous delivery/deployment process could be another story. What if you are deleting files from your solution? Deploying Web deploy packages does not retract these files: it a) removes the complete website root or, b) adds new files and alters existing files. So how should cases like these be handled? Especially when working with Unicorn, this is a problem. Another issue is the downtime issue: adding new content will trigger an application pool recycle which conflicts with the zero downtime requirement. It seems that we have two objectives:

  • A stable deployment
  • Deployment without downtime

Stable deployment – redeploy Sitecore and your application

At Achmea, we always completely redeploy our application: this means cleaning up the website root, re-deploy Sitecore and deploy the application on top of that fresh Sitecore installation. This way we make sure that no “old files” will remain and that we are always working with the latest version of our Sitecore baseline and application.

The same pattern can be used in the cloud:

  • Clean the website root
  • Deploy a Sitecore web deploy package
  • Deploy your application

There is, however, a big disadvantage when using the default Sitecore web deployment packages: they deploy a new database including content (dacpac) and require a LOT of parameters, including the database connection strings, application insights keys and a lot of other parameters, which are usually provided by the ARM templates.

When this package will be deployed, all parameters need to be provided, otherwise the deployment will fail. But what is the value of the connectionstrings? What is the serviceURL of the cloud search parameter? And if these parameters are known and provided, then the deployment of this package will recreate the databases; something that shouldn’t happen as well. The user generated content (“Sitecore Experience platform – Modified content”) would be lost. The first step to work towards zero downtime, is to solve these issues.

Another way of Configuring Connection Strings in Azure

This is important information to work towards zero downtime deployments. In the current approach, the connectionstrings are stored in the connectionstrings.config, which means that they need to be redeployed for each deployment:

Thanks Rob Habraken for showing off this neat little trick! Navigate to Development tools\console to see what’s actually installed on your app service!

However, Azure offers an alternative way of storing connection string values as well! Navigate to the Application settings where an app settings and connection settings can be found. When entering connection strings here, these settings will be used over the settings specified in the Connectionstrings.config. They are persistent as well, so after a removal of all the content in your app service, these Connection strings are still valid. This also means that a new deployment of your application, without Connection strings, will still work. This also means that those parameters aren’t needed anymore at deployment time.

It’s also possible to enable “slot setting” – Slots will be explained later in this article.

Deploy the application – not the database

As seen in the parameters.xml, there are a lot of references to the dacpacs and custom SQL scripts, which all need input as well. As those databases already have been installed during the provisioning of the application, all those references can be removed.

Challenge + workaround – license.xml

The snippet above doesn’t show the license.xml – with a reason. Deploying the license.xml using the setparameters.xml, seems to be hard: I wasn’t able to include the licenseXML value yet, which means that another solution had to be found. For the time being a solution with a custom cargo payload containing the license will do, although it’s not a nice solution.

Create a new web deploy package for deployment

From the 22 initial parameters that were needed, 16 have been removed. The snippet below shows a stripped parameters.xml that can be used after the initial provisioning.

This parameters.xml only needs the necessary input for application deployments.

How those packages can be created is described in my previous blog posts::

Note: The dacpac files can’t be removed from the web deploy packages using the Sitecore Azure Toolkit – they are put in when the first package is created. For now, I didn’t investigate how to remove them easily – I left them in the package.

Deploy to Azure using PowerShell

As mentioned before, I saw some great blogposts on how to deploy your customizations to the cloud using Visual Studio. This is great during development time, but I wouldn’t recommend to do this when working towards continuous delivery. Aside from that you can’t deploy web deploy packages using visual Studio.  Christof Claessens provided a nice script on how to use web deploy from the commandline. This script had to be slightly modified to deploy the web deploy package, since this script builds the solution first, there was a problem with the contentPath used and I had to include the setparameters.xml to supply my 6 parameters. Below, my modified script can be found:

This will start the web deploy from the command line towards Azure. Let’s see what happens:

  • the web app becomes unresponsive and even may throw some errors.
  • After the deployment, the cache needs to be build up; there is no warmup of the pool
  • There are no connection strings specified in the ConnectionStrings.config:
  • The connectionstring values are default values, as they appear in the default installation files

  • After warmup, we do see a working website, with the changed content in place:

Objective 1: check – a working website, is accomplished, which still preserving the user generated content. No passwords, usernames or connection strings were provided, while the website still works. A new “vanilla” Sitecore web deploy package was created and deployed. The next step is to deploy a new application. This can be done in various ways

  • Deploy two packages: the vanilla Sitecore package using msdeploy and deploy your changes (application) using web deploy using visual studio. (I don’t recommend this from a continuous deployment perspective, but I will use this in my screencast and my blogpost)
  • Deploy two packages: the vanilla Sitecore web deploy package and an application web deploy package
  • Deploy one web deploy package: Use the Sitecore Azure toolkit to create a cargo payload for your application(s) and build a new web deploy package, including the Sitecore root.

Note: I am not sure what scenario I like the most, scenario 2 or 3. While developing/testing, scenario 2 might be the best as creating a full new package using the Sitecore toolkit takes quite some time , from a Continuous delivery perspective scenario 3 has a lot of advantages.

To “prove” that new applications can be deployed, a new solution has been created. Three changes have been made:

  • The default.css has slightly been modified: the contentTitle class got a red background.
  • The “sample item” template got an extra field: “note” (via Unicorn. The reason why this tool was used over TDS will become clear later in this blogpost)
  • The “sample rendering XSLT” has been modified to show the “note” content

These changes result in a few files that can be deployed using web deploy: the CSS, the XSLT and the serialized data in yml format. This results in the following, astonishing beautiful look & feel:

Note: Unicorn was already added in my Sitecore baseline (as a cargo payload, similar to the way I added Sitecore Powershell Extensions to my baseline in this post), so only the Unicorn configuration for my application had to be added to my project.

Deployment without downtime – meet the deployment slots!

The next objective is to deploy a new application without downtime and errors. In Azure, it’s possible to use “deployment slots“. These slots are separate web applications with their own configuration, their own website root and their own publishing profiles. They can be used to test your changes, before deploying the application to the production stage. As seen in the image below, a “staging” slot has already been created:

These slots provide the possibility of cloning the configuration sources (the app and connection strings). Remember the “enable/disable” slot settings? When enabled, they are specific to the slot they are configured in, when not the settings will be copied along with the application. In this example the settings are not slot specific, which means that those settings are clone: the same core, master and web databases will be used.

Note: When having slot specific settings, your application will get a cold start!

After testing the staging environment and the official acceptance by business owners, a “swap” can be executed. Both a direct swap and a swap with preview are possible:

“Swap” leads to a direct swap of web applications: your staging environment becomes production and vice versa.

“Swap with preview” does basically the same, but leaves a moment to cancel the swap before the slots get swapped (this is demonstrated in the video as well). It does exactly the same as a direct swap, but doesn’t switch load balancers yet.

Note: these actions can be triggered from Microsoft Release Manager as well, for a fully automated deployment!!

Note2: I learned from Rob Habraken that staging slots can freely be used in the Sitecore perpetual server based license model.

When the project has been deployed to the staging environment, the environment can be validated, tested and warmed up. After a swap, the new, provisioned application is available to everyone!

Objective 2: Zero downtime deployments – accomplished!

Convenient Roll back with Unicorn

Earlier in this article was stated that I wanted to use Unicorn. When rolling back deployments, it’s important to roll back the filesystem and the deployed applicative content. With Sitecore, this is only possible by

  • Restoring SQL backups for the core, master and web database – this causes downtime
  • Deploying a new Sitecore package – simply restoring the old package will probably not cut it

With Unicorn, the “old” situation is still available in the deployment slot (after swapping). By simply syncing that content and swapping the application back to production, the roll back is executed – no deployment of old web packages is involved in this process.

Conclusion

It’s not too hard to set up a small environment to deploy without downtime, and have a very convenient rollback scenario without downtime. However, there are some drawbacks, which can be overcome, but some work is needed

  • Sitecore offers ARM templates to provision Sitecore with an initial deployment
  • The web deploy packages that are offered by Sitecore cannot be used for redeployment, as they recreate the Sitecore databases. The parameters provided also require the use of parameters that should be provided once.
    • This leads to “initial deployment” or “provisioning” web deploy packages and “deployment” web deploy packages. This leads to extra work when altering baselines and: custom archive.xml’s, parameter.xml’s and configuration files.

Apart from the drawbacks, I am very enthusiastic on the possibilities for Sitecore on Azure: convenient ALM, scaling, zero downtime. This is definitely going to help towards a continuous deployment scenario. Happy Azureing (is that a word? ;))

PS: Special thanks to Rob Habraken, Christof Claessens, Pete Navarra, Sean Holmesby and George Chang for their reviews

Viewing all 55 articles
Browse latest View live