Okay, even with the number of diagrams in my last post, there was still some confusion. So, I’m going to try to make it totally clear. Emphasis on “try”.
Regression Testing isn’t just about regression testing one part of a solution
Firstly, I got some feedback from SAP Support when I tried to raise this issue with them. They didn’t seem to understand my concern that I tested the whole recruit to hire to fire employee life cycle every time that SAP released new functionality for SuccessFactors in the preview environment before it got released to the productive environment. This was because:
“Since IAS productive and test instances are of the same version, there is from IdP perspective no difference at all.“
“As already mentioned above: IAS test- and productive instances are of the same release. There is no such thing as a preview environment for IAS. Thus again one single IAS tenant will be fully sufficient to handle the described scenarios from IdP perspective“
I’m not sure how I can make this clearer. But, yes, I do want to test that during the preview release update period the changes that SuccessFactors make do not impact the provisioning and login processes that I have configured in IPS/IAS. I also want to ensure as well as ensure that these are working in the non-prod environment so that if I push up a change to fix a release impact, it doesn’t break stuff. I’m confused how the Identity services teams seem to think their solutions works in isolation! If it isn’t for the systems that use their services and provide the user details the solutions aren’t worth anything!
How does this all look if we try to connect it up?
Okay – here’s a relatively complex diagram. It shows how you’d have to wire up the provisioning and authentication trust when using more that one SuccessFactors system with a single IAS.
But there is the slim possibility it could work!
And it could work provided that you:
Did some configuration in the IPS and IAS that SuccessFactors has not at all been documented for customers
that populated userid into different custom attributes of the IAS user record per system
then used that system specific field in the assertion sent to the different systems.
AND – never maintained different employee attributes for the same email address in the two systems (if you don’t want the system to get hella confused.)
Unfortunately that’s about as likely as me winning the Powerball lottery. Whilst the first point is terribly technical and pedantic (therefore will be loved by half the people I know and hated by the other half) the second point pretty much means this is never going to happen. The reason for the second restriction is that you cannot have the same email address against multiple “user” records in IAS. Whilst through some technical wizardry I might get the same record to point to two different system ids, whether it is active or inactive is a non-system specific piece of user data. What name that user has is a non-system specific piece of user data.
So assume I do have the same email address assigned to an employee in both systems A and B. Terminating that employee in system A will cause a delta in the employee record that will get picked up by the IPS and deactivate the user in the IAS. Even though they should be active to be allowed to log in to system B, they won’t be able to as they are now “inactive”.
It is possible that customers will just decide that they will ensure consistency between user records in different systems, use same name, have both active, have both inactive, but I very much doubt it.
Likewise not using the same email address for employees in both systems is going to be hard (not to mention hard to track if it mistakenly happens).
Would be nice, but unlikely
In my previous post, I assumed that the previous scenario (especially as it involves undocumented configuration) would not be customers’ default. There I assumed that customers would use the default configuration that is deployed when a customer implements the SuccessFactors IAS “upgrade”. That then allows for all sorts of mischief!
How to abuse authentication when you control the data it is based upon.
Let’s have some fun. Assume that default IPS configuration is being used and the employee records from systems A and B are both trying to update the IAS user master record. Assume system B contains sensitive payroll data (for example it is a copy of productive system). Only Annie Admin has the roles in system B to see this data. I, Chris Creative, have access to system A where I’m doing some project work. I have a role where I can hire and fire employees (tends to happen in HR systems!)
I firstly terminate an employee with Annie Admin’s email address in system A. If there isn’t one, I’ll just hire her then term her.
This will trigger the IPS to update the IAS with the user with Annie’s email address as inactive so Annie can’t log into either system and stop my nefarious fun.
Then I hire a new employee that has the same personnel number as Annie had in system B and put a newly generated email address (that I have access to) against the employee’s record
I get emailed by the IAS that I have a new user record set up 🙂 what a cool system! I set/reset the password.
Now I can use this email address to access system B and it thinks I’m logging in as Annie! Which is awesome as when they try to trace who downloaded all the payroll details from the system the audit logs are going to clearly point to Annie’s user! (Okay yes a bit of further checking and someone might see that it was my IP address, but I can hide that using a VPN, the audit logging in IAS is hard to get to (API access only) so going to be hard for them to find the random gmail address I made up and trace it.
I access a bunch of data I’m not supposed to
I go back to system A, change the email address of the fake Annie back to her email.
She gets reactivated
I just got away with accessing a system that I had no rights to access because I had rights to another system that was provisioning the same authentication system.
Are you worried yet?
Can you see now why I’m a little worried about this setup? This is why I also assumed that to prevent this sort of thing happening, most organisations if forced down the path of using just one IAS will choose the most secure SuccessFactors system to provision user data to the IAS. But that, as I wrote about in the previous post will cause a bunch of other problems.
In case it’s not obvious enough, there is a simple solution. Provide another IAS instance per SuccessFactors instance that a customer has.
So, firstly, at this moment in time it feels a little petty writing about this. The world is just starting to realise how huge a problematic space we are in with COVID-19. So I wish all of you out there reading this, health and the wisdom to look after yourselves and others. So, now back to me being needy.
SAP SuccessFactors are only offering customers one non-prod IAS system. One non-prod IAS is not enough. Read more to understand what that is and why I think this is a mistake.
A new service to support new functionality
Recently SAP SuccessFactors announced that they are going to move all customers to a new platform authentication solution which uses SAP Cloud Identity Authentication*. This new background service is a requirement to use the new People Analytics that use SAP Analytics Cloud and is also required to use new internal facing Career Site builder functionality. In short, to take advantage of some of the newest and coolest looking features of SuccessFactors, you’re going to need to implement this!
*There’s a handy overview video (strangely enough not behind SuccessFactors community wall) that explains in detail what SAP is doing, it’s a little more in depth than the detail I provide in this post, it’s not bad although very detailed.
Why are SAP SuccessFactors doing this?
Honestly, this is a good move. Authentication is a common problem in all cloud based solutions, so why not have a single service that can be leveraged by all SAP solutions to solve this? Customers get the benefit of multiple development teams within SAP all pooling to produce a better product, SAP gets the benefit of reducing cost of having to maintain and upgrade their authentication solutions for each cloud product. Win – Win!
It’s not yet a requirement for customers to migrate to this solution (and yes for many customers this will require some work to implement), but in order to take advantage of several new SAP SuccessFactors functionalities, customers are going to have to move. So I’m pretty sure that SAP will get close to its goal of migrating by end of year (although possibly with a covid related bump in that progress).
“Our goal is to have customers migrated to SAP Cloud Identity Authentication (IAS and IPS) by the end of 2020. This is not a “forced” migration, we are just encouraging customers to migrate. At some point all customers will need to be using this authentication method, that date is not yet determined. “
from SAP SuccessFactors Community, Platform Resources Blog
However, in going through the process to migrate some of my own systems, I’ve discovered a bit of a problem. I’ve tried to raise this multiple times, but so far, I just don’t seem to be able to clearly articulate why this problem is so important. So, I thought I’d try putting together a simple blog post with LOTS of pictures. Hopefully more pictures means easier to understand and then we can get somewhere!
Background – How does it work?
Okay, before diving into the deep end, it’s probably worth trying to articulate what on earth this SAP Cloud Identity Authentication is, and why I’m so happy that it’s coming.
So, here’s how it works in general.
Happy SuccessFactors user uses their computer to
Access SuccessFactors website, which
Redirects them to their own company’s identity provider (IDP) which asks them to log on (or possibly realises they are already signed in and does single sign on. Which then
Sends them back to SuccessFactors having verified that they are a valid user and they can access their SuccessFactors system, so
The Ecstatically Happy SuccessFactors user is even more happy!
Pretty simple really! There’s some technical stuff like SAML2 assertions and stuff that happens – but generally the experience and security are what matters.
Importantly, SAP SuccessFactors aren’t really aiming to change any of that experience, there just a small technical difference. In between steps 3 and 4 there’s going to be a little more going on under the hood.
Steps 3 and 4 of our previous diagram are replaced with 4 other steps. Quite simply SAP Cloud Identity Authentication Service (IAS) becomes a middleman to/from SuccessFactors and the Corporate IDP.
SuccessFactors sends unauthenticated requests to IAS.
IAS redirects request to IDP for authentication.
IDP tells IAS that user is identified and correctly authenticated
IAS tells SuccessFactors that user is identified and correctly authenticated.
There are a few benefits here. The SAP Cloud IAS is set up to be maintainable by customers. Whereas setting the details of a corporate IDP into SuccessFactors was/is restricted to SAP and certified partners, using the IAS is something that most customer’s technical teams should be able to manage. (I’ll caveat that with note that I expect that most customers will probably get their implementation partner to guide them through the initial setup. But at least it is something that customers can do, if they want to.) SAP SuccessFactors automagically set up the link between SAP SuccessFactors and the IAS.
There are many options to be able to configure the IAS to do some nice things, setting up rules for access, making login screens pretty, etc. It’s a good tool.
It is important, however, to understand that the IAS isn’t just a simple pass-through proxy service for authentication, it stores a list of all active users. A user MUST exist in the IAS in order to be able to log on to SuccessFactors.
So, what’s the problem?
Well so far everything seems hunky dory, no? Well, that’s probably because I’ve just talked about the productive use of the solution (which to be fair is what most people care about.) However, for many customer non-productive environments (and even some productive ones!) a corporate IDP and single sign on is not used.
Logs on with username/password (stored in SuccessFactors)
This scenario is updated also with IAS in the picture.
Can we just assume that you’re as tired of reading numbered lists as I am of typing them? No? Okay – just to be complete:
User bounces to their laptop…
Uses browser to access SuccessFactors
Which redirects them to IAS which asks them for username and password (as no corporate IDP configured)
IAS tells SuccessFactors that user successfully identified and authenticated.
User has great time using SuccessFactors without SSO or a corporate IDP
There is some great functionality that can be leveraged here too! SAP Cloud IAS can implement multi-factor authentication – pretty damn good to have that available without having to use corporate IDP!
Sidebar – there is an IPS too
Yes, sorry more terminology! The way that the IAS works it needs to have a list of users provisioned into it. It doesn’t just use the list of employees that are in SuccessFactors, these need to be populated into the IAS.
The IPS reads SuccessFactors regularly to find any changes to employee/user records
It passes any details if finds to the IAS which updates its list of active users for SuccessFactors (and potentially any other system that authenticates against it).
Now that might sound like a retrograde step – it means that we need to provision (yep IPS stands for Identity Provisioning Service) users from SuccessFactors into IAS on a very regular basis. But, realistically, there are very few productive use scenarios that require a new employee to have access to the SuccessFactors system instantaneously, a delay of an hour is normally manageable. And there are ways to make that sync happen faster if needed.
Again, this seems all to be good news! There are a few minor niggles. For instance, the IAS must keep a list of usernames and email addresses in order to allow it to function. One additional restriction is that all users must have unique email addresses (you can’t set all the non-used users’ emails to firstname.lastname@example.org!) Again this doesn’t seem to be much of a problem. Until one considers that many customers have more than one non-productive system.
There can be only one!
I was so tempted to put a Highlander meme in there… But I didn’t because it might distract from the seriousness of this next bit – which seems to be the official SAP position at the moment.
” SAP is offering free [IAS] licenses to SuccessFactors customers for the purpose of logging-in to SF; they will receive one production and one test instance (by region). “
SAP SuccessFactors IAS FAQ, highlighting and info in [brackets] added by me
SAP will only provide customers with one IAS tenant for their productive environment and one for their non-productive environments.
Why is this a concern?
This seems to be the bit that I’m having trouble explaining to people, so I hope the following diagrams and explanations help.
For many SuccessFactors Employee Central (Core HR) customers they have been provisioned by default with three instances. Customers can use them however they like, but I tend to advise people to do the below:
Note that these systems are very often linked to payroll environments that reflect similar levels of detail to the productive environments. As such, the levels of control of access are very important. This is personal and private information and needs care and attention!
However, very often one of the systems will have anonymised data, or even just made up data, and be used for prototyping and building solutions. Generally in this system users may be given access to areas that they don’t normally have access to. Training and/or testing may take place. The key thing is that users may be running scenarios with different access levels than normal.
It’s important to note that whilst there is clearly an overlap in user records between all three systems, there are plenty of users in both non-productive environments that don’t overlap in normal usage of these systems.
Again, so why is this concerning?
Well, to reiterate, SAP have said they will only provide as part of customers’ existing subscriptions, two IAS instances. One for productive use and one for non-productive/test environments.
The non-productive IAS must provide authentication services for both non-prod environments. However, the feed that populates the list of users into the IAS can realistically only come from one of the two environments.
I’d better make QAS my source of user data then
Due to security concerns for data access, I must then make the source of user data for the IAS the QAS system, where I have locked down user access. Otherwise, if the source of the user data was the test system, then a test user could update a user in the test system records to have details that could allow themselves to log on to the QAS environment and see data that they were not supposed to see. That can’t be allowed!
Real email addresses at a premium for testing, double handling.
It will be quite some hassle to maintain all the Test system users in the QAS system, but it could be done. However, it may be very difficult to test scenarios that rely on real email addresses. Because the email address must be unique in the IAS, it won’t be possible to swap emails around easily. This becomes especially problematic when email address is used to authenticate users to downstream systems (particular cases where I have seen this occur are SSO into SAP ERP systems, where email address is used as the unique identifier.) Changing an email address to allow testing of a process may mean creating a new employee in the QAS system and reassigning email addresses there in addition to updating the test system.
But hold on, what about regression testing?
However, during the 6 monthly release cycle, there is a need to test all the new SAP SuccessFactors functionality. This needs to be done in the preview environment. So perhaps during this period I lock down all access to the preview environment, make user access as per the QAS system? Then I switch my IAS source to be the preview environment? This has some problems associated with it too. It really restricts how I can carry out my regression testing and who I can use to do it. But sure as a very sure thing, I will want to test out user provisioning and definitely want to check that the configuration I’m using to populate the user list into my IAS from SuccessFactors hasn’t been impacted by the half yearly release.
So where does this leave us?
Very soon, with the release of SuccessFactors embedded analytics (People Analytics), more and more customers are going to want/need to implement SAP Cloud IAS within their SuccessFactors environments. I would imagine that due to the overheads that I described that many customers would opt to purchase an additional subscription for another non-productive IAS instance. If I think of the cost/overhead of:
maintaining all non-productive users in pre-prod (imagine having to hire someone in the pre-prod environment just so you could test a hire in the test environment!)
Separating out test system users/data from pre-prod environment to ensure regression testing payroll changes.
Migrating IAS and IPS (tool used to synchronise user record to IAS) configuration from one system to another to enable release testing, then flipping back if urgent production support testing needs to occur.
Restricting release regression testing to only those users that have pre-production access and only to their regular roles.
Then I think I can see the cost benefit of purchasing an additional IAS system to handle that. But I really don’t want to be forced into paying additional subscription to handle a scenario that works just fine right now.
Given that when I raised this point in SuccessFactors community site I was told that:
…By design, you don’t need a 1 to 1 relationship between SF instances/tenants and IAS.
IAS is designed like any other Identity Management product to handle logins for many different systems. Customers don’t buy a new copy of Ping Identity or Microsoft Azure for every application they use it for. Your IAS configuration will control which SF instance users log into.
Additionally, having the 1:many approach makes it a lot easier to manager [sic]. If you had multiple copies of IAS/IPS you would need to figure what is connect to what every time you want to manage anything. When you re-use the same IAS for many applications, you only have to configure one time and then re-use them. ie…Password policy settings, Corporate Identity Provider connections etc. The corporate IDP is a huge one since you have to work with your internal SSO folks any time you change anything there.
I think there is a disconnect between how some SuccessFactors product managers think customers are using their product and how it is being used. Many customers do not use SSO into their non-productive environment by design!
I really hope that by spending the time to put this post together I can raise some awareness of this problem before it becomes a bigger issue for customers. Ideally, I’d love SAP SuccessFactors to re-evaluate their stance on providing only one IAS instance for all non-productive SuccessFactors instances. It should clearly be (in my not so humble opinion) one IAS per SuccessFactors instance. (on a technical note, happy for just one non-prod IPS). Let’s see. If you’re reading this and you have some influence with SAP SuccessFactors it would be great if you let me know what you think and perhaps let others know too.
Last thought on solution parity
Finally let me leave you with one last diagram/thought…
I love working with the SAP SuccessFactors software and I think that the IAS is some great functionality. However, no new functionality that replaces existing functionality that a customer has should ever require them to purchase additional subscription to retain parity with their existing solution.