It was with a fair bit of sadness that this week I was part of the end of something that I think was a bit special.
The SAP SuccessFactors IAS/IPS open office hour has been running for what I think was 2 years now, and for many of those Thursday evenings (it ran at 6:30pm to 7:30pm my local time) I would join, sometimes with a beer or wine in hand and would sit, listen and contribute.
I think Richard Feynman the American Nobel Laureate and Physic Professor once said :
“If you want to master something, teach it. The more you teach, the better you learn.”
Richard P. Feynman
Well, I think that’s what happened to me! By spending so much time in this call, trying to help other people figure out how to solve their IAS and IPS issues, I’ve learnt so much! I’m waiting for an IAS/IPS certification to come out and nice easy one to pass!
During this journey, I’ve developed some great relationships with SAP teams in this space. I’ve presented at SAP TechEd on the topic, had multiple influence requests turned into reality and been involved in beta testing new functionality. And more importantly, made some good friends.
There is still so much more to learn, but this opportunity to gather weekly to help out other SAP customers that were stuck in a tight spot is now over. It probably had run its time and it was time to focus the support efforts elsewhere, but whilst it was there I really did enjoy it.
Right, this feels kinda awkward… I’m about to give Microsoft kudos and point out how I wish that some SAP processes were closer to what I’ve seen from the team from Microsoft. So bear with me if I seem a little less hyperboly than regularly…
This isn’t the Microsoft you remember
Recently I was working through the options for integrating SAP SuccessFactors personnel records into Microsoft AD, it’s something that every organisation that doesn’t have a dedicated IAM or (IdAM, however you want to make up your TLAs or FLAs) is likely to need in their environment. Have to say, I love working with new “start-up” orgs that don’t use an on-prem AD, but there’s not quite so many of those that are large enough to pick up SuccessFactors that they are probably still a minority.
Documentation is a skill that is distinct from development
Anyway, I happened to look at the Azure AD online doco about SuccessFactors integration and discovered it had been written by a developer. Well, that’s a guess, but seriously, who digs through the results of an API call to get config values out of a system when you can just use the standard tooling to do it? And then makes some poor sod document how to use Postman to do the same? So I suggested an update.
no – just use the UI!
So, I was feeling benevolent and thought I’d offer my advice that perhaps there was a better way. I clicked the feedback button…
Shock horror – I wasn’t redirected to another site and asked to create a new user, I was asked to create a Github issue! (Okay if you don’t have a Github user, you’ll be asked to create one, but seriously, you don’t have a Github user id?)
Then I resigned myself to never hearing back about it again… But I did!
Issue was triaged and assigned to the document author to review that very same day! (that’s not normal is it?)
I was – wow!
Then things got surreal…
Not only did I have someone look at and action the feedback that I gave, they then went and found my tweet on the subject and personally responded to it! Wat?
And now, the update to the documentation is about to go live:
And hopefully that will make some poor consultant/tech support person’s life a little bit easier.
Meanwhile, back at the ranch
So let’s compare and contrast. And I know this isn’t apples – doco is different to application UI changes, but, lets compare the process at least.
I was working on the new SAP SuccessFactors IAS/IPS integration on my own company’s system when had an issue – I couldn’t figure out how to change some value in the config. Fortunately there is a partner community that SAP have set up for partners to discuss these sorts things and get some assistance from each other.
(Sidebar – Yes, I know it’s a bizarre idea, consultants helping our competing firms consultants do stuff. But in the scheme of things, the other consultants are all good people, they just aren’t lucky enough to work for my company, and helping others tends to do pretty good things for your own internal skills too.)
If you don’t know, then ask!
So I raised the issue in the forum and the really nice SAP person how has to read all my grumbles and moderate the forum raised it in the fortnightly call that SAP hosts for partners (it’s at 12:30am my local time, which makes it a bit fun, but better that than 6am!)
If you have to attend training to do something, it isn’t intuitive.
Let’s just say I wasn’t impressed with the UX and I realised why I hadn’t figured out how to do this myself! Because colouring something BLUE in an SAP UI5 app is possibly the least intuitive thing on planet to do to indicate that it is editable if you click it. Possibly the developers had played one too many games of Day of the Tentacle and thought users needing to randomly waving their mouse around the screen to see if it changes pointer shape is a good way to indicate to people that something is clickable? (Okay I doubt that was actually the case, more likely someone threw a guideline at them that didn’t make sense and they had to get inventive to work around it (been there!)). Pretty much everything in standard SAP UI5 apps are cyan or blue, and I’m not checking everyone one of them to see if it’s different.
So I gave some feedback on the forum.
I even tweeted about it. Cause that’s what you do, right? (Well it’s what I do. I mean there’s a certain type of person who stays up late at night writing blog posts about these sort of things, so what do you expect?)
This then lead to a bit of a conversation in my DM’s with someone from SAP (since it was DM’s I’ll not share, private stays private) who suggested that I really needed to raise this with support since it was an issue, and that helps track that people have issues. Likewise in the forums I was directed to raise it formally.
To whom it may concern
So I raised a SAP Support ticket (low priority since I already had a fully working work-around.)
I would happily have bet on the response, and I’d have won!
Thanks Mike – yep, the ole “Raise an enhancement request” gambit. That place where good ideas go to die.
“Once more unto the breach, dear friends, once more; Or close the wall up with our English dead”
But by this time I was, “right whatever, lets see how far this sucker goes!” So I raised that enhancement request.
Oh – and whilst I was doing that, I came across a small issue…
The feedback site is hosted in Europe. I am not in Europe. But that’s cool because there’s this concept called CDNs, yeah, that allow large websites that are accessed around the world to be accessed in a reasonably fast manner from everywhere.
Yup – CDN wasn’t enabled. It is now – so the rest of you can thank me for suffering on your behalf!
Sod it though I’m gonna get this bugger filed! Oooh flashy light on my phone…
Anyway after much self flagellation
I got the request raised! had to attach my diagram as an “attachment” not able to be viewed inline in the request – but hey – it was raised.
And there it sat…
Three weeks later my request was “Acknowledged”.
What’s the German word for “million to one chances happen constantly”?
And in the weird way that the universe works, whilst I have been typing this up, I got a comment on the request, strangely I didn’t get a notification (yet) but keeping my fingers crossed that sometime tonight. I did just check my spam email folder too, and interesting that it’s about 30/70 banking phishing scams and webinar invites, sure it used to be far more interesting. But nothing to notify me that my request had an update.
The really nice lead designer for the product reached out and asked me what I thought about their thoughts about making some UI changes to make things easier to use!
The response was awesome! I Loved it!
they ended the message with a request for my thoughts!
“Please let us know what you think.”
YES, YES, YES!
Well – I can say I was totally stoked, so happy! And then I tried to find the button to reply….
The irony of wanting to reply to a conversation about improving UI to make things more obvious and easier for people to use and then not being able to because the UI of the tool in which the conversation is happening doesn’t facilitate it.
Anyway. I did what I always do. Tweet lots, then try to figure out what to do…
It would appear that someone thought that it would make more sense for new comments to appear at the top of the conversation, not the bottom. So by clicking on the comments “tab” at the top of the page I was navigated up the screen and saw that I could enter a new comment. I did. And I tried to be very nice in my feedback (given the amount of huffing and puffing I’d been doing seconds before.)
Two ways of doing things, both with good result
So, we have two different scenarios, both ended up (or will hopefully end up with) some change in the product as influenced and suggested by me. Two out of two is pretty good going. However, one took 3 weeks, the other, over 3 months. One was painless and easy, the other painful and frustrating. As I said earlier, we’re not comparing apples to apples – getting a product changed is much harder than getting some doco changed. And I have heard anecdotally that some areas of SAP are even faster:
Interesting to note that the area that got the same day response that Robin mentions is also using the Microsoft Github tooling to manage issues. I wonder if tooling impacts delivery approach?
Yes – AND?
So what do I want to achieve by writing all this (other than hopefully amusing a few of you with the tale)? Well, I think it’s important that it’s documented how difficult giving good and constructive feedback can be. Only by taking a look at what’s happening can we get on the right path to working together to make everything we do better and easier.
I’ll finish by just mentioning that EVERYONE that I have dealt with when providing feedback at both SAP and Microsoft have been AWESOME. Both organisations do understand and value feedback. It’s not a people problem.
Okay, even with the number of diagrams in my last post, there was still some confusion. So, I’m going to try to make it totally clear. Emphasis on “try”.
Regression Testing isn’t just about regression testing one part of a solution
Firstly, I got some feedback from SAP Support when I tried to raise this issue with them. They didn’t seem to understand my concern that I tested the whole recruit to hire to fire employee life cycle every time that SAP released new functionality for SuccessFactors in the preview environment before it got released to the productive environment. This was because:
“Since IAS productive and test instances are of the same version, there is from IdP perspective no difference at all.“
“As already mentioned above: IAS test- and productive instances are of the same release. There is no such thing as a preview environment for IAS. Thus again one single IAS tenant will be fully sufficient to handle the described scenarios from IdP perspective“
I’m not sure how I can make this clearer. But, yes, I do want to test that during the preview release update period the changes that SuccessFactors make do not impact the provisioning and login processes that I have configured in IPS/IAS. I also want to ensure as well as ensure that these are working in the non-prod environment so that if I push up a change to fix a release impact, it doesn’t break stuff. I’m confused how the Identity services teams seem to think their solutions works in isolation! If it isn’t for the systems that use their services and provide the user details the solutions aren’t worth anything!
How does this all look if we try to connect it up?
Okay – here’s a relatively complex diagram. It shows how you’d have to wire up the provisioning and authentication trust when using more that one SuccessFactors system with a single IAS.
But there is the slim possibility it could work!
And it could work provided that you:
Did some configuration in the IPS and IAS that SuccessFactors has not at all been documented for customers
that populated userid into different custom attributes of the IAS user record per system
then used that system specific field in the assertion sent to the different systems.
AND – never maintained different employee attributes for the same email address in the two systems (if you don’t want the system to get hella confused.)
Unfortunately that’s about as likely as me winning the Powerball lottery. Whilst the first point is terribly technical and pedantic (therefore will be loved by half the people I know and hated by the other half) the second point pretty much means this is never going to happen. The reason for the second restriction is that you cannot have the same email address against multiple “user” records in IAS. Whilst through some technical wizardry I might get the same record to point to two different system ids, whether it is active or inactive is a non-system specific piece of user data. What name that user has is a non-system specific piece of user data.
So assume I do have the same email address assigned to an employee in both systems A and B. Terminating that employee in system A will cause a delta in the employee record that will get picked up by the IPS and deactivate the user in the IAS. Even though they should be active to be allowed to log in to system B, they won’t be able to as they are now “inactive”.
It is possible that customers will just decide that they will ensure consistency between user records in different systems, use same name, have both active, have both inactive, but I very much doubt it.
Likewise not using the same email address for employees in both systems is going to be hard (not to mention hard to track if it mistakenly happens).
Would be nice, but unlikely
In my previous post, I assumed that the previous scenario (especially as it involves undocumented configuration) would not be customers’ default. There I assumed that customers would use the default configuration that is deployed when a customer implements the SuccessFactors IAS “upgrade”. That then allows for all sorts of mischief!
How to abuse authentication when you control the data it is based upon.
Let’s have some fun. Assume that default IPS configuration is being used and the employee records from systems A and B are both trying to update the IAS user master record. Assume system B contains sensitive payroll data (for example it is a copy of productive system). Only Annie Admin has the roles in system B to see this data. I, Chris Creative, have access to system A where I’m doing some project work. I have a role where I can hire and fire employees (tends to happen in HR systems!)
I firstly terminate an employee with Annie Admin’s email address in system A. If there isn’t one, I’ll just hire her then term her.
This will trigger the IPS to update the IAS with the user with Annie’s email address as inactive so Annie can’t log into either system and stop my nefarious fun.
Then I hire a new employee that has the same personnel number as Annie had in system B and put a newly generated email address (that I have access to) against the employee’s record
I get emailed by the IAS that I have a new user record set up 🙂 what a cool system! I set/reset the password.
Now I can use this email address to access system B and it thinks I’m logging in as Annie! Which is awesome as when they try to trace who downloaded all the payroll details from the system the audit logs are going to clearly point to Annie’s user! (Okay yes a bit of further checking and someone might see that it was my IP address, but I can hide that using a VPN, the audit logging in IAS is hard to get to (API access only) so going to be hard for them to find the random gmail address I made up and trace it.
I access a bunch of data I’m not supposed to
I go back to system A, change the email address of the fake Annie back to her email.
She gets reactivated
I just got away with accessing a system that I had no rights to access because I had rights to another system that was provisioning the same authentication system.
Are you worried yet?
Can you see now why I’m a little worried about this setup? This is why I also assumed that to prevent this sort of thing happening, most organisations if forced down the path of using just one IAS will choose the most secure SuccessFactors system to provision user data to the IAS. But that, as I wrote about in the previous post will cause a bunch of other problems.
In case it’s not obvious enough, there is a simple solution. Provide another IAS instance per SuccessFactors instance that a customer has.
So, firstly, at this moment in time it feels a little petty writing about this. The world is just starting to realise how huge a problematic space we are in with COVID-19. So I wish all of you out there reading this, health and the wisdom to look after yourselves and others. So, now back to me being needy.
SAP SuccessFactors are only offering customers one non-prod IAS system. One non-prod IAS is not enough. Read more to understand what that is and why I think this is a mistake.
A new service to support new functionality
Recently SAP SuccessFactors announced that they are going to move all customers to a new platform authentication solution which uses SAP Cloud Identity Authentication*. This new background service is a requirement to use the new People Analytics that use SAP Analytics Cloud and is also required to use new internal facing Career Site builder functionality. In short, to take advantage of some of the newest and coolest looking features of SuccessFactors, you’re going to need to implement this!
*There’s a handy overview video (strangely enough not behind SuccessFactors community wall) that explains in detail what SAP is doing, it’s a little more in depth than the detail I provide in this post, it’s not bad although very detailed.
Why are SAP SuccessFactors doing this?
Honestly, this is a good move. Authentication is a common problem in all cloud based solutions, so why not have a single service that can be leveraged by all SAP solutions to solve this? Customers get the benefit of multiple development teams within SAP all pooling to produce a better product, SAP gets the benefit of reducing cost of having to maintain and upgrade their authentication solutions for each cloud product. Win – Win!
It’s not yet a requirement for customers to migrate to this solution (and yes for many customers this will require some work to implement), but in order to take advantage of several new SAP SuccessFactors functionalities, customers are going to have to move. So I’m pretty sure that SAP will get close to its goal of migrating by end of year (although possibly with a covid related bump in that progress).
“Our goal is to have customers migrated to SAP Cloud Identity Authentication (IAS and IPS) by the end of 2020. This is not a “forced” migration, we are just encouraging customers to migrate. At some point all customers will need to be using this authentication method, that date is not yet determined. “
from SAP SuccessFactors Community, Platform Resources Blog
However, in going through the process to migrate some of my own systems, I’ve discovered a bit of a problem. I’ve tried to raise this multiple times, but so far, I just don’t seem to be able to clearly articulate why this problem is so important. So, I thought I’d try putting together a simple blog post with LOTS of pictures. Hopefully more pictures means easier to understand and then we can get somewhere!
Background – How does it work?
Okay, before diving into the deep end, it’s probably worth trying to articulate what on earth this SAP Cloud Identity Authentication is, and why I’m so happy that it’s coming.
So, here’s how it works in general.
Happy SuccessFactors user uses their computer to
Access SuccessFactors website, which
Redirects them to their own company’s identity provider (IDP) which asks them to log on (or possibly realises they are already signed in and does single sign on. Which then
Sends them back to SuccessFactors having verified that they are a valid user and they can access their SuccessFactors system, so
The Ecstatically Happy SuccessFactors user is even more happy!
Pretty simple really! There’s some technical stuff like SAML2 assertions and stuff that happens – but generally the experience and security are what matters.
Importantly, SAP SuccessFactors aren’t really aiming to change any of that experience, there just a small technical difference. In between steps 3 and 4 there’s going to be a little more going on under the hood.
Steps 3 and 4 of our previous diagram are replaced with 4 other steps. Quite simply SAP Cloud Identity Authentication Service (IAS) becomes a middleman to/from SuccessFactors and the Corporate IDP.
SuccessFactors sends unauthenticated requests to IAS.
IAS redirects request to IDP for authentication.
IDP tells IAS that user is identified and correctly authenticated
IAS tells SuccessFactors that user is identified and correctly authenticated.
There are a few benefits here. The SAP Cloud IAS is set up to be maintainable by customers. Whereas setting the details of a corporate IDP into SuccessFactors was/is restricted to SAP and certified partners, using the IAS is something that most customer’s technical teams should be able to manage. (I’ll caveat that with note that I expect that most customers will probably get their implementation partner to guide them through the initial setup. But at least it is something that customers can do, if they want to.) SAP SuccessFactors automagically set up the link between SAP SuccessFactors and the IAS.
There are many options to be able to configure the IAS to do some nice things, setting up rules for access, making login screens pretty, etc. It’s a good tool.
It is important, however, to understand that the IAS isn’t just a simple pass-through proxy service for authentication, it stores a list of all active users. A user MUST exist in the IAS in order to be able to log on to SuccessFactors.
So, what’s the problem?
Well so far everything seems hunky dory, no? Well, that’s probably because I’ve just talked about the productive use of the solution (which to be fair is what most people care about.) However, for many customer non-productive environments (and even some productive ones!) a corporate IDP and single sign on is not used.
Logs on with username/password (stored in SuccessFactors)
This scenario is updated also with IAS in the picture.
Can we just assume that you’re as tired of reading numbered lists as I am of typing them? No? Okay – just to be complete:
User bounces to their laptop…
Uses browser to access SuccessFactors
Which redirects them to IAS which asks them for username and password (as no corporate IDP configured)
IAS tells SuccessFactors that user successfully identified and authenticated.
User has great time using SuccessFactors without SSO or a corporate IDP
There is some great functionality that can be leveraged here too! SAP Cloud IAS can implement multi-factor authentication – pretty damn good to have that available without having to use corporate IDP!
Sidebar – there is an IPS too
Yes, sorry more terminology! The way that the IAS works it needs to have a list of users provisioned into it. It doesn’t just use the list of employees that are in SuccessFactors, these need to be populated into the IAS.
The IPS reads SuccessFactors regularly to find any changes to employee/user records
It passes any details if finds to the IAS which updates its list of active users for SuccessFactors (and potentially any other system that authenticates against it).
Now that might sound like a retrograde step – it means that we need to provision (yep IPS stands for Identity Provisioning Service) users from SuccessFactors into IAS on a very regular basis. But, realistically, there are very few productive use scenarios that require a new employee to have access to the SuccessFactors system instantaneously, a delay of an hour is normally manageable. And there are ways to make that sync happen faster if needed.
Again, this seems all to be good news! There are a few minor niggles. For instance, the IAS must keep a list of usernames and email addresses in order to allow it to function. One additional restriction is that all users must have unique email addresses (you can’t set all the non-used users’ emails to firstname.lastname@example.org!) Again this doesn’t seem to be much of a problem. Until one considers that many customers have more than one non-productive system.
There can be only one!
I was so tempted to put a Highlander meme in there… But I didn’t because it might distract from the seriousness of this next bit – which seems to be the official SAP position at the moment.
” SAP is offering free [IAS] licenses to SuccessFactors customers for the purpose of logging-in to SF; they will receive one production and one test instance (by region). “
SAP SuccessFactors IAS FAQ, highlighting and info in [brackets] added by me
SAP will only provide customers with one IAS tenant for their productive environment and one for their non-productive environments.
Why is this a concern?
This seems to be the bit that I’m having trouble explaining to people, so I hope the following diagrams and explanations help.
For many SuccessFactors Employee Central (Core HR) customers they have been provisioned by default with three instances. Customers can use them however they like, but I tend to advise people to do the below:
Note that these systems are very often linked to payroll environments that reflect similar levels of detail to the productive environments. As such, the levels of control of access are very important. This is personal and private information and needs care and attention!
However, very often one of the systems will have anonymised data, or even just made up data, and be used for prototyping and building solutions. Generally in this system users may be given access to areas that they don’t normally have access to. Training and/or testing may take place. The key thing is that users may be running scenarios with different access levels than normal.
It’s important to note that whilst there is clearly an overlap in user records between all three systems, there are plenty of users in both non-productive environments that don’t overlap in normal usage of these systems.
Again, so why is this concerning?
Well, to reiterate, SAP have said they will only provide as part of customers’ existing subscriptions, two IAS instances. One for productive use and one for non-productive/test environments.
The non-productive IAS must provide authentication services for both non-prod environments. However, the feed that populates the list of users into the IAS can realistically only come from one of the two environments.
I’d better make QAS my source of user data then
Due to security concerns for data access, I must then make the source of user data for the IAS the QAS system, where I have locked down user access. Otherwise, if the source of the user data was the test system, then a test user could update a user in the test system records to have details that could allow themselves to log on to the QAS environment and see data that they were not supposed to see. That can’t be allowed!
Real email addresses at a premium for testing, double handling.
It will be quite some hassle to maintain all the Test system users in the QAS system, but it could be done. However, it may be very difficult to test scenarios that rely on real email addresses. Because the email address must be unique in the IAS, it won’t be possible to swap emails around easily. This becomes especially problematic when email address is used to authenticate users to downstream systems (particular cases where I have seen this occur are SSO into SAP ERP systems, where email address is used as the unique identifier.) Changing an email address to allow testing of a process may mean creating a new employee in the QAS system and reassigning email addresses there in addition to updating the test system.
But hold on, what about regression testing?
However, during the 6 monthly release cycle, there is a need to test all the new SAP SuccessFactors functionality. This needs to be done in the preview environment. So perhaps during this period I lock down all access to the preview environment, make user access as per the QAS system? Then I switch my IAS source to be the preview environment? This has some problems associated with it too. It really restricts how I can carry out my regression testing and who I can use to do it. But sure as a very sure thing, I will want to test out user provisioning and definitely want to check that the configuration I’m using to populate the user list into my IAS from SuccessFactors hasn’t been impacted by the half yearly release.
So where does this leave us?
Very soon, with the release of SuccessFactors embedded analytics (People Analytics), more and more customers are going to want/need to implement SAP Cloud IAS within their SuccessFactors environments. I would imagine that due to the overheads that I described that many customers would opt to purchase an additional subscription for another non-productive IAS instance. If I think of the cost/overhead of:
maintaining all non-productive users in pre-prod (imagine having to hire someone in the pre-prod environment just so you could test a hire in the test environment!)
Separating out test system users/data from pre-prod environment to ensure regression testing payroll changes.
Migrating IAS and IPS (tool used to synchronise user record to IAS) configuration from one system to another to enable release testing, then flipping back if urgent production support testing needs to occur.
Restricting release regression testing to only those users that have pre-production access and only to their regular roles.
Then I think I can see the cost benefit of purchasing an additional IAS system to handle that. But I really don’t want to be forced into paying additional subscription to handle a scenario that works just fine right now.
Given that when I raised this point in SuccessFactors community site I was told that:
…By design, you don’t need a 1 to 1 relationship between SF instances/tenants and IAS.
IAS is designed like any other Identity Management product to handle logins for many different systems. Customers don’t buy a new copy of Ping Identity or Microsoft Azure for every application they use it for. Your IAS configuration will control which SF instance users log into.
Additionally, having the 1:many approach makes it a lot easier to manager [sic]. If you had multiple copies of IAS/IPS you would need to figure what is connect to what every time you want to manage anything. When you re-use the same IAS for many applications, you only have to configure one time and then re-use them. ie…Password policy settings, Corporate Identity Provider connections etc. The corporate IDP is a huge one since you have to work with your internal SSO folks any time you change anything there.
I think there is a disconnect between how some SuccessFactors product managers think customers are using their product and how it is being used. Many customers do not use SSO into their non-productive environment by design!
I really hope that by spending the time to put this post together I can raise some awareness of this problem before it becomes a bigger issue for customers. Ideally, I’d love SAP SuccessFactors to re-evaluate their stance on providing only one IAS instance for all non-productive SuccessFactors instances. It should clearly be (in my not so humble opinion) one IAS per SuccessFactors instance. (on a technical note, happy for just one non-prod IPS). Let’s see. If you’re reading this and you have some influence with SAP SuccessFactors it would be great if you let me know what you think and perhaps let others know too.
Last thought on solution parity
Finally let me leave you with one last diagram/thought…
I love working with the SAP SuccessFactors software and I think that the IAS is some great functionality. However, no new functionality that replaces existing functionality that a customer has should ever require them to purchase additional subscription to retain parity with their existing solution.