Invalid Planning Version

Now, I've seen a few errors on the SCM systems in my time. But this one really took me around a lot of corners.
It all started with a simple homogenous system copy. We were changing provider, and had to move our systems. After the migration, I did a few spot checks on the Live Cache, was it running, etc. And those simple checks were ok. But when we handed over the system to business, a simple forecasting ended my world. It started with a "livecache not accessible" error. This lead to a slew of missing parameters, missing configuration items, and authorization issues on the host. However, after several days of back-and-forth, the provider finally said "now it's working". So we tested again. Yes, live cache was now available, but the forecasting now gave an "invalid planning version". WTF ?!?

Clicking the error just took me to the S000 menu. So not much was offered in the way of help from SAP.
However, I COULD go to transaction /SAPAPO/MVM, and the planning version this time at least gave me an error:

An application error has occurred in the LiveCache

Message no. /SAPAPO/MVM075

Ok. I had something to go on... I tried to run a consistency check in /SAPAPO/OM17. And I do mean tried. It failed horribly too. Shortdump. What was that about ? So I ran the consistency check without selecting any object to test, and lo and behold, I caught a fish. 

Right, the planning versions. Here they were, plain as day. Clearing inconsistencies, quickly removed the entry not in the database. But why wouldn't the planning version load from the database ? All my checks on the Livecache said it was running ! Various SAP notes suggested restarting. I tried. No help. I even found a note that said to recreate the object manually (note 2419054), if the clearing didn't make the error go away. But I had no clue about how many objects were impacted, as I couldn't run the consistency check in its entirety. It could be a LOT of objects that required manual intervention, so I dismissed that idea for the time being. It had to be something closer to home. I needed to find the cause of the problem, not fix the symptoms.
I finally went back to LC10, and went into the monitoring. Everything looked ok, but something was fishy, it had to be. Something in the back of my head told me to check the datafiles. they were there. so I checked the tables within... And ... CRAP. They were empty.

So no wonder the livecache couldn't write to the database, or read from it. No wonder I got a shotdump when trying to read the data tables. There was just a clean database, with no imported tables. Not a single APO entry in sight. So I had my culprit. Now I just have to find out how the provider could have mucked up the restore. Not to mention, having them restore everything again.

Installing SAP content server on a windows 2012 with IIS 8.5

As I may have said once or twice before. SAP is "old" technology, and things built on legacy technology doesnt always hold up to modern times. A prime example is the Content Server, which certainly works on Windows 2012, and with IIS 8.5 You just can't install it with the installer (!!!)

Now, the problem isn't bigger than you have to do the installation manually (like we did before the installer tried to do everything for us). The major issue is that SAP has removed those installation guides, because the technology is no longer supported (Someone go smack the guy who decided to remove good, working knowledge from a knowledge database). So what you do is follow these simple steps (you may have to change the paths, depending on where you installed):

1. Install with the installer, as you normally would. Everything will look fine. But there will be no websites installed, even though you clicked the "install website".

2. Go to Internet Information Services Manager 

3. Right Click on Sites > "Add Website" to create a new Website.
- Site name  "SAP Content Server"
- port "1090"
- Physical Path "d:\sap\ContentServer"
- Application Pool > Select "DefaultAppPool"
- Click "OK"

4. Right click on the newly created website >> "Add Application"
- Alias "ContentServer"
- Application Pool > Select "AppPool_SAP Content Server"
- Physical Path "d:\sap\ContentServer"
- Click "OK"

5. Double Click on the website > Handler Mappings
- Select "ISAPI.dll" from the list >> Edit Feature Permissions
- Check the tick boxes for "Script" and "Execute"
- Click "OK"

6. Double Click on the website > Authentication
- Enable "Anonymous Authentication" (if it isn't already)
- Enable "Basic Authentication"
- Enable "Windows Authentication"

7. Double Click on the IIS Server name > ISAPI and CGI Restrictions >> Click on "Add"
- ISAPI or CGI Path "d:\sap\ContentServer"  > selecting "ContentServer.dll"
- Description "ContentServer.dll"
- Check the tick box "Allow extension path to execute"
- Click "OK"

8. Add rules to the windows firewall (ingoing port 1090 and 1095 if you are doing cache server too)

Lo and Behold: One running website with Content Server. And DO remember to set the storage somewhere you have enough space :)

As you may be able to imagine, the Cache Server .dll is very much the same steps.

Weird connection requests to xxxx.wdf.sap.corp in the ICM trace

Maybe you have seen one or two systems that for whatever reason fills up your wonderful log with totally useless information.
One of the more annoying ones, is when SAP accidentally leave stuff in place that should never have gone into a customer system. One of these is when you see messages like:

[Thr 139801848358656] Sat Nov 19 01:19:34 2016
[Thr 139801848358656] *** WARNING => Connection request from (8/9/0) to host: hs2086.wdf.sap.corp, service: 2000 failed (NIEHOST_UNKNOWN)
[Thr 139801848358656]  {00081aac} [icxxconn_mt.c 2108]
[Thr 139801845188352] Sat Nov 19 01:24:33 2016
[Thr 139801845188352] *** WARNING => Connection request from (7/8/0) to host: arcdev.wdf.sap.corp, service: 1080 failed (NIEHOST_UNKNOWN)
[Thr 139801845188352]  {00081abe} [icxxconn_mt.c 2108]
[Thr 139801845188352] Sat Nov 19 01:29:34 2016

These will populate your ICM logs, and clog them up forever if you dont do something about it.

Now, the smart person might quickly realize that these are http connection requests (why else would they be in the ICM ?), and you may even have guessed (either based on the name or the port) that this is a content service of some kind. But when you go to transaction OAC0, there are tonnes of SAP standard stuff in there. Which one is it that you need to get rid of ?
Well, fortunately, the answer is simple. Go to transaction CSMONITOR, fold out the "ContentServer", and keep folding out, untill you see the names of the archivelinks. (in a netweaver 7.3, it is usually named A2 and B2)
then simply go to transaction OAC0 and mark and delete these two offenders

S_RFC_ADM.... the auth check we all love to hate

So many times since SAP decided to create the S_RFC_ADM, but also choose not to include it in the S_A.SYSTEM role, I have hated having to do usually BI data source stuff. So often I have to re-create an RFC destination, and in so many source systems I just dont have a role that contains the ability to create an RFC destination.
Ofcourse I can wait a week or two for the auth guy to make me one. But often times the users dont really sympathize with my plight when I tell them how hard it is to get access.
So my little workaround: In most systems there is an authorization to maintain archivelink. I ask for that instead. It's usually much simpler to get a default auth, than to have a custom one created.

the auth in question is S_AL_ALL. and it contains "just" the ability to maintain RFC destinations, and archivelink destinations. Perfect for my needs.

SAP application giving HTTP error 403.2 - Forbidden

You may have seen a screenshot like this a few times in the past.

It's fairly common when you run peripheral services for your business suite on a windows environment. Maybe it's TREX, maybe it's a content server, maybe something entirely third.

And the cause is rather simple: Microsoft supports the use of ASAPI extensions and filters, but prefers that you use modules instead. So the IIS has ASAPI disabled by default. But SAP is "old" technology, so they don't change, just because Microsofts recommendation does. So the trick is to simple enable it.
If you are running server 2008, start the server manager, go to roles, click on the webserver IIS, and in the right hand pane, scroll down to role services. Here you enable ISAPI extensions, and ISAPI filters.
Reboot the server, and notice the difference !

Now, if the difference is you get a permission DENIED instead of forbidden, you may also want to look at if your service user actually has the rights to access files in the virtual directories.

Automatically starting SAP after reboot of windows

SAP was never really made to run on Windows. We all know it, we all accept it, and we all have to deal with having a couple of IDES, or sandbox systems, or even just something from old times on a windows server somewhere. Maybe you are even one of those people who have an entire suite of SAP systems on Windows. So you will be even more annoyed by the tedious AV, patching and update process, that just continually needs to reboot "your" server.
So you have to get weird calls at odd hours about SAP systems not running, or even go through the tedious process of starting the systems manually after every reboot. That takes time. Lots of time.... But it really doesn't have to.
First step is to create a batchfile that does the actual starting of your SAP system. It's fairly simple actually, you just use the sapcontrol.exe
My script looks something like this:

D:\usr\sap\SID\DVEBMGS00\exe\sapcontrol.exe -user sidadm password_here -nr 0 -function Start
D:\usr\sap\SID\DVEBMGS00\exe\sapcontrol.exe -user sidadm password_here -nr 1 -function Start

the number after the -nr parameter is the instance number on the host. You may also have a hostagent that needs to start. remember to add your SID, SIDadm user and his password. Tehnically, you may want to elevate the script to be able to run without using a user/password, or even create a dynamic script to adapt to the use of a changing password, or even consider creating a service user that only runs this script. But for demonstration purposes, this is fine. test your script by closing down the SAP systems in the MMC, and start them again with the script.

On Unix we can simply put in a startsap call at the boot time. But Windows seems to implement this much differently. So how do we run the script once windows starts up ? you may have tried the startup folder, or even added a script to the boot sequence in regedit. And you have likely failed.
But it is actually rather easy. Use windows task scheduler.

You just have to use these settings: 

* Run whether user is logged on or not

* Begin the task at startup

* Delay task for 1 minute (otherwise your database will be down, and the process will fail)

* Action start a program (browse to your script)

And hey presto.... You dont need to get up at night for a virus definition update.

If you want to be very sure, add a pause in your script, and call the sapevt and run a DBcheck (or whatever automated process you may have for validation), followed by a mail or text to your phone. 

Do note that shutting DOWN the system wont be this easy. So if you want to avoid have to "massage" your DB back to life after an unexpected powerdown, you may also want to consider automating the shutdown of the instances (and the DB service).

Patching without using solution manager

Ohhhh, I was so miffed the first time I ran into the idea that you HAD to use solution manager to patch your SAP systems. I knew SAP at some point in time would do something to force adaptation of this horrible bastard technology. But I was cross when I found out it had to be in the basis functions.
Dont get me wrong, when it works, is correctly configured, and you have a decent change and support porcess to mirror the processes within the maintenance optimizer, it does mean you dont have to read tonnes of notes about package dependencies, or the likes. A calculated queue is easily created, downloaded and applied.... IF the process works as supposed. I have never seen that happen even ONCE in the last 6-7 years I've worked with the maintenance optimizer.

And if you just need to download 4 packages and get on with your work, it would be nice to be able to skip the MOPZ entirely.... and you can !!!!
Simply find the packages you need as usual, now they will be in the "waiting for approval" step in your download basket, because.... Well, that's how SAP forces you to use solman. Now simply go to solman and run the se38 report /TMWFLOW/MO_UI_BASKET_AUTHORIZ
And voila. You can now approve the basket. Go back to the downloadbasket, and start downloading, and apply packages they way we have done it the last 25 years...

Gaining access to a client without restarting SAP

The other day I was in a meeting with a supplier. We were discussing responsibilities of joint systems among other things.
It turned out that we both thought the other side held the usernames and passwords to the technical SAP clients. And much to my surprise, what was thought of as a problem, was not that noone were responsible, but that noone knew how to gain access to the clients without restarting the systems, which was problematic, as the manufacturing systems needs to be running 24/7.

Well, technically, noone in that room knew, besides me. Because I have in the past used a small oracle script to gain access to clients where the passwords to users had been lost.

The cause of the problem is that the SAP function of the no_automatic_user_sapstar parameter requires that we activate that parameter, in order to be able to login with the default user. and that means restarting the system. And once access was gained, another restart would be required to set the parameter back. This functionality is onnly meant to be used for creating a new client, so ofcourse it is not very conveniently built.

So to circumvent the issue of the restarts, I shared with our supplier, this little Oracle script they could run to gain access into client 001, without restarting the SAP system. We have Admin access to the Oracle, so ofcourse we can simply manipulate the data in the database:

rem This part will reset the lock incase you accidentally locked the user trying to get in
UPDATE sapsr3.usr02
   SET uflag = 0
WHERE mandt='001' AND bname='SAP*';

rem This part will take the password from client 000s sapstar user and give it to the same user in 001
UPDATE sapsr3.usr02
   SET bcode = ( SELECT bcode FROM sapsr3.usr02 WHERE bname ='SAP*' AND MANDT='000' )
WHERE mandt='001' AND bname='SAP*';
COMMIT
;

Now the only real downside to this approach is that we have now only changed the data on the database. Not in active memory. Well, depending on the release version of SAP and whatever parameters you have set, it may be loaded into memory when you access it.
I would recommend trying to logon with the client 000 password, if you have tried this and you fail, likely it is because the data hasn't migrated into memory yet. We can fix that by simply calling the transaction /$sync as any user and in any client you have access to.

But try to run this transaction at low usage periods. It can kill a lot of cache.

Enjoy carefree access to your technical clients.

SRM system showing approval mail with saphtmlp link

The other day, one of my colleagues came over to me and asked my why our approval emails sent out to all the managers had a weird logon link in them
I logged on to the system and said, "what do you mean ?". As I saw nothing wrong.

"Well", my colleague replied, "when I look at the properties of the mail in SOST or SOSG, it contains a lot of weird stuff in front of the actual address, but when I remove that it works".

I looked, and I saw what he meant. There was a reference to the sap HTML viewer, even the approval link had the sapevent infront of it. And the reason for sending out these mails was exactly to be able to send them to someone without access to SAP. I pondered this for a while, and I looked at notes, read documentation. Then it occured to me - But if we hadn't changed this in the last 6 months, then why had noone complained ?
So on a hunch that SAP would render this differently in SOST than after transmission, I asked my colleague to get a hold of a mail from one of the recipients. And lo and behold, SAP DOES infact render this differently when viewed from within the SAPGUI. The links were perfectly created in the mails people recieved.
Note to self: Ensure the ability to test as an end user.....

SMQS scheduler status shows RES_LACK

Today I was confronted with the fact that our BI system wasn't reciving data from our R3. After a short investigation, I found that the IDOCs were not processed. Manually processing them sent them to the tRFC queue. However, from here they just ended in "transaction recorded".
From past experience I knew to check the registrations in the SMQS. However, today I was met with a weird situation. The registration was fine, but the scheduler had a weird status.

After a while of looking, I noticed that the "host ID" was that of another system (!)
My suspicions immediately went to last weeks homogenous system copy. So I went through every step of the RFC process, from data creation to the REF destinations. And lo and behold.... the RFC logon groups (RZ12) had not been configured. Simple fix, furtunately.

SAP line opener showing correupted XML file recieved

If like me, you are forced to use an archaeic line opener, that cannot even run as a service. You may on occasion run into problems, and wonder why there is no support to get. And in truth I cant tell you. Sometimes I get angry and just continuously ping the sapserv1 server, and then go home, knowing that things are working, but saddend by the fact that it's not running optimally.

However, at least one function I have managed to find a solution for. I frequently got the error:

Error: 04-08-2014 09:44:38: Function startProcess : Corrupted XML file received
Error: 04-08-2014 09:44:38: parseXML: non-XML file
04-08-2014 09:44:38:  begin of make_https
04-08-2014 09:44:38: Getting SAPRouter data for LOP in the SMP

I tried reinstallting the LOP, I tried using a different user, I've tried pretty much anything, but it never worked. Untill I found out that it was caused by the company proxy. Not that the proxy itself poses a problem, but the way SAP tried to connect, isn't entirely compatible with modern proxies (You will know this if you have ever tried to replace a SAP webdispatcher with a real reverseproxy).

So how did I resolve the problem ? Simple. I just logged into the SAP marketplace from the server running the LOP. Then the proxy information is passed correctly by IE, which also means the LOP wont have to.

Logical system name for this BI is not allowed

Following a homogenous systemcopy of our BI, we were following the normally outlined strategy from SAP regarding copying BW/BI systems, in SAP note 886102.
In this case, however, a successfull, albeit very fast run BDLS job, resulted in the following error message when attempting to rename the logical system. Even SAP were unable to provide a timely solution to the problem.

This was, in retrospect ofcourse caused by the fact that the BDLS job only LOOKED successfull, but in fact stopped after only a few tables. One of the tables was ofcourse the T000 (which meant that the client would try to use a logical system name which didn't correspond to the technical name), aswell as the DBLS table(s). So everything looked to have finished ok. There also wasn't any error messages.

The trick was, how could I see that the job actually hadn't doen what it was supposed to ? BDLS runs for other logical systems ran just fine, and this had never been an issue in the past. I had to run a debugging of the BDLS transaction.

I had in the past had many discussions with a colleague about the BDLS functionality. I said there was a dynamic program containing the fields to be converted, he said that all fields defined as "logsys" would be converted. It turned out We were both only partly right. There IS a program, which creates a dynamic mapping of all the fields containing the logsys. And then the conversion starts based on this BDLSMAP. However, there apparently isn't any controls available to determine if this mapping fails. and in my case it did.

So I ended up finding a service program for the BDLSMAP that could clear the mapped data, After running that, the BDLS run finished fine, and all of our logical system problems went away. I'm seriously considering if this program, RBDLSMAP_RESET shouldn't ALWAYS be run prior to a BDLS run.

Selected system is not a quality assurance system

The other day my developer colleagues started reporting problems with them being unable to release transports for testing. the STMS_QA transaction was throwing an error.  

I immediately looked at the STMS configuration, and found that the system attributes for the QA system did not have a checkmark at "delivery after confirmation". So I logged into the domain controller, set the checkmark and redistributed the configuration. That should have been the end of it.
But lo and behold, it didn't work..... I logged on the the QA system again and had a look, the changes wasn't distributed, even though I pressed the button (!!!). Luckily the "adjust with controller" DID grab the configuration from the controller...

That fixed the problem. Now to figure out why the distribution didn't work.....

Service parameter [something] contains special characters. Please correct

This thing started to annoy me at one point in time. I've had the error in the past as well, with a setting for ~sap-language, but now I also had it with a HTTP-HTTS (yes, I know, it should be HTTPS, but this is how SAP has named the parameter)

I bing'ed, searched notes, the SCN, various SAP groups, and everyone had an answer that was completely useless to me. Untill I found out I was actually searching for the wrong thing. Despite getting the error message here:

I found out I was actually looking for the wrong thing. What I should be focusing on was the fact that this was a special character, and not what it was, or why it was. I then found a note saying that the use of special characters was not supported for all basis versions.... I was astounded that this note did not turn up in my initial search, but note 1780585, actually did resolve my issue. Now I could add in the minus sign in my parameter. Yay ! Oh and please dont ask me why I would want to add in the HTTP-HTTPS parameter, unless you want to see a grown man cry.

Moving your SAP system to UTC

'Tis the the season for shutting down your SAP systems.... Again. And you likely did it last year aswell. Because SAP says you should, as the SAP system does not support the double timestamps that might occur.

Well, then you have the option of using "extended time". And some people might be very happy about this.
But what really happens, is that you get the wrong timestamp on items, up to half an hour off, and some businesses might not be happy with that.

Fortunately it's fairly simple to migrate to UTC (or UTC+1, or any other non-DST timezone) in your SAP system, even if you even missed it again this year, because the good news is: You have an entire Year to implement your changes. This will become an issue again next year.
The problem is that SAP apparently aren't able to provide a guide for how to do this, so many companies choose to have the added cost of shutting down the systems. But you really dont have to, changing to UTC time is really simple.
And all you have to do is:

Set servertime to UTC (or any non-DST timezone equivalent), on ALL servers in the instance

Set timezone on each client using transaction STZAC (while users are not active) identical to servertime

Set timezone for each user in transaction SU01 (or using a masschange or similar)

Download zip file from note 198411 and apply to SAP system using report TZCUSTIM

Verify timezone data consistency with report TZONECHECK, correct any problems as they show up.

Be aware that changing to UTC may also require downtime. This is required if you are moving your system back in time (eg from UTC+4 or equivalent to UTC+2 or equivalent), but not if you move it forward in time.

Benefits:

No need for downtime during start of DST

No need for NTP service during end of DST

Less confusion about time across multiple timezones

Easier to implement the use of various timeservices, NTP etc.

No time differences in documents of the type seen using SAPs “extended time”

Drawbacks:

Users may need time to adjust, as timestamps may vary between user- and servertime depending on programming, eg. Posting time is usually based on servertime, so end/beginning of time period may move a few hours.
Batchjobs are based on servertime, and jobs required to run at specific times may thus need to be rescheduled

Timezones must be set identical for ALL application servers, database servers, listeners etc for an entire system.

           Need to maintain time zones on a regular basis (they change a lot), see note 198411

Performing OS commands from within SAP

We all know it, we all need it once in a while. The options for running stuff from within SAP (like for instance file manipulations) just isn't really that well developed within SAP. However, you can often run single commands within SAP using transaction SM69. However this transaction has severe limitations. You can only run one command, and you cannot stack them by sending command interpreted characters along (like the pipe | ). Similarly, using -exec as a parameter is also not a legal call, so you can pretty much only do very basic manipulation from within SAP.
that leaves two options. One, you give permission to use SE38 to whatever user needs to run the report. Auditing will look very disfavorable upon that option. Or two, you make a script that does what you need, and call it.

Now, in the past, I have simply created my scripts on the OS, and caleld it from within SAP. It was simple, clean and functional. I did it, because it was easier to do, and because I had access to the OS.
But this week, I was posed with a problem of having to delete multiple files from a directory, of a certain date and with a specific syntax. something that I needed a script for. But I have no OS access to the systems. So what do I do ?

Well, I look at the type of OS my SAP is running on, from my system information:

then I create a small ABAP program with the following data

REPORT  ZZ_CLEANUP_INTERFACE_FILES_1.

DATA : my_command(255).
DATA: BEGIN OF MYTABL OCCURS 0,             " defines output
            LINE(255),
      END OF MYTABL.

my_command = 'find /usr/sap/interfaces/*zip* -mtime +15 -exec rm {} \; '.   " the Linux command

CLEAR MYTABL. REFRESH MYTABL.
CALL 'SYSTEM' ID 'COMMAND' FIELD my_command
              ID 'TAB'     FIELD MYTABL-*SYS*.

LOOP AT MYTABL.                             " displays output
  write :/ MYTABL-LINE.
ENDLOOP.

Had the system been a windows box, I could simply have typed in a windows command in the my_command field instead.
Do I need to run more than one command, I can just do a new my_command followed by a new SYSTEM call.
Do I need manipulation of the data, I have the output, line for line in MYTABL.

The really huge upside to this strategy is that: It's totally hardware independant (ok, it will only run on a similar OS, but I can move to another server and it will still work), I can limit access to the report via normal SAP authorizations, and I even get reporting, documentation and alerting similar to any SAP report.

I am certainly one happy camper, and might entirely forego the use of OS scripting in the future.

SoapDoc::Init: Cannot set inout buffer

A while ago I was setting up some SOAP connectors for our BI guys. I went into the SICF transaction and activated the connectors they needed.

They called me back, complaining that they would get this error message in the SOAPUI:

<p>URL http://xxx.xxx.xxx:yyyy/sap/bc/soap_sap-client=zzz call was terminated because the corresponding service is not available.</p>

I investigated. I tried to go to the webpage and see if it would return anything to me. I went into my browser and disabled the "nice error messages" option, so I could see it all. I then went to the URL and found:

 

My users immediately complained that "there's an error - Look it says cannot set inout buffer, the problem must be in the ICM, you fix it". But I didn't actually call the interface with any data. So it was just returning what it was asked.... An XML with absolultely nothing in it. Sending data through the interface would make the error go away.

Lesson learned: A "nice" error message is problematic, but a misleading one isn't any better ;)