Syed Saad Ali

Oracle ACE Pro

Oracle Solution Architect

Oracle E-Business Suite

Oracle Cloud Infrastructure

Oracle Fusion Middleware

Oracle Database Administration

Oracle Weblogic Administration

Syed Saad Ali

Oracle ACE Pro

Oracle Solution Architect

Oracle E-Business Suite

Oracle Cloud Infrastructure

Oracle Fusion Middleware

Oracle Database Administration

Oracle Weblogic Administration

Month: May 2023

May 22, 2023 The worker should not have status Running or Restarted
APPS DBA

AutoPatch error: The worker should not have status ‘Running’ or ‘Restarted’ at this point.

ERROR:

R12.2 ADOP:

AutoPatch error:
The worker should not have status ‘Running’ or ‘Restarted’ at this point.

Telling workers to quit…
All workers have quit.
Connecting to APPS……Connected successfully.
AutoPatch error:

Error running SQL and EXEC commands in parallel

CAUSE:

  1. The adpatch or adop process was terminated abruptly at the OS level while the patch was being applied.
  2. The database was either shut down or terminated unexpectedly.
  3. Error encountered when tried to re-start adop patch.

SOLUTION:

Step 1: Launch Adctrl and check the status of the workers. Are the workers currently running or have they been started?

Select the following options in Adctrl: 
   Option 1 to Show Worker Status then
   Option 4 to Tell manager that a worker failed its job

Step 2. Restart the ADOP patch from the following given OPTIONS.

  • If you wish to restart a failed patch from the beginning, specify the following option
adop phase=apply patches=<Patch_Number> restart=no abandon=yes
  • If you wish to restart a failed patch from where it left off, simply specify the following option

adop phase=apply patches=<Patch_Number> restart=yes abandon=no

May 20, 2023 OPMNCTL startall Fails with status 4200
APPS DBA

OPMNCTL startall Fails with unexpected exit: status 4200

When attempting to initiate the opmnctl services in E-Busines Suite R12.2, an error with the code 4200 is encountered as shown below.

Error: 

opmnctl startall: starting opmn and all managed processes… 

[opmn] [ERROR:1] [] [internal] /u01/apps/fs1/FMW_Home/webtier/opmn/bin/opmn: unexpected exit: status 4200 

opmnctl startall: opmn failed to start. 

 

Reason:

The inability of OPMN and OHS to start is attributed to the default wallets generated by Oracle Fusion Middleware, which include self-signed certificates signed using md5WithRSA. However, the updated security library no longer supports certificates signed by md5WithRSA. To resolve this issue, the certificate signature algorithm must be sha256WithRSA or a more advanced algorithm.
 

FIX:

By following the below steps, you can generate a new wallet that includes a self-signed certificate meeting the required standards. Subsequently, you can deploy this wallet to the default locations, enabling the successful startup of OHS and OPMN.
 
 
Step 1: Begin by logging in as the user responsible for the application tier installation. Then, access the run file system environment and the $FMW_HOME/SetWebtier.env  file
 
$ source <EBS base install directory>/EBSapps.env run 
$ source $FMW_HOME/SetWebtier.env 
 
Step 2: Proceed with setting an alias for the appropriate orapki command.
 
$ alias orapki=$FMW_HOME/oracle_common/bin/orapki 
 
Step 3: In the third step, generate a new wallet with an acceptable self-signed certificate in the directory $HOME/ss  Here’s an example:
 
$ mkdir ~/ss 
$ cd ~/ss 
$ orapki wallet create -wallet ./ -auto_login_only 
$ orapki wallet add -wallet . -dn "CN=FMWSmallCircleOfTrust" -asym_alg RSA -keysize 2048 -sign_alg sha256 -self_signed -validity 3652 -auto_login_only 
$ orapki wallet display -wallet . 
 

To ensure the successful creation of the wallet, you can use the final command (orapki wallet display -wallet .) to verify its status.

Step 4: In the fourth step, identify the instanceName of the current environment and save it as an environment variable for future reference.
 
tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' 
EBS_web_OHS1 
iName=$(tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' ) 
 
Step  5: Proceed to navigate to the instance directory as part of the fifth step.
 
$ cd $FMW_HOME/webtier/instances/$iName
 

Locate the default wallets utilized by this instance.

$ find . -name cwallet.sso | fgrep -v /webgate/ 
./config/OPMN/opmn/wallet/cwallet.sso 
./config/OHS/EBS_web/proxy-wallet/cwallet.sso 
./config/OHS/EBS_web/keystores/default/cwallet.sso 
 
Step  6: As part of the sixth step, ensure that each wallet solely consists of the self-signed certificate for verification purposes.
 
$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo -e "n$w"; orapki wallet display -nologo -wallet $w ; done 
 

For default wallets, confirm that each wallet exhibits a single “User Certificate” and a matching “Trusted Certificate.” Refer to the following example for clarification:

./config/OPMN/opmn/wallet/cwallet.sso 
Requested Certificates: 
User Certificates: 
Subject: CN=Self-Signed Certificate for EBS_web_OHS120,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US 
Trusted Certificates: 
Subject: CN=Self-Signed Certificate for EBS_web_OHS120,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US 
 
Note: Please note that if a wallet is not a default wallet, it is advisable to create a backup copy of the wallet before proceeding further. 
 
Step  7:  Duplicate the newly created wallet with a SHA-256 signature and place it in all the designated default wallet locations
 
$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo $w; cp -p ~/ss/cwallet.sso $w ; done 
 

By completing the aforementioned step, you have successfully copied the newly generated self-signed wallet to the default locations within the run file system. This approach eliminates the need to modify the OPMN and OHS configuration files to reference an alternative wallet or wallet directory.

 
Step  8: In preparation for implementing the changes to the patch file system, proceed with the modification of the adop_sync.drv file located in $APPL_TOP_NE/ad/custom. Include the following updates:
 
#Oracle HTTP Server Wallet - cwallet.sso 
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso 
#OPMN Wallet - cwallet.sso 
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso 
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso 
 
Once you execute the steps outlined in Section 4 during the prepare phase (adop phase=prepare) of the online patching process, the aforementioned changes will be applied to the patch file system. These changes will come into effect following a successful cutover (adop phase=cutover).
 
Once you have completed this procedure, it is important to refrain from removing or modifying the updated adop_sync.drv file.
 
Step 9: Proceed with the re-registration of OHS if it is deemed necessary.
 
Retrieve the host, port, and user name variables for WLS Admin from CONTEXT_FILE.
 
$ aHost=$( tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_wls_admin_host"/ {print $(NF-1)}' ) 
$ aPort=$( tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_wls_adminport"/ {print $(NF-1)}' ) 
$ aUser=$( tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_wls_admin_user"/ {print $(NF-1)}' ) 
 
Perform the re-registration of OHS by executing the following commands:
 
$ cd $FMW_HOME/webtier/instances/$iName/bin 
$ ./opmnctl unregisterinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser -instanceName $iName 
$ ./opmnctl registerinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser 
 

Once the new wallet is in place, OPMN and OHS should be able to initiate successfully.

To verify the startup of only OPMN and OHS, you can utilize the commands adopmnctl.sh start and adapcctl.sh start.

Alternatively, you have the option to execute adstrtal.sh, which will attempt to start all components simultaneously.

 
 
 

 

May 16, 2023 Rename Datafile In Oracle Standby on New Location
Errors/Workarounds19c

Rename Datafile In Oracle Standby on New Location

Recently, I encountered a situation while working for a customer where there was a space constraint on the DR server. Unfortunately, we were unable to allocate additional space. However, some directories had approximately 90GB of available free space. To resolve this issue, I came up with a solution to rename the data file and move it to a new mount that had sufficient space. The process of renaming a data file on a standby database is relatively straightforward.

 
Step 1: Terminate the managed recovery process and transition the physical standby database to the mount state.
 
SQL> Alter database recover managed standby database cancel;
SQL> shutdown immediate
SQL> startup mount
 
Step 2: Now, Modify the value of the standby_file_management parameter to MANUAL. By default, this parameter is set to AUTO, which results in the automatic addition of standby datafiles whenever a datafile is added in the primary database.
 
SQL> alter system set standby_file_management=MANUAL;
 
Step 3: Relocate the datafile to a new location using the appropriate OS command. Since I am operating in the 19c environment, I have the capability to perform the move online, ensuring uninterrupted accessibility.
 
ALTER DATABASE MOVE DATAFILE '/u01/oracle/db/UATDB/APEX002.dbf' TO '/u20/oracle/db/UATDB/APEX002.dbf';
 
Step 4: Restore the standby_file_management parameter back to its default setting of AUTO.
 
SQL> alter system set standby_file_management=AUTO;
 

Step 5: Resume the managed recovery process that was halted in Step 1.

 
SQL> Alter database recover managed standby database using current logfile disconnect;
 
 
 
 
May 14, 2023 Steps to change password for APPS in EBS R12.2
APPS DBA

Steps to change password for APPS in EBS R12.2

Today, we will delve into the process of changing the EBS APPS password in Oracle E-Business Suite R12.2. While this is a routine and fundamental task for every appsdba, it is essential to perform the steps cautiously to avoid any potential risks or unforeseen complications.

Lets see the Defination of The Administrative Schema APPS, APPLSYS, APPLSYSPUB in Oracle E-Business Suite 

What is the Apps Schema?

The Apps schema refers to a schema that does not contain any tables of its own. It serves as the shared runtime schema for all E-Business Suite products. It includes synonyms for all tables in Oracle apps and also consists of packages, functions, and procedures. The default password for this schema is “apps.”

What is the Applsys Schema?

The Applsys schema comprises all the necessary tables for administrative purposes. The default password for this schema is also “apps.” It consolidates the database objects of various technical products into a single schema called Applsys.

What is the Applsyspub Schema?

The Applsyspub schema primarily handles password checking. Its default password is “pub.” Applsyspub is utilized for authentication through read-only views.

Why should the Apps and Applsys passwords always be the same?

The requirement for having the same password for both the Apps and Applsys schemas is due to the authentication process. When signing on to Apps, it initially connects to a public schema named APPLSYSPUB. This schema validates the AOL username and password entered, typically using the guest user account (operations/welcome). Once this authentication is confirmed, selecting a responsibility triggers validation by the APPLSYS schema, followed by the connection to the APPS schema.

[edsanimate_start entry_animation_type= “fadeIn” entry_delay= “0” entry_duration= “2.5” entry_timing= “linear” exit_animation_type= “” exit_delay= “” exit_duration= “” exit_timing= “” animation_repeat= “infinite” keep= “yes” animate_on= “load” scroll_offset= “” custom_css_class= “”]

NOTE: It is crucial to take backups of the following tables before proceeding with the password change for the APPS, APPLSYS, and APPS_NE users:

  1. FND_USER
  2. FND_ORACLE_USERID

 

[edsanimate_end]

Step 1. Taking backup to FND Tables

SQL> create table FND_USER_140523 as select * from FND_USER ;

Table created.

SQL> create table FND_ORACLE_USERID_140523 as select * from FND_ORACLE_USERID ;

Table created.

Step 2. Shutdown the EBS Application

Invoke application environment from the Run File System which in our case is of fs2

[applmgr@ebslive PROD]$ cd /u01/app/PROD/fs2/EBSapps/appl

[applmgr@ebslive PROD]$ . APPSPROD_uat.env

[applmgr@ebslive sql]$ cd $ADMIN_SCRIPTS_HOME

[applmgr@ebslive scripts]$ adstpall.sh

Step 3. To modify the password for the APPLSYS user, there are two available approaches to choose from.

A. FNDCPASS:

Use the below syntax:

FNDCPASS <logon> 0 Y <SYSTEM username>/<SYSTEM password> SYSTEM APPLSYS <new_password>

For example, the following command changes the APPLSYS password to ‘WELCOME’:

FNDCPASS apps/<appspwd> 0 Y system/manager SYSTEM APPLSYS WELCOME
 

B. AFPASSWD

In this case, we will utilize the AFPASSWD utility to update the password.

syntax:- AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -s <APPLSYS>

[applmgr@ebslive appl]$ AFPASSWD -c apps@PROD -s APPLSYS

Enter the ORACLE password of Application Object Library 'APPSUSER':

Connected successfully to APPS.

Enter the password for your 'SYSTEM' ORACLE schema:

Connected successfully to SYSTEM.

Log file: AFPWD_PROD_433423.log

Enter new password for user:

Verify new password for user:

AFPASSWD completed successfully.

Step 4. Execute the autoconfig command on the AppsTier, ensuring that the recently updated password for the app’s user is taken into account.

[applmgr@ebslive PROD]$ cd /u01/app/PROD/fs2/EBSapps/appl

[applmgr@ebslive PROD]$ . APPSPROD_uat.env

[applmgr@ebslive scripts]$ cd $ADMIN_SCRIPTS_HOME

[applmgr@ebslive scripts]$ adautocfg.sh

AutoConfig completed successfully.

Step 5. Proceed to start the WebLogic admin server by utilizing the following script.

[applmgr@ebslive scripts]$ cd $ADMIN_SCRIPTS_HOME
[applmgr@ebslive scripts]$ pwd
/u01/app/PROD/fs2/inst/apps/PROD_uat/admin/scripts
[applmgr@ebslive scripts]$ sh adadminsrvctl.sh start

You are running adadminsrvctl.sh version 120.10.12020000.2

Enter the WebLogic Admin password:

Enter the APPS Schema password:

Step 6. Confirm the Status of Admin server

[applmgr@ebslive scripts]$ sh adadminsrvctl.sh status

 

Step 7: Modifying the Password in WebLogic

A) If your EBS R12.2 AD TXK Delta is 7 or higher, you should follow the subsequent steps.

[applmgr@ebslive scripts] $perl $FND_TOP/patch/115/bin/txkManageDBConnectionPool.pl
Please select from list of valid options
        updateDSPassword - Update WebLogic Datasource Password
        updateDSJdbcUrl  - Update WebLogic Datasource Connection String
Enter Your Choice : updateDSPassword
Enter the full path of Applications Context File [DEFAULT -]:
Enter weblogic admin server password: <enter Weblogic Password>
Enter the APPS user password: <enter New APPS Password>

B) If your EBS R12.2 AD TXK Delta is lower than 7, you should utilize the following steps.

  1. Access the WebLogic console through a web browser.
  2. Click on the “Lock and Edit” button.
  3. Navigate to the “Domain structure” section and select “Services” -> “Data Sources.”
  4. Locate and select the “EBSDataSource” entry.
  5. In the “Settings of EBSDataSource” page, go to the connection pool section.
  6. Enter the updated password for the apps user and save the changes.
  7. Finally, click on “Activate changes” to apply the modifications.

 

Step 8 . Start the application services.

[applmgr@ebslive appl]$ pwd

  /u01/app/PROD/fs2/EBSapps/appl

[applmgr@ebslive appl]$ adstrtal.sh

 You are running adstrtal.sh version 120.24.12020000.6

 Enter the APPS username: apps 

 Enter the APPS password:

 Enter the WebLogic Server password:

 All enabled services for this node are started.

 adstrtal.sh: Exiting with status 0

 

Step 9: Validate WLS Datastore Modifications by Following These Steps

  1. Access the WLS Administration Console and log in.
  2. In the Domain Structure tree, expand the “Services” section, then choose “Data Sources.”
  3. Locate and select “EBSDataSource” on the “Summary of JDBC Data Sources” page.
  4. Navigate to the “Settings for EBSDataSource” page and select “Monitoring” followed by “Testing.”
  5. Select “oacore_server1” from the available options.
  6. Click on “Test DataSource.”
  7. Look for the confirmation message stating “Test of EBSDataSource on server oacore_server1 was successful.”
  8. If Integrated SOA Gateway (ISG) is implemented, repeat the steps for “OAEADataSource”. Verify “oafm_cluster*” managed servers are started successfully.