Weblogic 12.2: AdminServer overwrite by restarting all your changes in the OHS mod_wl_ohs.conf file

For using Forms and Reports in Weblogic 12.2 with OHS server we changed Oracle HTTP Server Configuration in the $DOMAIN_HOME/config/fmwconfig/components/OHS/instances/ohs1/mod_wl_ohs.conf according the Oracle Documentation. But after the restarting of AdminServer the configuration file was always overwritten with default content and all our changes were gone.

By the starting of AdminServer we get following lines in the logs:

 NMProcess: Mär 24, 2017 7:51:15 AM oracle.ohs.plugin.nodemanager.OhsReplicationPlugin commit

 NMProcess: INFORMATION: Instanz ohs1 wird aktualisiert

We found, that the content of above mod_wl_ohs.conf file will be overwritten with the content of same named file from $DOMAIN_HOME/config/fmwconfig/components/OHS/ohs1 directory.

So for changes in the OHS configuration you need to edit only the file $DOMAIN_HOME/config/fmwconfig/components/OHS/ohs1/mod_wl_ohs.conf and copy it into the instance/ohs1 directory.

 

New feature for starting of Node Manager with WLST in Weblogic 12.2: waiting for successful connection.

For starting of the Weblogic infrastructure  our customer used a WLST script. The script looks like this:

startNodeManager(NodeManagerHome=<nmHome> ListenPort=5556,ListenAddress=localhost)
sleep(60)

nmConnect('weblogic','xxx','localhost','5556','MYDOMAIN')
nmStart('AdminServer')
nmStart('WLS_FORMS')
nmStart('WLS_REPORTS')

First step is the starting of the node manager. The node manager process needs some time before the listener on the port 5556 is active. The “some time” is depend on several factors and the waiting of 60 sec can be not enough for the starting of the node manager.  In this case the following connection to the node manager (nmConnect) will be fail and all components of the Weblogic infrastructure (except Node Manager:-) ) remain down. For the automatically start of the Weblogic server after an unplanned rebound of the system it’s not optimal solution.  Oracle understood the problem too and extended the startNodeManager procedure in the version 12.2.1.2 with the following parameters:

block—Specifies whether WLST should block until it successfully connects to Node Manager or fails to connect within the specified timeout.

    nmConnectOptions—When block is true, use this argument to specify a list of Node Manager connection options.

    timeout—The number of milliseconds to wait for Node Manager to connect.

With this extension we can wait for successful connection to the Node Manager by starting and you don’t need the sleep() command anymore. Start node manager command will be looks like this (the blue marked parameters are the nmConnect parameters):

                  startNodeManager(block='true', 
                  timeout=600000, 
                  NodeManagerHome='<NodeMangerHome>',
                  ListenPort='5556',
                  ListenAddress='localhost', 
                  username='weblogic',
                  password='xxx',
                  host='localhost',
                  port='5556',
                  domainName= 'MYDOMAIN')

Now only if the connection to the node manager is successful, the modified script starts Admin Server and other in script defined components.

Oracle Reports: REP-56133: access is denied to write to the specified location

Last weeks I had a new challenge for me. Migration of a Forms & Reports Application to Weblogic Server version 12c. They are enough documentation about installation and configuration of Forms & Reports (I used very good Installation step-by-step manual written by Borys Neselovskyi ). In my blog I will describe only the problems encountered by the migration.

The first one: REP-56133 error by report generating.

Application: The migrated application include a forms application, batch jobs and uses standalone report server for reports generating and. The batch Jobs generate Reports in for the jobs defined output subdirectories of the directory /reports/output. After the migration we got a “REP-56133: access is denied to write to the specified location” error by report generating. In the current Reports version you need to explicit allowing access on the non-default directories for reading and writing. The solution is the adding the following text lines into the standalone server configuration file $DOMAIN_HOME/config/fmwconfig/components/ReportsServerComponent/<Rep.Serv.>/rwserver.conf

<folderAccess>

      <write>/reports/output</write>

</folderAccess>

Note: the downloaded Report Version (12.2.1.2) have a bug, if you are using wildcards for folder access (/reports/output/*). Wildcards don’t work and you get the error REP-56133 message. For solving this problem you need to install the Oracle Report Patch 22334822.

 

Configure Oracle GoldenGate autorstart in Non-RAC Environment with Grid Infrastructure and XAG.

If your database and golden gate are running on a server with Oracle Grid Infrastructure, you can use Oracle Grid Infrastructure Bundled Agent (XAG) for the autostart configuration of your GoldenGate instance.  XAG is a set of components that provide the HA framework and contains pre-defined Oracle Grid Infrastructure resource configurations and application agents for complete application HA. With XAG you don’t need to write your own scripts and configurations for GoldenGate integrating into Grid Infrasturcture CRS stack.

Hire is my experience with configuration of the GoldenGate instance into Grid Infrastructure in Non-RAC Oracle environment:

XAG installation

Although that the XAG is since GI version 12.1 a component of the Grid Infrastructure, in my case for a single host GI installation was the XAG  not propably configured .I decided to download and install the actual XAG version separately from GI_HOME.

  1. XAG can be downloaded from the Oracle Clusterware Downlad Page:

http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/index.html

  1. Unzip downloaded file in a temporary directory:
unzip xagpack.zip
  1. Change to the unpacked directory and install as oracle user the XAG into a new created XAG_HOME:
./xagsetup.sh --install --directory /u01/app/oracle/product/xag

 

  1. Add GoldenGate  instance configuration into CRS.
agctl add goldengate gg_1 --gg_home /golden_gate/ogg --instance_type source --databases ora.ggsource.db --oracle_home /u01/app/oracle/product/12.1.0/dbhome_1 --jagent_autostart yes

If you use GoldenGate Monitoring Agent and you want to start it automatically with the GoldenGate instance, you must add –jagent_autostart yes to the configuration command(since GoldenGate version 12.2 Oracle removed autostart jagent parameter from the manager process configuration).  For monitoring Agent is important that the default JVM version on the system is greater as minimal required for the agent. OpenJDK can’t be used for the monitoring agent!

crsctl stat res -t shows that the new resource was successful configured:

[root@gg1 gg1]# crsctl stat res -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details      

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

               ONLINE  ONLINE       gg1                      STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       gg1                      STABLE

ora.asm

               ONLINE  ONLINE       gg1                      Started,STABLE

ora.ons

               OFFLINE OFFLINE      gg1                      STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

      1        ONLINE  ONLINE       gg1                      STABLE

ora.diskmon

      1        OFFLINE OFFLINE                               STABLE

ora.evmd

      1        ONLINE  ONLINE       gg1                      STABLE

ora.ggsource.db

      1        ONLINE  OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.ggstest.db

      1        ONLINE  OFFLINE                               STABLE

xag.gg_1.goldengate

      1        OFFLINE OFFLINE                               STABLE

 

Starting of the Oracle GoldenGate instance.

Now we want to start our GoldenGate Instance with the GridInfrastructure:

crsctl start res xag.gg_1.goldengate

But…It’s not working.

A look into a ggserror.log file:

2016-10-19 15:16:25  ERROR   OGG-00664  Oracle GoldenGate Manager for Oracle, mgr.prm:  OCI Error beginning session (status = 1034-ORA-

01034: ORACLE not available

ORA-27101: shared memory realm does not exist

Linux-x86_64 Error: 2: No such file or directory

Additional information: 3640

Additional information: -1977179833).

2016-10-19 15:16:25  ERROR   OGG-01668  Oracle GoldenGate Manager for Oracle, mgr.prm:  PROCESS ABENDING.

=> Check the target database: the database is available.

Look into the XAG log file (locates in the directory XAG_HOME/log)

2016-10-19 15:04:55: executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl status res ora.ggsource.db -f

2016-10-19 15:04:55: GEN_USR_ORA_INST_NAME\@SERVERNAME\(gg1\)

2016-10-19 15:04:55: getting instancename using GEN_USR_ORA_INST_NAME@SERVERNAME(gg1) on ora.ggsource.db

2016-10-19 15:04:55: Exported ORACLE_SID ora.ggsource.db

 

The XAG uses the GI database resource name ora.ggsource.db instead of the real db instance name ggsource. I looked into the XAG scripts and found the following functionality: To recognize the database SID the XAG reads all CRS configuration attributes for the used database and takes as the SID a value of the  GEN_USR_ORA_INST_NAME@SERVERNAME(<host>) property. It works for RAC but not for single host oracle GI installation. For Non-RAC environment is the value empty and in this case uses the XAG the CRS database resource name as the ORACLE_SID.

To fix it in the current XAG Version you need to modify the agcommon.pm file in the XAG_HOME directory.

Open the file and find the following lines,

my $sidAttrName = 'GEN_USR_ORA_INST_NAME@SERVERNAME(' .
                          $nodeName . ')';

comment they out, and insert after the lines the functionality replacement for single host environment.

        my $sidAttrName = 'GEN_USR_ORA_INST_NAME';

You will have now the following code block:

      #my $sidAttrName = 'GEN_USR_ORA_INST_NAME@SERVERNAME(' .
      #                    $nodeName . ')';
      my $sidAttrName = 'GEN_USR_ORA_INST_NAME';

Now we can start our GoldenGate instance as a GI resource.

crsctl start res xag.gg_1.goldengate
xag.gg_1.goldengate

      1        ONLINE ONLINE                               STABLE

 

[oracle@gg1 ogg]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.0 OGGCORE_12.2.0.1.0_PLATFORMS_151101.1925.2_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Nov 11 2015 03:53:23
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2015, Oracle and/or its affiliates. All rights reserved.



GGSCI (gg1) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           
JAGENT      RUNNING

 

 

 

Exadata: Grid Infrastructure starting hangs after DB Node OS patching.

Recently we patched an Exadata Eighth Rack by our Customer. The Exadata Machine has RAC One Configuration with 12.1 Grid Infrastructure and many 11.2 Database Instances. After the successful OS patching on the first node with dbnodeupdate.sh utility without any errors or warnings, we didn’t get stared the CRS stack during the post patch action:

  (*) 2016-11-05 13:14:27: Locking and starting Grid Infrastructure (/u01/app/12.1.0.2/grid)

  (*) 2016-11-05 13:16:27: Sleeping another 60 seconds while stack is starting (1/15)

  (*) 2016-11-05 13:17:27: Sleeping another 60 seconds while stack is starting (2/15)

Manually start of the crs stack didn’t work too.   The Output of the crsctl stat res –t –init showed, that the resource ora.storage hangs by starting:

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        OFFLINE OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.cluster_interconnect.haip

      1        ONLINE  ONLINE       exaserv-dbadm-01         STABLE

ora.crf

      1        ONLINE  OFFLINE                               STABLE

ora.crsd

      1        ONLINE  OFFLINE                               STABLE

ora.cssd

      1        ONLINE  ONLINE       exaserv-dbadm-01         STABLE

ora.cssdmonitor

      1        ONLINE  ONLINE       exaserv-dbadm-01         STABLE

ora.ctssd

      1        ONLINE  ONLINE       exaserv-dbadm-01         OBSERVER,STABLE

ora.diskmon

      1        ONLINE  ONLINE       exaserv-dbadm-01         STABLE

ora.drivers.acfs

      1        ONLINE  ONLINE       exaserv-dbadm-01         STABLE

ora.drivers.oka

      1        OFFLINE OFFLINE                               STABLE

ora.evmd

      1        ONLINE  INTERMEDIATE exaserv-dbadm-01         STABLE

ora.gipcd

      1        ONLINE  ONLINE       exaserv-dbadm-01         STABLE

ora.gpnpd

      1        ONLINE  ONLINE       exaserv-dbadm-01         STABLE

ora.mdnsd

      1        ONLINE  ONLINE       exaserv-dbadm-01         STABLE

ora.storage

      1        ONLINE  OFFLINE      exaserv-dbadm-01         STARTING

 

Wrong was also that the target status of the ora.asm resource after the node patching was set to OFFLINE. To repair this we started the ora.asm service during the after patch procedure manually:

crsctl start res ora.asm -init

The second node had after the OS Patching the same problem.