silent installation of 11.1.0.7 on Linux

There are Oracle installations that simplier to do in command line
and one of them is patchset installation.

To start it I used ORACLE_HOME and ORACLE_HOME_NAME
in addition to usual -silent -responseFile options

./runInstaller -silent \
-responseFile /u02/install/patch/Disk1/response/patchset.rsp \
ORACLE_HOME="/u03/app/oracle/product/11.1.0/db_1" \
ORACLE_HOME_NAME="OraDb11g_home1"


But the execution failed with error:

SEVERE:Values for the following variables could not be obtained
from the command line or response file(s):
  METALINK_USERNAME(MetaLinkUsername)
Silent install cannot continue

Well, if it is not provided why not to use just null value. For some reason
installer did not want to do it and I added METALINK_USERNAME
(together with METALINK_PASSWORD) defined to empty string:

/u02/install/patch/Disk1> ./runInstaller -silent \
-responseFile /u02/install/patch/Disk1/response/patchset.rsp \
ORACLE_HOME="/u03/app/oracle/product/11.1.0/db_1" \
ORACLE_HOME_NAME="OraDb11g_home1" \
METALINK_USERNAME="" METALINK_PASSWORD=""


And the installation went further but stopped asking me:

You may not have enough space on your drive for a successful install.
Do you still want to continue?

Funny but I tried to answer "y", "Y", "YES" but it was failing with
"-bash: n: command not found". And then I checked the log
of installation: "INFO: Installation cannot continue. To Ignore Disk Warning, please use "-ignoreDiskWarning" command line option" was there.

I added -ignoreDiskWarning and was able to install patchset without problems
and questions

./runInstaller -silent\
 -responseFile /u02/install/patch/Disk1/response/patchset.rsp \
ORACLE_HOME="/u03/app/oracle/product/11.1.0/db_1" \
ORACLE_HOME_NAME="OraDb11g_home1" \
METALINK_USERNAME="" METALINK_PASSWORD="" \
-ignoreDiskWarning

Another thing that is worth to mention that root.sh script generated
during silent installation also has parameter not to show output:

grep -i silent /u03/app/oracle/product/11.1.0/db_1/root.sh
OUI_SILENT=true
...


With TRUE value it won't show anything on a screen during execution
but change to FALSE will provide the output as usual.


Have a good day!

silent installation of Enterprise Manager Grid Control 10.2.0.3 on Linux x86-64

The first full release of Grid Control 10g rel. 2 for Linux x86-64 is 10.2.0.3
To be prepared for future release 10.2.0.5 (I hope it will come soon) on x86-64
I have decided to try silent installation of Grid Control using existing database.

A database had been created, up and running together with listener on 1521 port.

Based on contents of the response file I added variables to start silent installation

/u02/install/gc/Disk1> ./runInstaller -silent \
-responseFile /u02/install/gc/Disk1/response/em_using_existing_db.rsp \
BASEDIR="/u02/grid" INSTALLATION_NAME="EMHome_1" \
s_reposHost="rac2" \
s_reposPort="1521" \
s_reposSID="repdb" \
s_reposDBAPwd="sys24" \
s_mgmtTbsName="/u01/oradata/repdb/mgmt.dbf" \
s_ecmTbsName="/u01/oradata/repdb/mgmt_ecm_depot1.dbf" \
s_reposPwd="man24" \
s_reposPwdConfirm="man24" \
s_securePassword="man24" \
s_securePasswordConfirm="man24" \
-ignoresysprereqs use_prereq_checker=false

Later it failed with:

SEVERE:OUI-10029:You have specified a non-empty directory
to install this product. It is recommended to specify either
an empty or a non-existent directory. You may, however, choose
to ignore this message if the directory contains Operating System
generated files or subdirectories like lost+found.

But why it happened, the directory defined under BASEDIR did not even exist!

Checking log of installation I found that home for OMS
was not defined like I wanted:

INFO: Setting variable 'ORACLE_HOME'
to '/u01/app/oracle/product/10.2.0/db_1/OracleHomes/oms10g'.
Received the value from the command line.

Ok, I defined ORACLE_HOME, ORACLE_HOME_NAME
pointing to the same location where I would like to install oms
and reran the installation again:

/u02/install/gc/Disk1> ./runInstaller -silent \
-responseFile /u02/install/gc/Disk1/response/em_using_existing_db.rsp \
ORACLE_HOME="/u02/grid/oms10g" ORACLE_HOME_NAME="oms10g" \
s_reposHost="rac2" \
s_reposPort="1521" \
s_reposSID="repdb" \
s_reposDBAPwd="sys24" \
s_mgmtTbsName="/u01/oradata/repdb/mgmt.dbf" \
s_ecmTbsName="/u01/oradata/repdb/mgmt_ecm_depot1.dbf" \
s_reposPwd="man24" \
s_reposPwdConfirm="man24" \
s_securePassword="man24" \
s_securePasswordConfirm="man24" \
-ignoresysprereqs use_prereq_checker=false

But when it started I noted that OMS and agent homes were set not
to locations that I supposed to see:

OMS Home    : /u01/app/oracle/product/10.2.0/db_1/OracleHomes/oms10g
                         Space Required: 2.4GB
AGENT Home: /u01/app/oracle/product/10.2.0/db_1/OracleHomes/agent10g
                         Space Required: 0.5GB

So I cancelled the installation and checked log file:

INFO: Setting variable 'ORACLE_HOME' to '/u02/grid/oms10g'.
Received the value from the command line.
INFO: Setting variable 'ORACLE_HOME_NAME' to 'oms10g'.
Received the value from the command line.
INFO: Setting variable 's_omsHomeName' to 'oms10g'.
Received the value from the command line.
INFO: Setting variable 's_omsHome'
to '/u01/app/oracle/product/10.2.0/db_1/OracleHomes/oms10g'.
Received the value from the command line.

Even with sets of ORACLE_HOME and ORACLE_HOME_NAME
oms and agent homes were set to some default location
using the same ORACLE_HOME as for repdb database.

Detailed analysis of the log file showed existence of variables:
s_omsHome, s_omsHomeName, s_agentHome, s_agentHomeName
I removed ORACLE_HOME and added variables defined above:

/u02/install/gc/Disk1> ./runInstaller -silent \
-responseFile /u02/install/gc/Disk1/response/em_using_existing_db.rsp \
s_omsHome="/u02/grid/oms10g" s_omsHomeName="oms10g" \
s_agentHome="/u02/grid/agent10g" s_agentHomeName="agent10g" \
s_reposHost="rac2" \
s_reposPort="1521" \
s_reposSID="repdb" \
s_reposDBAPwd="sys24" \
s_mgmtTbsName="/u01/oradata/repdb/mgmt.dbf" \
s_ecmTbsName="/u01/oradata/repdb/mgmt_ecm_depot1.dbf" \
s_reposPwd="man24" \
s_reposPwdConfirm="man24" \
s_securePassword="man24" \
s_securePasswordConfirm="man24" \
-ignoresysprereqs use_prereq_checker=false

After start and during the checks installer asked me about the space:

You may not have enough space on your drive for a successful install.
Do you still want to continue?

I canceled it again and found that ORACLE_HOME was pointed to the
home of RDBMS of the database that I used for repository:

Global Settings
    Source: /u02/install/gc/Disk1/oms/Disk1/stage/products.xml
    Oracle Home: /u01/app/oracle/product/10.2.0/db_1/OracleHomes/oms10g (oms10g)
    Installation Type: Core

Ok, there is nothing difficult to add ORACLE_HOME as well,
started installation again:

/u02/install/gc/Disk1> ./runInstaller -silent \
-responseFile /u02/install/gc/Disk1/response/em_using_existing_db.rsp \
ORACLE_HOME="/u02/grid/oms10g" ORACLE_HOME_NAME="oms10g" \
s_omsHome="/u02/grid/oms10g" s_omsHomeName="oms10g" \
s_agentHome="/u02/grid/agent10g" s_agentHomeName="agent10g" \
s_reposHost="rac2" \
s_reposPort="1521" \
s_reposSID="repdb" \
s_reposDBAPwd="sys24" \
s_mgmtTbsName="/u01/oradata/repdb/mgmt.dbf" \
s_ecmTbsName="/u01/oradata/repdb/mgmt_ecm_depot1.dbf" \
s_reposPwd="man24" \
s_reposPwdConfirm="man24" \
s_securePassword="man24" \
s_securePasswordConfirm="man24" \
-ignoresysprereqs use_prereq_checker=false

Everything was going ok but failed at "OMS configuration" step.
The main log did not show much information but since it failed
during execution of configuration tools I checked log files
in the directory where they are kept - $OMS_HOME/cfgtoollogs/cfgfw

There were messages about errors during connection
to the respository database:

WARNING: oracle.sysman.top.oms:getConnection::
Connection refused:rac2::repdb:Exception=Io exception:
Invalid number format for port number
INFO: oracle.sysman.top.oms:
Failed to connect to Database for s_SysUser=SYS,
s_encrSysPwd=******, connectString=rac2::repdb,
using default for node=rac2,sid=repdb.
ConnectException=Io exception:
Invalid number format for port number
INFO: oracle.sysman.top.oms:fixNodeNames:
The repository host name specified contains invalid format
for domain extention=rac2:Fixin is not done!
...
Could not connect to SYS/(DESCRIPTION=(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=rac2)(PORT=)))
(CONNECT_DATA=(SID=repdb))):
ORA-12533: TNS:illegal ADDRESS parameters
(DBD ERROR: OCIServerAttach)
...
INFO: oracle.sysman.top.oms:RepManager Create Repository Error = 50.
Please check the log file at /u02/grid/oms10g/sysman/log/
emca_repos_create11_45_57.log

"Invalid number format for port number" why is that?
The port had been defined so why it happened?

Search on Metalink revealed the note about the similar problem
Oms Config Failed with Error: Invalid number format for port number
and the workaround is simple - to remove s_reposPort from
command line and put it to the response file.

That was done (copied original file under /tmp and changed
s_reposPort to 1521) and I started installation again
(removed entries from inventory.xml and oratab,
deleted installed directories from /u02/grid):

/u02/install/gc/Disk1> ./runInstaller -silent \
-responseFile /tmp/em_using_existing_db.rsp \
ORACLE_HOME="/u02/grid/oms10g" ORACLE_HOME_NAME="oms10g" \
s_omsHome="/u02/grid/oms10g" s_omsHomeName="oms10g" \
s_agentHome="/u02/grid/agent10g" s_agentHomeName="agent10g" \
s_reposHost="rac2" \
s_reposSID="repdb" \
s_reposDBAPwd="sys24" \
s_mgmtTbsName="/u01/oradata/repdb/mgmt.dbf" \
s_ecmTbsName="/u01/oradata/repdb/mgmt_ecm_depot1.dbf" \
s_reposPwd="man24" \
s_reposPwdConfirm="man24" \
s_securePassword="man24" \
s_securePasswordConfirm="man24" \
-ignoresysprereqs use_prereq_checker=false

Now it was executed without problems, all assistances went through
and installation had been finished. I was able to connect to
Grid Control Console.

A couple of things to mention:
* 10.2.0.4 database was used for repository
but certified is 10.2.0.3 for Linux x86-64
* Linux release is CentOS 5 64-bit that's why
-ignoresysprereqs use_prereq_checker=false were used
* Got "Create Repository Error = 14" and "Create Repository Error = 12"
when not all objects of repository were cleaned from previous
installations.
To delete them:
declare
begin
for cr in (select * from dba_synonyms where table_owner = 'SYSMAN') loop
execute immediate 'drop public synonym '||cr.synonym_name;
end loop;
end;
/

drop user sysman cascade;

drop user mgmt_view cascade;

drop role MGMT_USER;

drop tablespace MGMT_TABLESPACE including contents and datafiles;

drop tablespace MGMT_ECM_DEPOT_TS including contents and datafiles;




I hope your silent installation will be successful!

Have a good day!

ORA-16401: archivelog rejected by RFS

How big is your list of "known" issues for systems that you support?

How many of them are related to bugs in Oracle
and what portion is the "expected" behaviour that covered
by existing technical/business requirements and processes?

How often do you revise the list to check of the "known" issues
can be fixed and removed from the "list"?

Every time when new job is taken first months are spent just to get
understanding what are the "known" issues/problems, how to react to them,
whom to call, what to write, etc. And since you are the new one here
you would like to show how cool and smart you are, you are trying to
fix the issues suggesting different approaches, methods. And when
time comes to do changes - you stuck with existing environment
and your strength is going away little by little and your list of "known" issues
is growing as well :(

How is the said before related to the error that is in the subject.
Well based on a description it can be ignored and there is nothing bad
to add it to ignore list or to the list of "known" issues.
Will be anything missed and hidden under that error? Who knows :)

In this case the error had periodic occurrence and after some checking
on primary side I found out that crontab job was doing backup of archive
log files at the same time time every hour and archive_lag_target was set
to 1800 (30 mins). So every second log file switch initiated by the parameter
coincided with switch initiated by RMAN backup command.

Difference of several seconds in log switches was too much for RFS to handle
quickly and was reported in alert log.

Tue Feb 24 00:05:02 2009
Primary database is in MAXIMUM PERFORMANCE mode
kcrrvslf: active RFS archival for log 1 thread 1 sequence 1117
RFS[12]: Successfully opened standby log 5: '+DATA/test/onlinelog/group_5.272.677123691'
Tue Feb 24 00:05:02 2009
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[21]: Assigned to RFS process 27094
RFS[21]: Identified database type as 'physical standby'
Primary database is in MAXIMUM PERFORMANCE mode
kcrrvslf: active RFS archival for log 2 thread 1 sequence 1118
RFS[12]: Successfully opened standby log 4: '+DATA/test/onlinelog/group_4.273.677123691'
Errors in file /u01/app/oracle/diag/rdbms/testb/TEST/trace/TEST_rfs_27094.trc:
ORA-16401: archivelog rejected by RFS


Finally I changed archive_lag_target to 2400 (40 mins)
and it helped to avoid consecutive log switches within seconds.

No errors so far in alert log and I believe that "known" issue
will be removed from the list soon.

Have a good day!

oracle recommended bundle patches

Every time when new oracle home is installed there is a wish
to apply latest patches and also some other patches to avoid
as much as possible problems/errors in the future.
I was installing new 10.2.0.4 oracle home recently and while
checking Metalink I found "bundle patches" for CRS, RAC
and for a database in general:

10.2.0.4 RAC RECOMMENDED BUNDLE #2
10..2.0.4 CRS BUNDLE2 TRACKING BUG
10.2.0.4 GENERIC RECOMMENDED BUNDLE# 3.

It is also worth doing to check bugs that appeared after the patchset
as for 10.2.0.4

10.2.0.4 Patch Set - Availability and Known Issues


Have a good day!

splitting table partitions with BLOB in streams

The last hours of the last day of the last year I spent trying to get
capture process of streams working.

Developers executed the script that prepared application tables
for new year splitting and rebuilding indexes on them.

After that capture process was aborted due to
ORA-04031 for streams pool.

The setting for stream_pool_size was 384M only but increase of
the value up to 1, 2 even 5Gb did not help either.

Capture was failing with the errors:

ORA-04031: unable to allocate ... bytes of shared memory ("streams pool","unknown object","streams pool","kol raw")
ORA-04031: unable to allocate ... bytes of shared memory ("streams pool","unknown object","streams pool","sob_kgqmrec")
ORA-04031: unable to allocate ... bytes of shared memory ("streams pool","unknown object","streams pool","kolccst obj")
ORA-04031: unable to allocate ... bytes of shared memory ("streams pool","unknown object","streams pool","sob_kgqmrec")


After research on Metalink one interesting note had been found:

STREAMS CAPTURE ERRORS OUT WITH ORA-4031 ON SPLIT PARTITION COMMAND

which exactly describes the problem that we have.
But they were able to go through all transactions for capture process
only when streams_pool_size was increased up to 16Gb!
Unfortunately there is no chance for now to allocate
so much memory for streams and there is no patch for it yet.

Might be upgrade (from 10.2.0.4) to 11g will help,
might be additional memory will be found
or Metalink will come with some ideas, will see...

Have a good day!

orapwd and standby in 11g

I was trying to build standby for Oracle 11g database to test new
features of Data Guard and stuck with problem that archive logs had
not been transferred to standby side because of the following error:

Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
      returning error ORA-16191
------------------------------------------------------------

The password files have been created on both sides using orapwd:
orapwd file=orapwdb11 password=sys
and remote_login_passwordfile parameters were set to EXCLUSIVE.
And I was able to login as sysdba to remote sites using either TNS or
easy naming methods.

After different attempts I recreated password files with password
in upper-case (thinking about case sensitivity for password in 11g):
orapwd file=orapwdb11 password=SYS force=y
and it worked - primary started to send logs to standby.

Trying to find the reason of such behavior I recognized that there is
the parameter that responsible for password sensitivity in 11g -
SEC_CASE_SENSITIVE_LOGON.
I set it to FALSE, changed passwords to lower-case, bounced databases
and... the same problem as before :(

That meant the parameter does not influence on case sensitivity of
password during sysdba connection.

And only after that case I found out that there is a new parameter for
orapwd utility - ignorecase

Usage: orapwd file= password= entries= force= ignorecase= nosysdba=

  where
    file - name of password file (required),
    password - password for SYS (optional),
    entries - maximum number of distinct DBA (required),
    force - whether to overwrite existing file (optional),
    ignorecase - passwords are case-insensitive (optional),
    nosysdba - whether to shut out the SYSDBA logon (optional Database Vault only).

I changed SEC_CASE_SENSITIVE_LOGON back to TRUE,
recreated password file with lower-case characters and ignorecase=y,
bounced databases and everything went smoothly!

Have a good day!

switching $ORACLE_BASE using oraenv in 11g

I recently installed a couple of Oracle 11g on VM. First one was installed
to /u01 but after testing different things I added one more to /u05.
But it does not really matter, the main thing that 11g raised importance
of $ORACLE_BASE variable added it to different places - oraenv script
(you have noticed sentence about oracle base after execution of the script)
and as parameter to init file.

All that means the $ORACLE_BASE should be dependent on
ORACLE_HOME and be saved somewhere. Moreover the script should
change it! But it was not so. Everytime I ran it $ORACLE_BASE left the
same even they based on the same path as $ORACLE_HOME:

[oracle@oel1 bin]$ . oraenv
ORACLE_SID = [db11106] ?
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u05/app/oracle
[oracle@oel1 bin]$ . oraenv
ORACLE_SID = [db11106] ? db11a
The Oracle base for ORACLE_HOME=/u05/app/oracle/product/11.1.0/db_2 is /u05/app/oracle
[oracle@oel1 bin]$ . oraenv
ORACLE_SID = [db11a] ? db11106
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u05/app/oracle

Checking the script I found the part that responsible for
oracle base change and comments there:


# Set the value of ORACLE_BASE in the environment. Use the orabase
# executable from the corresponding ORACLE_HOME, since the ORACLE_BASE
# of different ORACLE_HOMEs can be different.
# The return value of orabase will be determined based on the following :
#
# 1. Value of ORACLE_BASE in the environment.
# 2. Get the value of ORACLE_BASE from oraclehomeproperties.xml as
# set in the ORACLE_HOME inventory.



Going further I found that oraclehomeproperties.xml is under
$ORACLE_HOME/inventory/ContentsXML and it contains properties
and one of them is:

...
property name="ORACLE_BASE" val="/u05/app/oracle"
...

(/u01/app/oracle was for the other oracle home)

That's the place where $ORACLE_BASE and $ORACLE_HOME variable
are put together. But the script doesn't switch base if it was already set.
And the reason of it - orabase script ($ORACLE_HOME/bin).
If you have $ORACLE_BASE value set - orabase returns this value.
If not - it gets value from oraclehomeproperties.xml file:

[oracle@oel1 bin]$ . oraenv
-- switched to another home and got wrong $ORACLE_BASE
ORACLE_SID = [db11a] ? db11106
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u05/app/oracle
-- wrong value
[oracle@oel1 bin]$ echo $ORACLE_BASE
/u05/app/oracle
-- orabase returns wrong value
[oracle@oel1 bin]$ which orabase
/u01/app/oracle/product/11.1.0/db_1/bin/orabase
[oracle@oel1 bin]$ orabase
/u05/app/oracle
-- unset $ORACLE_BASE
[oracle@oel1 bin]$ export ORACLE_BASE=
[oracle@oel1 bin]$ echo $ORACLE_BASE

-- and now orabase returns right value
[oracle@oel1 bin]$ orabase
/u01/app/oracle

Finally I added the "export ORACLE_BASE=" entry before
"ORABASE_EXEC=$ORACLE_HOME/bin/orabase" to null the value that
orabase could accomodated the real value of $ORACLE_BASE variable.

[oracle@oel1 bin]$ vi /u01/app/oracle/product/11.1.0/db_1/bin/oraenv
[oracle@oel1 bin]$ vi /u05/app/oracle/product/11.1.0/db_2/bin/oraenv
[oracle@oel1 bin]$ . oraenv
ORACLE_SID = [db11106] ?
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u01/app/oracle
[oracle@oel1 bin]$ . oraenv
ORACLE_SID = [db11106] ? db11a
The Oracle base for ORACLE_HOME=/u05/app/oracle/product/11.1.0/db_2 is /u05/app/oracle

Have a good day!