Em Agent install error: org.xml.sax.SAXParseException:

Last week I was installinging the OEM Agent 10.2.0.5.0 on a RedHat 5.4 x64 machine and got the following error message:

org.xml.sax.SAXParseException: <Line 1, Column 37>: XML-20201: (Fatal Error) Expected name instead of ?.
	at oracle.xml.parser.v2.XMLError.flushErrorHandler(XMLError.java:415)
	at oracle.xml.parser.v2.XMLError.flushErrors1(XMLError.java:284)
	at oracle.xml.parser.v2.XMLReader.scanNameChars(XMLReader.java:1101)
	at oracle.xml.parser.v2.XMLReader.scanQName(XMLReader.java:1911)
	at oracle.xml.parser.v2.NonValidatingParser.parseAttr(NonValidatingParser.java:1515)
	at oracle.xml.parser.v2.NonValidatingParser.parseAttributes(NonValidatingParser.java:1465)
	at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1304)
	at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:353)
	at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:299)
	at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:213)
	at oracle.sysman.oii.oiii.OiiiInstallXMLReader.readHomes(OiiiInstallXMLReader.java:126)
	at oracle.sysman.oii.oiii.OiiiInstallInventory.readHomes(OiiiInstallInventory.java:629)
	at oracle.sysman.oii.oiii.OiiiInstallAreaControl.loadPartialInstallInv(OiiiInstallAreaControl.java:656)
	at oracle.sysman.oii.oiii.OiiiInstallAreaControl.initInstallInv(OiiiInstallAreaControl.java:701)
	at oracle.sysman.oii.oiii.OiiiInstallAreaControl.loadInstallInventory(OiiiInstallAreaControl.java:568)
	at oracle.sysman.oii.oiii.OiiiInstallAreaControl.initAreaControl(OiiiInstallAreaControl.java:1785)
	at oracle.sysman.oii.oiii.OiiiInstallAreaControl.initAreaControl(OiiiInstallAreaControl.java:1738)
	at oracle.sysman.prov.prereqs.OracleInventory.initInventorySession(OracleInventory.java:65)
	at oracle.sysman.prov.prereqs.OracleInventory.init(OracleInventory.java:44)
	at oracle.sysman.prov.prereqs.OracleInventory.<init>(OracleInventory.java:34)
	at oracle.sysman.prov.prereqs.OracleInventory.<clinit>(OracleInventory.java:28)
	at oracle.sysman.prov.prereqs.PortHandlerUtil.getBusyPorts(PortHandlerUtil.java:53)
	at oracle.sysman.prov.prereqs.PortChecks.checkPortAvailability(PortChecks.java:104)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at oracle.sysman.oip.oipc.oipcr.OipcrRulesEngine.executeRule(OipcrRulesEngine.java:325)
	at oracle.sysman.oip.oipc.oipcp.OipcpPrereqChecker.executeCheck(OipcpPrereqChecker.java:495)
	at oracle.sysman.oip.oipc.oipcp.OipcpPrereqChecker.runChecks(OipcpPrereqChecker.java:450)
	at oracle.sysman.oip.oipc.oipcp.OipcpPrereqChecker.executePrereqs(OipcpPrereqChecker.java:351)
	at oracle.sysman.oip.oipc.oipcc.OipccPreReqSession$1.run(OipccPreReqSession.java:332)
	at java.lang.Thread.run(Unknown Source)

After some searching I found that my inventory was corrupt. There are tools available to restore / recreate the inventory. Instead I removed the inventory and let the OUI recreate it, when the installation finished I added the Oracle Home of the database that was installed on this machine and everything is working as it should be.

For restoring / recreating your Oracle Inventory see: http://download.oracle.com/docs/cd/B19306_01/em.102/b16227/oui3_manage_oracle_homes.htm#CJAGHABI

Cleanup log files

Yesterday I wrote a script which can clean file locations by using a catalogfile. In this catalog file are locations defined with retention min size, etc.

I wrote this script for an environment where we have to clean over 60 location daily, so you can understand we wanted to automate it. Below you’ll find the script and an example of the catalog.

rm_file.sh:

#!/bin/sh
#
# [21-07-2009] rudi.heinen    - Initial Creation
#
# Cleans and removes files which are defined in the catalogfile

# Global variables
base="`echo $(dirname $(which $0)) | rev | cut -d '/' -f1- | rev`"

#Help
function help {
	echo "Usage `basename $0`"
	echo ""
	echo "Cleans and removes files."
	echo " -c Clean folders defined in the catalog."
	echo " -o Logfile when using the catalog"
	echo " -f Clean file."
	echo " -l Lines to save in file."
	echo ""
	echo "Example catalog file:"
	echo " <search path>;<retention in days>;<size in Kilo bytes>;<search parameter>;<max directory depth>"
	echo " ex: /user_home/log;0;500;log;1"
	echo " For comments begin line with #"
	echo " When retention 0 then file is cleaned instead of removed."
	echo ""
	echo "`basename $0 -c` relies on a catalogfile!!"
	echo " ex: `basename $0` -c <catalogfile> (-o <logfile>)"
	echo "Or you can clean / remove a file, if lines are not defined, file is removed."
	echo " ex: `basename $0` -f <file> (-l <lines>)"
	exit 1
}

#Clean / remove files
function cleanfile {
	# $1 = File     # $2 = Lines
	if [ -z $2 ]; then
		# Check with fuser if file is open
		if [[ `/sbin/fuser -u $1` == ""  ]]; then
			rm $1
		else
			> $1
		fi
	else
		tail -n$2 $1 1>/tmp/cleanfile.$$ 2>/dev/null
		if [ $? == 0 ]; then
			cat /tmp/cleanfile.$$ > $1
		else
			echo "Error cleaning file $1."
		fi
		test -f /tmp/cleanfile.$$ && rm /tmp/cleanfile.$$
	fi
}

#Getfile from catalogfile function
function cleancatalog {
	# $1 = catalogfile      # $2 = logfile
	if [ -z $2 ]; then
			cleanlog="/dev/null"
	else
			test -f $2 && > $2
	fi
	if [[ -z $1 || -e $1 ]] ; then
		for line in `cat $1 | grep -v "#"`; do
			x=0
			for l in `echo ${line} | tr ';', ' '`;do
				cat[$x]=$l
				((x=x+1))
			done
			files=`test -e ${cat[0]} && find ${cat[0]} -name "*${cat[3]}*" -size +${cat[2]}k -mtime +${cat[1]} -maxdepth ${cat[4]}`
			for file in ${files}; do
				if [[ "${path}" != "${file}" ]];then
					if [[ ${cat[1]} == "0" ]];then
						echo "- `ls -ltrh ${file}` will be cleaned" >> $2
						cleanfile ${file} 8000
					else
						echo "* ${file} will be removed" >> $2
						cleanfile ${file}
					fi
				fi
			done
		done
	else
		echo "Catalog file does not exists!"
		echo -e "Create one.\n"
		help
	fi
}

#Main script
while getopts "c:f:l:ho:" options; do
	case $options in
		h) help;;
		c) catalog=$OPTARG;;
		f) dfile=$OPTARG;;
		l) rline=$OPTARG;;
		o) log=$OPTARG;;
		*) help;;
	esac
done

if [ -z ${catalog} ]; then
	if [ -z ${dfile} ]; then
		help
	else
		cleanfile ${dfile} ${rline}
	fi
else
	cleancatalog ${catalog} ${log}
fi

Catalog:

#Catalog file for rm_file.sh
#<search path>;<retention in days>;<size in Kilo bytes>;<search parameter>;<max directory depth>

#Datapump locations
/backup/data_pump;7;0;.dmp.gz;1
/backup/data_pump;7;0;.log;1

#Portal
/oracle/portal/opmn/logs;7;1000;log;1
/oracle/portal/Apache/Apache/logs;7;0;log;1
/oracle/portal/j2ee/oc4j/logs;7;1000;log;2

503 Service Unavailable when opening OEM

Today I had to solve a problem with one of our clients Oracle Enterprise Manager Grid Control. When they went to the console, they got the following error:

503 Service Unavailable
Servlet error: Service is not initialized correctly. Verify that the repository connection information provided is correct.

When I looked in the emoms.log in $ORACLE_HOME/sysman/log I found an error mentioning that the user credentials where incorrect. When I checked the status of the database, listener and opmn, everything was up and running. After some searching a found that the file emoms.properties was empty. I found 2 Metalink notes (418159.1 and 733401.1) concerning this issue and it appears to be a bug in OEM 10.2.0.4.0.

I did the following to solf this:
I created a new emoms.properties in $ORACLE_HOME/sysman/config with the following

oracle.sysman.emSDK.svlt.ConsoleServerName=<AS_server>\:<OEM_console_port>_Management_Service
emdrep.ping.pingCommand=</bin/ping -c 3 -w 30 <hostname>
em_oob_shutdown=false
LargeRepository=false
oracle.sysman.eml.mntr.emdRepPort=1521
em_email_address=_NOT_AVAILABLE_
em_oob_crash=false
em.oms.dumpModules=omsThread,repos
oracle.sysman.emRep.dbConn.statementCacheSize=50
oracle.sysman.db.isqlplusUrl=%ISQLPLUS_PROTOCOL%\://%ISQLPLUS_HOST%\:%ISQLPLUS_PORT%/isqlplus/dynamic
em_oob_startup=false
oracle.sysman.emSDK.svlt.ConsoleServerPort=4889
em_from_email_address=_NOT_AVAILABLE_
em_from_email_name=_NOT_AVAILABLE_
oracle.sysman.emSDK.svlt.ConsoleServerHost=<AS_server>
oracle.sysman.db.isqlplusWebDBAUrl=%ISQLPLUS_PROTOCOL%\://%ISQLPLUS_HOST%\:%ISQLPLUS_PORT%/isqlplus/dba/dynamic
oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort=1159
em_email_gateway=_NOT_AVAILABLE_
oracle.sysman.eml.mntr.emdRepSID=<Repository Database SID>
oracle.sysman.eml.mntr.emdRepServer=<DB_server>
oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=(ADDRESS_LIST\=(ADDRESS\=(PROTOCOL\=TCP)(HOST\=<DB_server>)(PORT\=1521)))(CONNECT_DATA\=(SERVICE_NAME\=<Database Service Name>)))
oracle.sysman.emkeyfile=<$ORACLE_HOME>/sysman/config/emkey.ora
em.ip.ui.enable=true
oracle.sysman.emSDK.svlt.PublicServletEnabled=true
oracle.sysman.eml.mntr.emdRepPwd=<sysman password>
oracle.sysman.eml.mntr.emdRepUser=sysman
oracle.sysman.eml.mntr.emdRepPwdEncrypted=FALSE
oracle.sysman.eml.mntr.emdRepPwdSeed=4100593566537048326

Normally the password is encrypted, but by setting emdRepPwdEncrypted to FALSE you can put in your password as cleartext. After restarting opmn the password will by encrypted and emdRepPwdEncrypted set to TRUE. You can use this to change the password od sysman as well. Just change the password in the existing emoms.properties and set emdRepPwdEncrypted to FALSE and restart opmn.

So in sort:
- Change / create emoms.properties
- opmnctl shutdown
- opmnctl startall
And everything should be working again.

Install Enterprise Manager on OEL5.3 x86_64

Today I had to install Oracle Enterprise Manager Grid Control 10.2.5.0 on an Oracle Enterprise Linux 5.3 64 bit machine, which I installed also. I installed a base install of linux, I hate to have X installed on a server where it’s not used. Basically I ran a default install, but deselected Gnome, X Server and all those other X tools like Firefox.

After I have done that I had to install some extra packages beside the standard packages Oracle needs. Below the commands that I used so I got start the OEM installation.

First add the public yum of Oracle to your repository

Install needed packages

yum install libXp.x86_64 compat-db.x86_64 control-center.x86_64 gcc.x86_64 gcc-c++.x86_64 libstdc++-devel.i386 libstdc++-devel.x86_64 openmotif.x86_64 openmotif.i386 sysstat.x86_64 glibc-devel.i386 gdbm.i386

Gdbm, glibc and libXp are extra packages, otherwise you get errors that the HTTP_Server couldn’t start, or the setup is unable to continue.

Create link libdb

ln -s /usr/lib/libgdbm.so.2.0.0 /usr/lib/libdb.so.2
chmod 755 /usr/lib/libgdbm.so.2.0.0
chmod 755 /usr/lib/libdb.so.2

Add oracle user

groupadd oinstall
groupadd dba
useradd -m -g oinstall -G dba oracle

change /etc/sysctl.conf

cat >> /etc/sysctl.conf <<EOF
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144
EOF
/sbin/sysctl -p

Change /etc/security/limits.conf

cat >> /etc/security/limits.conf <<EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF

Change /etc/pam.d/login

cat >> /etc/pam.d/login <<EOF
session required /lib/security/pam_limits.so
EOF

Change /etc/profile

cat >> /etc/profile <<EOF
if [ \$USER = "oracle" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF

Change /etc/csh.login

cat >> /etc/csh.login <<EOF
if ( \$USER == "oracle" ) then
limit maxproc 16384
limit descriptors 65536
umask 022
endif
EOF

Auto login with SSH

This something I use a lot and is very simple and you can find it any where, but i always forget the commands and have to look it up again.

This is how you can jump from machine to machine with SSH without password verification:

[client:~ ] test -f ~/.ssh/id_dsa.pub || ssh-keygen -t dsa
[client:~ ] scp ~/.ssh/id_dsa.pub user@server:~
[client:~ ] ssh user@server
[server:~ ] cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.backup
[server:~ ] cat ~/id_dsa.pub >> ~/.ssh/authorized_keys
[server:~ ] rm ~/id_dsa.pub
[server:~ ] logout

Failed to get ESB_HOME

Last week I had to deploy an application on a Oracle SOA Suite cluster. This is something I’ve done frequently the last few weeks and was never a problem until yesterday. Normaly when you deploy an application on IAS you deploy it to an OC4J container, when deploying to a clsuter you deploy it to the group where the containers are memeber of.

So when I deployed this application everything seemd fine, but when I look closer I saw that not all the BPEL processes and ESBs where deployed. When I looked in the opmn log of the esb container I saw the following error:

log4j:WARN No appenders could be found for logger (wsif).
log4j:WARN Please initialize the log4j system properly.
Warning: Unable to set up connection factory for a resource adapter in esb-dt: java.lang.RuntimeException: failed to get ESB_HOME: java.lang.NullPointerException
Oracle Containers for J2EE 10g (10.1.3.4.0)  initialized
oc4jadmin********************request.isUserInRole(oc4j-administrators))=true
oc4jadmin********************request.isUserInRole(oc4j-administrators))=true

Also when I deleted the BPEL Domain it didn’t removed from the ESB. I got the following error in the log.xml of th ESB:

<MESSAGE>
<HEADER>
<TSTZ_ORIGINATING>2009-05-12T09:32:22.993+02:00</TSTZ_ORIGINATING>
<COMPONENT_ID>tip</COMPONENT_ID>
<MSG_TYPE TYPE="ERROR"></MSG_TYPE>
<MSG_LEVEL>1</MSG_LEVEL>
<HOST_ID>(HOSTNAME)</HOST_ID>
<HOST_NWADDR>(IP)</HOST_NWADDR>
<MODULE_ID>esb.server.dispatch</MODULE_ID>
<THREAD_ID>16</THREAD_ID>
<USER_ID>oracle</USER_ID>
</HEADER>
<CORRELATION_DATA>
<EXEC_CONTEXT_ID><UNIQUE_ID>10.9.8.38:17610:1242113515884:3</UNIQUE_ID><SEQ>0</SEQ></EXEC_CONTEXT_ID>
</CORRELATION_DATA>
<PAYLOAD>
<MSG_TEXT>Destroying JMSDequeuer failed</MSG_TEXT>
<SUPPL_DETAIL><![CDATA[javax.jms.InvalidDestinationException: JMS-235: Can not link the durable subscriber name with a topic in unsubscribe method.
at oracle.jms.AQjmsSession.unsubscribe(AQjmsSession.java:3994)
at oracle.j2ee.ra.jms.generic.SessionWrapper.unsubscribe(SessionWrapper.java:455)
at oracle.tip.esb.server.dispatch.JMSDequeuer.unsubscribeDeferredSubscriber(JMSDequeuer.java:513)
at oracle.tip.esb.server.dispatch.JMSDequeuer.destroy(JMSDequeuer.java:453)
at oracle.tip.esb.server.dispatch.JMSDequeuer.destroy(JMSDequeuer.java:442)
at oracle.tip.esb.server.dispatch.agent.ESBWork.stop(ESBWork.java:185)
at oracle.tip.esb.server.bootstrap.DesignTimeResourceAdapter.stop(DesignTimeResourceAdapter.java:161)
at oracle.j2ee.connector.ResourceAdapterWrapper.stop(ResourceAdapterWrapper.java:266)
at com.evermind.server.connector.deployment.ConnectorArchive.stopResourceAdapter(ConnectorArchive.java:2192)
at com.evermind.server.connector.deployment.ConnectorArchive.stop(ConnectorArchive.java:2156)
at com.evermind.server.connector.deployment.ConnectorArchive.destroy(ConnectorArchive.java:2108)
at com.evermind.server.ApplicationStateRunning.stopOrDestroyActiveRunningStateObjects(ApplicationStateRunning.java:1102)
at com.evermind.server.ApplicationStateRunning.destroyActiveRunningStateObjects(ApplicationStateRunning.java:1051)
at com.evermind.server.Application.stateCleanUp(Application.java:3633)
at com.evermind.server.Application.destroy(Application.java:792)
at com.evermind.server.ApplicationServer.destroyApplications(ApplicationServer.java:2289)
at com.evermind.server.ApplicationServer.destroy(ApplicationServer.java:2086)
at com.evermind.server.ApplicationServerShutdownHandler.run(ApplicationServerShutdownHandler.java:93)
at java.lang.Thread.run(Thread.java:595)
]]></SUPPL_DETAIL>
</PAYLOAD>
</MESSAGE>

Solution to this problem was to set the opmn ports of both tiers of the cluster the same, some how the cluster does work when this is not the case, except from some part..

Stop the AS tier, check for a free port with netstat and change the opmn.xml. Set the local, request and remote ports of both tiers to the same value. In other words, set request to 6010, local to 6110 en remote to 6210 on both tiers.

Using mod_osso with Oracle AS

When you are using OssoConfigFile for virtualhost in Apache you can get the following error in the error log when you approach the server over an other URL than the one specified in the virtualhost. The following error is logged in the error log:

\n[OSSO] W14: mod_osso is loaded but OssoConfigFile does not appear to be set.\n

To solve this issue you have to put the following line in the httpd.conf, so the default page uses osso also.

OssoConfigFile {Location}/{osso_file}.conf

Put this line directly under the include of mod_osso, not in the virtualhost tag

Getting error during deployement of Oracle WSM

Today I redeployed the Oracle WSM with the wsmadmin script. During deployement I got the following error:

oc4j.checkSharedLibs:
[echo] oracle.wsm.ccore exists
[echo] oracle.wsm.coreman exists

deploy.runScript:
[echo] Configuring OC4J using commands:
..
[java] Caught exception running the command via the script file /appl/oracle/soa_cl_10.1.3/owsm/bin/admin_client.txt at line 1: PublishSharedLibrary error: publishSharedLibrary failed: Error publishing shared library to group.
BUILD FAILED
/appl/oracle/soa_cl_10.1.3/owsm/scripts/install.xml:2022: The following error occurred while executing this line:
/appl/oracle/soa_cl_10.1.3/owsm/scripts/deploy.xml:329: The following error occurred while executing this line:
/appl/oracle/soa_cl_10.1.3/owsm/scripts/deploy.xml:209: The following error occurred while executing this line:
/appl/oracle/soa_cl_10.1.3/owsm/scripts/deploy.xml:137: The following error occurred while executing this line:
/appl/oracle/soa_cl_10.1.3/owsm/scripts/oc4j.xml:477: Java returned: 1

Total time: 26 seconds
Error 1

Solution:
Remove the following shared libraries from the container:
oracle.wsm.ccore
oracle.wsm.coreman
oracle.wsm.policymanager
oracle.wsm.gateway

Some how Oracle does not regonise the the installed libraries are of the same version it wants to install.

Backup and restore your Oracle IAS

Today I was faced with a challange to restore an Oracle AS installation, in this case a SOA Suite 10.3.4.0 mlr 4.
Every day we make a configuration backup of our installation and when we install a new tier we create a root (node) backup and an on-line backup. How this is done is described below. What I found today is that when I restored the tier from the root backup and ran the restore from the configuration backup, the newly create containers including their apps where not restore. Thus when you create a new container, run an on-line or off-line backup off your tier!

In the following examples I assume that the database is running an a separate machine and that RMAN, or an other backup utility is used to backup and restore your database.

Configuring and creating as backup

Following command are used to configure your tier for as backup:

cd $ORACLE_HOME/backup_restore/config
cp config.inp config.inp.org
vi config.inp

Change/add:

log_path={AS_BACKUP_PATH}/log
config_backup_path={AS_BACKUP_PATH}

cd $ORACLE_HOME/backup_restore/
./bkp_restore.sh -m configure_nodb (Create backup without database)
./bkp_restore.sh -v -m node_backup -o prepare (Prepare your tier for backup)

Create the root (node) backup

su - root
export ORACLE_HOME=<oracle_home>
mkdir -p {BACKUP_PATH_AS}/node
cd {ORACLE_HOME}/backup_restore
./bkp_restore.sh -v -m node_backup -o image_backup -P {BACKUP_PATH_AS}/node/

Commands below are used to create on-line and config backups.

./bkp_restore.sh -v -m backup_instance_cold

./bkp_restore.sh -v -m backup_instance_online

Restore as backup

Restore your root (node) backup:

su - root
export ORACLE_HOME=/appl/oracle/{AS_HOME}
cd /
tar -xvpf {BACKUP_PATH_AS}/node/{ARCHIVE_NAME}

Create a new oratab ant oui metadata

cd $ORACLE_HOME/backup_restore
./bkp_restore.sh -m node_restore -o sys_init

Register the restored tier in the inventory and oratab

./bkp_restore.sh -m node_restore -o inst_register

Log in as the oracle user and start restore from the online/config backup

./bkp_restore.sh -m node_restore -o inst_reconfigure -t config_bkp_timestamp

With the above command you will get a list of availeble backups (timestamps), choose one and execute the following command

./bkp_restore.sh -m node_restore -o inst_reconfigure -t {TIMESTAMP}

Normaly oracle will ask you to restore the opmn state of the last backup, witch means oracle will start the AS for you.

Keep in mind the oracle will not clean-up old as backups (retention) like RMAN. For this I have created a script and is explained earlier.

Remove Database including RMAN

Following shows how to remove a database including the registration in the RMAN catalog.

$ sqlplus "/ as sysdba"

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount exclusive
ORACLE instance started.

Total System Global Area  285212672 bytes
Fixed Size                  1218992 bytes
Variable Size             100664912 bytes
Database Buffers          180355072 bytes
Redo Buffers                2973696 bytes
Database mounted.
SQL> alter system enable restricted session;
System altered.
SQL> exit

$ rman target /
RMAN> drop database including backups;