Clone a WebSphere Portal Cell

Clone a WebSphere Portal Cell

This post will describe how to move a WebSphere ND Cell with a managed WebSphere Portal cluster from one or more servers
to another Dmgr and one or more cluster members allowing for hostname change during move.
If required, it is also simple to create more, or less number of target cluster members in the target cell,
but it is out of the scope for this document. If you after gone through thees steps and still don't see the
pattern of cell/server and relationship, send me a message and I'll see if I can provide a follow-up document.

I also want to state that this instruction by no means is perfect, I've done this operation a couple of times, so I may have been blind about some of the manual steps I myself perform to complete my "clone".

The steps shown here could for example be beneficial to automatically create a WebSphere (not necessarily Portal) backup cluster.
PS. If a full clone where to be used as a backup-cluster, make sure you modify the unique cluster member ID's, since the cluster members in the newly created cell will have inherited the original CloneID's.
Read more about this in InfoCenter: HttpSessionCloneId
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/rprs_custom_properties.html

This is also valid for any other WebSphere Application Server Cell-moving, not only WebSphere Portal Clusters.
Although I think few other application providers and developers are able to entangle and create relations between
layers like WebSphere Portal. The Portal instance, the managed Portal cluster in the Cell and the database are
tightly connected and you cant simply move or rename cell members or portal instances which has meta-data all over.
One of the benefits from performing this in a Portal installation, is that it is possible to keep all development, test and production systems identical in terms of WAR/EAR relationships and names. You can deploy the "same" EAR in a test-environment and after successful tests you can deploy the "same" EAR in production and guarantee that it contain the same applications. This is otherwise a struggle in this environment where customization of code and copying portlets render unique WAR-names which can cause confusion.
Please, do not think that this make xmlaccess & releasebuilder obsolete. It is mearly to be seen as a compliment. Releasebuilder is still a good way to persist changes..

The trick is to get all application EARs, WARs and portlets

( deployed Web-application configured via the Portal webadmin interface ) in one single chunk and copy it "as is" to the new cell.
I've done trials to take only Database, Cell and WebSphere Portal configuration and re-deploying all applications
from a Dmgr perspective, but the meta-data in the database ( seen from within Portal webadmin ) is still floating around,
so unless you want to action-empty-portal, drop-database-tables-all or similar to start out fresh this is a bad idea.

Everything in a WebSphere Cell is configurable, and by changing the master data (xml) on the Dmgr, one can change anything.
But, beware! keeping ID's consistent is crucial. One false move and all work could be in vain.
Make sure you know what you are changing and what impact it has.
Sure, the IBM WebSphere Console has a useful configuration validation tool, and with it in its maximum validation
setting you can get any valid configuration to report problems! :-)

My first choice would have been editing everything from within Emacs,
( since there are a limited numbers of files and a simple macro would have replaced Everything upon open,
even deployment descriptors within deployed EARs. ) but for simplicity, and not at least for the sake of
exemplifying the work in this post. I will provide a sample script which replaces all names and references in the files
that need to be updated. The shortlist is xml, sh/bat files and properties.
The script can be seen last in this post.

If you are one of the unlucky people that has to manage WebSphere and/or WebSphere Portal on a Microsoft platform! I only say this..
Install cygwin and run the script from within a shell.. either using a mounted file-system or having a local copy of the filesystems.


1. Backup Cell
Backup the source Cell using backupConfig.bat/sh. ( better safe than sorry.. )

# cd /opt/IBM/WebSphere/AppServer/bin
# call ../profiles/Dmgr01/bin/setupCmdLine.sh
# ./backupConfig Dmgr01_backupConfig.jar -username [Admin] -password [password] -profileName Dmgr01

2. Backup and restore database
Backup all databases and transfer to the new database server with same names, or if rename is required,
make sure these changes is reflected in the substitution-script.

WARNING. If you have a database called WMMDB, I'm sure you understand that you can't simply search&replace all XML and
properties files with 's/WMMDB/NEWWMMDB/g' for example.
Since the string "WMMDB" may occur in many places.. make sure you know what you are doing and
what patterns you are able to automatically substitute.
For this specific example you would substitute 's/wmm.DataSourceName=WMMDB/wmm.DataSourceName=NEWWMMDB/g'

3. Backup and restore the individual nodes
Backup/restore using whatever means you can, I prefer to copy whole directory-structures.

# cd /opt/IBM
# find WebSphere | cpio -oc | remsh [target_server] "(cd /opt/IBM ; cpio -icvDumb)"

For each node-type:

Source Destination
===========================================
source Dmgr target Dmgr
source primary Portal Node target primary Portal-node
source web-server target web-servers
and so on..

4. Running substitution-script.
The magic should be performed on the dmgr under [profile_home]/config/cells/ and if WebSphere Portal is used,
under [portal_home]/config/ and [was_home]/profiles/wp_profile/ on each Portal node.

But since it is basically only *.bat/sh, xml and properties/props files it doesn't take that long to run
the script on the whole [portal_home] and [profile_home]
and if you thorough in your substitution-specification it will repay itself later.
In a WebSphere ND environment all "WebSphere" properties are federated into the cell configuration,
so only the wsadmin.properties and maybe sas/soap.client.props contains valid info.

PS. If you are using the default keys you are in the clear, otherwise you have to manage the certificates manually.


Here is an example of what is being substituted in the deployed application:
file://[dmgr_profile]/config/cells/Cell02/applications/wps.ear/deployments/wps/deployment.xml

The sample contains tags, <\appdeployment> & <\deployedObject> : remove backslash "\" on tag in real code..

** initial values **
<\appdeployment:Deployment xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:appcfg="http://www.ibm.com/websphere/appserver/schemas/5.0/appcfg.xmi" xmlns:appdeployment="http://www.ibm.com/websphere/appserver/schemas/5.0/appdeployment.xmi" xmi:id="Deployment_1172582367002"\>
<\deployedObject xmi:type="appdeployment:ApplicationDeployment" xmi:id="ApplicationDeployment_1172582367002" deploymentId="0" startingWeight="10" binariesURL="$(APP_INSTALL_ROOT)/Cell01/wps.ear" useMetadataFromBinaries="false" enableDistribution="true" createMBeansForResources="true" reloadEnabled="false" appContextIDForSecurity="href:Cell01/wps"\>

** new values **
<\appdeployment:Deployment xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:appcfg="http://www.ibm.com/websphere/appserver/schemas/5.0/appcfg.xmi" xmlns:appdeployment="http://www.ibm.com/websphere/appserver/schemas/5.0/appdeployment.xmi" xmi:id="Deployment_1172582367002">
<\deployedObject xmi:type="appdeployment:ApplicationDeployment" xmi:id="ApplicationDeployment_1172582367002" deploymentId="0" startingWeight="10" binariesURL="$(APP_INSTALL_ROOT)/Cell02/wps.ear" useMetadataFromBinaries="false" enableDistribution="true" createMBeansForResources="true" reloadEnabled="false" appContextIDForSecurity="href:Cell02/wps">

..another catch. If you are using one dedicated server for all your deployment-managers,
make sure you update the Cell configuration for all ports to avoid port conflicts.
If automatic substitution is being done: use syntax: 's/port="8879"/port="8880"/g' for all unique ports in:
file://[dmgr_profile]/config/cells/Cell02/nodes/CellManager02/serverindex.xml

5. Manually ( or scripted) rename directories on target nodes
You also have to rename any directories referring to node-names and log-directories matching application server names.
Example:
[was_home]/profiles/Dmgr01 -> Dmgr02
[profile_home]/config/cells/Cell02/nodes/NODE055 -> NODE053
[profile_home]/logs/WebSphere_Portal -> WebSphere_Portal2
[profile_home]/installedApps/Cell01 -> Cell02
[profile_home]/config/cells/Cell02/clusters/DEV001 -> PROD004

Manually edit these files if not included in your substitution-script:

file://[portal_home]/jcr/lib/com/ibm/icm/icm.properties:
jcr.textsearch.indexdirectory=\\\\UNC_PATH\\TO\\SEARCHINDEXES # Windows UNC format in cluster
jcr.textsearch.indexdirectory=/PATH/TO/SEARCHINDEXES # Full path to local indexes

file://[portal_home]/config/wpconfig.properties # this value has to be unique for every node
PortalUniqueID=[uniqID]
Default is the stripped MAC-address of the primary network interface.

# Also, don't forget to set all cluster nodes which are not acting as primary nodes to false.

PrimaryNode=false

6. Clean-up
Remove old log-files and transaction-logs and other stuff like remote search collection data
which has to be re-configured after a successful move.

Example:
[profile_home]/tranlog/*
[profile_home]/temp/*
[profile_home]/wstemp/*

7. Backup your "new" initial configuration, so that if the start-up of the deployment manager goes south,
you can fix whatever went bad and retry the startup-operation.

8. Startup deployment manager after checking basic configuration stuff for successful value-substitution.
Example can be setupCmdLine.sh/bat in [dmgr_profile]/bin/, make sure WAS_CELL and WAS_NODE is correct.
Make a quick check in [dmgr_profile]/config/cells/, check directories and xml for cell, cluster and node names.

# cd [dmgr_profile]/bin
# . ./setupCmdLine.sh
# ./startManager -profileName Dmgr02

Check the [dmgr_profile]/logs/dmgr/SystemOut.log for any errors during startup.

9. Run syncNode

Prior to running syncNode, you have to make sure the directory structure under the local profile matches
the new node-names and application server names from the move/restore in step 3.
All local files which differs will be updated, but the directory structure is required.

# cd [profile_home]/bin
# . ./setupCmdLine.sh
# syncNode.sh [dmgr] 8880 -username WebSphereAdmin -password password -profileName wp_profile

Check the [profile_home]/logs/syncNode.log for any errors during sync.

10. Start nodeagents

Check the [profile_home]/logs/nodeagent/SystemOut.log for any errors during startup.

# cd [profile_home]/bin
# . ./setupCmdLine.sh
# ./startNode -profileName wp_profile


11. Fix any application unique URL-paths, host-names or endpoints.

In our case we had some custom, and environment-unique loadbalancer-names in the WPS screens due to our SSO solution,
so I had to modify a few JSP's which are involved in handling of session-timeout and so on.
Steps where just to export, expand, modify and collapse the EAR.

# earExpander.sh -ear wps.ear -operationDir wps -operation expand
ADMA4006I: Expanding wps.ear into wps

# rm *.ear ( save if required )

* Make required changes..

# earExpander.sh -ear wps.ear -operationDir wps -operation collapse
ADMA4007I: Collapsing wps into wps.ear

12. Validate connectivity and properties prior to an attempt to start the Portal Server.

See PortalUniqueID in step 5.

# WPSconfig.sh -DparentProperties="d:\IBM\WebSphere\PortalServer\config\wpconfig_dbtype.properties" -DSaveParentProperties=true
# WPSconfig.sh validate-database-driver
# WPSconfig.sh validate-ldap
# WPSconfig.sh validate-database-connection-wps
# WPSconfig.sh validate-database-connection-jcr
# WPSconfig.sh validate-database-connection-wmm
# WPSconfig.sh validate-database-connection-feedback
# WPSconfig.sh validate-database-connection-likeminds

13. Start the Portal Server

# cd [profile_home]/bin
# . ./setupCmdLine.sh
# ./startServer WebSphere_Portal

Same procedure as previous start-up steps, monitor SystemOut log carefully and fix any problem that may occur.
Once successful, you probably have a scenario which is re-producible.

=============== substitution-script cut here ===============

#!/usr/bin/sh

USAGE="echo `basename ${0}` [path] to search for xml, bat, sh and properties"

if [ -z "${1}" ];then echo $USAGE && exit 1;fi
INPUT_PATH="${1}"
export INPUT_PATH

perl_subst()
{
perl -i -pe 's/Cell01/Cell02/g' "${1}"
perl -i -pe 's/CellManager01/CellManager02/g' "${1}"
perl -i -pe 's/dmgr01/dmgr02/g' "${1}"
perl -i -pe 's/Dmgr01/Dmgr02/g' "${1}"
perl -i -pe 's/NODE055/NODE053/g' "${1}"
perl -i -pe 's/NODE056/NODE054/g' "${1}"
perl -i -pe 's/NODE059/NODE063/g' "${1}"
perl -i -pe 's/NODE050/NODE058/g' "${1}"
perl -i -pe 's/node055/node053/g' "${1}"
perl -i -pe 's/node056/node054/g' "${1}"
perl -i -pe 's/node064/node063/g' "${1}"
perl -i -pe 's/node050/node058/g' "${1}"
perl -i -pe 's/devWebserver1/prodWebserver1/g' "${1}"
perl -i -pe 's/NODE055Cell/NODE053Cell/g' "${1}"
perl -i -pe 's/NODE056Cell/NODE054Cell/g' "${1}"
perl -i -pe 's/NODE059Cell/NODE563Cell/g' "${1}"
perl -i -pe 's/DEV001/PROD004/g' "${1}"
perl -i -pe 's/DMgrSoapPort=8879/DMgrSoapPort=8880/g' "${1}"
perl -i -pe 's/com.ibm.ws.scripting.port=8879/com.ibm.ws.scripting.port=8880/g' "${1}"
perl -i -pe 's/jcr.textsearch.indexdirectory=\/IBM\/WebSphere\/devSearchcollections/jcr.textsearch.indexdirectory=\/IBM\/WebSphere\/prodSearchcollections/g' "${1}"
# port-definitions in cell ( only required if multiple dmgr-profiles exist on the same node )
perl -i -pe 's/port="8879"/port="8880"/g' "${1}"
perl -i -pe 's/port="7277"/port="7377"/g' "${1}"
perl -i -pe 's/port="9809"/port="9810"/g' "${1}"
perl -i -pe 's/port="9100"/port="9101"/g' "${1}"
perl -i -pe 's/port="9401"/port="9404"/g' "${1}"
perl -i -pe 's/port="9402"port="9405"/g' "${1}"
perl -i -pe 's/port="9403"/port="9406"/g' "${1}"
perl -i -pe 's/port="9060"/port="9061"/g' "${1}"
perl -i -pe 's/port="9352"/port="9353"/g' "${1}"
perl -i -pe 's/port="9043"/port="9044"/g' "${1}"
}

find ${INPUT_PATH}/ -type f -name '*.xml' -o -name '*.properties' \
-o -name '*.props' -o -name '*.bat' -o -name '*.sh' | while read line;do
FILENAME=`basename "${line}"`
echo "performing perl_subst() function on: ${FILENAME}"
perl_subst "${line}"
if [ -f "${line}.bak" ];then rm -f "${line}.bak";fi
done
=============== substitution-script cut here ===============

(provide 'moveDmgr)