SEARCH DOCS
info central: your site for Collage technical info
  CASSATT.COM   INFO CENTRAL
ACTIVE RESPONSE 5.1 TOPICS BLUEPRINTS TROUBLESHOOTING DOC INDEX


 

TOC
Prerequisites
The blueprint steps
Step 1: Reserve a node from the free pool to use as an image host
Step 2: Install and configure the operating system

Step 3: Perform Oracle 10g RAC pre-installation procedures

Step 4: Capture the image
Step 5: Create the tier
Step 6: Personalize the image instances
Step 7: Allocate nodes and activate the tier
   
know how:

Oracle 10g RAC: Capturing an Image and Creating a Tier

Intended for use with Cassatt Active Response Data Center Edition V5.1.

This blueprint walks through a method for preparing the Oracle 10g Real Application Clusters (Oracle 10g RAC) release 2 (10.2) software to run in the Cassatt Active Response environment using NAS for shared storage. This blueprint creates a generic image that you can use to deploy multiple Oracle RAC environments without reinstalling the software.

The personalization section (step 6) rebuilds the Oracle Cluster Ready Services (CRS) and other components required by Oracle. It also personalizes the image instances for the tier application nodes and sets up shared storage and networking. Once Oracle is running in your Cassatt Active Response environment, you can create a new database or use an existing one.

Oracle 10g RAC is a complicated application. You should have a good understanding of the Oracle 10g RAC application and you should carefully review the Oracle installation documentation before following the procedures in this blueprint. This blueprint applies only to installations that use NAS for shared storage and requires a shared ORACLE_HOME.

If you haven't read Understanding Blueprints for Deploying Applications, read that first for background and context.

Prerequisites

checkmark

SLA: Determine the SLA type and settings for the tier. To understand SLAs, see Understanding Tier Configuration and Personalization.
checkmark

Application node hardware requirements:

  • Application nodes with at least 36-GB local disk, 2 GB of RAM, remote power control, and two NICs to satisfy max nodes for the tier. Consider any other tiers in your domain that also require these attributes.
  • If you are installing a 64-bit version of RedHat ELAS, and you have not already done so, follow these steps to set a custom attribute to identify x86_64 servers and ensure they are the only nodes allocated to run the image:
    1. Identify all of the x86_64 nodes.
    2. Add a custom attribute to each of those nodes that indicates the nodes can be allocated to run images that require a 64-bit-capable architecture. Refer to the online help topic entitled "Using custom hardware attributes to specify hardware for your software" for information.
    3. When you capture the Oracle image, assign the same custom attribute to the image as you assigned to the nodes.
    4. When you create the Oracle tier, the custom attribute is assigned by default. Do not remove the custom attribute.
checkmark Storage requirements: 10 GB on an Oracle-compatible NAS
checkmark Image host requirements: If you are installing a 64-bit version of of RedHat ELAS, select a node with the x86_64 custom attribute.

Software requirements:

  • Oracle 10g Real Application Clusters (Oracle 10g RAC) release 2 (10.2)
  • Red Hat ELAS 3 or 4
    or
    Solaris 10 (x86)

Networking requirements:

  • IP addresses for 2 times max nodes available in the primary network you will use for the Oracle tier; if using Solaris x86 with IP multipathing, allocate 3 times max nodes

    If necessary, add a new network to Cassatt Active Response using the Controller.

    In the Controller, reserve 1 times max nodes IP addresses from the primary network for use as virtual IP addresses. Oracle 10g RAC uses virtual IP addresses to accommodate failover within the Oracle tier. You will assign these addresses during personalization.

    See also, Network Addresses: Calculating Requirements.

  • A network IP address space for the private interconnect Oracle 10g RAC requires

    Add the network to Cassatt Active Response using the Controller. You will assign the address space as a dedicated network during tier creation.

    Do not assign this network to any other tier in Cassatt Active Response. For more information, read Understanding Cassatt Active Response Network Manager.

  • If you will be adding networks and have not yet added switches or VLANs, you must do so before adding tiers. For more information, see Understanding Cassatt Active Response Network Manager.
checkmark Installation document for your operating system:

top

The blueprint steps

This blueprint follows the standard high-level blueprint steps.

Step 1: Reserve two nodes from the free pool to use as image hosts

This blueprint uses two image hosts so you can set up Oracle RAC prior to capture.

  1. From the Controller, click the Free Pool.
  2. Select a node to reserve.
  3. Select Node Actions > Reserve for Image Capture.
  4. Repeats steps 1–3.
  5. Note the nodes' IP addresses, which you will use when you install the OS and capture the image.

Step 2: Install and configure the operating system on both image hosts

Follow the instructions to install and configure:

Solaris x86
Or
ELAS 3 (32-bit only) or 4 (32-bit or 64-bit): Red Hat ELAS.

Red Hat Enterprise Linux 5 is not supported for this blueprint.

If you are using Red Hat ELAS 4, install the following RPM:

libaio-0.3.103-3.i386.rpm

Step 3: Install Oracle 10g RAC

This document uses the same values for environment variables as are used in the Oracle documentation. If you use your own values, substitute those for the standards used here.

  1. Install Oracle as directed in the Oracle installation documentation using NAS as the shared storage device:

    Oracle Clusterware and Oracle Real Application Clusters Installation Guide 10g Release 2 (10.2) for Linux

    Oracle Clusterware and Oracle Real Application Clusters Installation Guide 10g Release 2 (10.2) for Solaris Operating System
  2. Make a note of the mount point for the Oracle application files.
  3. Log in as root on either image host and load the Oracle environment:

    . ~oracle/.bashrc (Linux)
    OR
    . /home/oracle/.profile (Solaris x86)
  4. As the oracle user, verify that the database is running:

    srvctl status database -d database instance

    Where database instance is your database instance.
  5. Verify SQLNET connectivity:

    sqlplus system/password@database instance
    select sysdate from dual;
  6. Shut down Oracle as follows:

    $CRS_HOME/bin/crsctl stop crs

top

Step 4: Capture the image

In this step, you will capture the base image from the image host using the cccapture command—an interactive script. Although you can change the values later, it is most efficient to supply valid values now.

Follow these steps:

  1. Log into the active control node as root.
  2. Enter the following command to start the capture of the base image from the either image host:

    /opt/cassatt/bin/cccapture
  3. Answer each prompt.

    The next table lists the suggested values for the Oracle 10g RAC blueprint; unless you have site-specific reasons, accept the default values for other cccapture prompts.

    For names, descriptions, and other discretionary fields, use values that are meaningful to you. In the image name, do not use spaces or characters (for example, /) special to Unix-based file systems such as Linux.

    Specifying the applications and versions contained in this image is not required for this blueprint, but you may wish to do so for tracking purposes.

    For file system selection: selecting unneeded file systems wastes space but is otherwise harmless.

    4. At this prompt...

    Enter...

    Notes
    Enter the image type (linux,vmhost,windows,solaris):

    linux
    or
    solaris

    		  

    Do any of the applications in the image require personalization (additional configuration once the image is booted)? [n]

    y

    		  

    Do your applications need write access to any directories in the image other than /etc, /dev, / var, /tmp, and /root? [n]

    y

    		 Linux only

     

     

    Enter the absolute directory pathname:

    /home/oracle

    Are there any additional directories to which applications
    included in this image will need to write? [n]

    y

    Enter the absolute directory pathname:

    /opt

    If a tier using this image must always run on a specific set of nodes,
    custom attributes can be specified for the tier. These same attributes
    must then be specified on each node that will be reserved for the tier.
    The Cassatt system will then match on these attributes to ensure that
    those nodes are always allocated for a tier with this image.

    Do you want to specify any custom attributes for tiers using this image? [n]

    		 y or n If you are installed a 64-bit version of the OS, specify the x86_64 server architecture custom attribute
    Enter an attribute value for nodes to use: x86_64attribute  

    Enter the hardware architecture for this image

    (ia32,sparc): [ia32]

    		 ia32 Select "ia32" even when your image requires a 64-bit-capable server. A custom attribute is required to ensure allocation of 64-bit-capable servers.

    Do you want to specify the hardware requirements for a tier using this image? [n]

    y

    		  

    Enter minimum amount of RAM (MBytes) per node for this image (128, 256, 512, 1024, 2048, 4096, 8192): [no minimum]

    2048

    		  

    Enter minimum number of network interface cards (NICs) for this image: [0]

    2

    		  

    Enter maximum number of network interfaces for this image (>= 2): [2]

    4

     

    Does this image require a local disk? [n]

    y

     

    Enter the minimum size in Gbytes for the required local disk: [0]

    disk size

    Cassatt Active Response uses this value in determining which nodes to allocate to tiers running this image. Use the smallest size you believe will provide adequate space for swap and temp (no smaller than 6 GB). Nodes with smaller disks will not be allocated to run this image.

    We recommend at least 36GB for this blueprint.

    Does this image require local swap space? [n]:

    y

     

    Enter the required amount of swap space (in Gbytes) [8]:

    8

    The space remaining on the local disk after swap partitions are created is assigned to local temp space.

    Does this image require local temp space? [n]:

    y

    		  

    Should this image be installed on the local disk (allowing the node to boot locally)? [n] y

    		 y or n

    See Alternatives to NFS: Image on Local Disk for details.

    Monitor via ping?

    y

     
    File System Selection   Be sure to capture the filesystem that contains the Oracle binaries.

  4. In the Controller, release the image hosts back to the free pool by selecting the nodes using the checkboxes, then select Node Actions > Release from Image Capture.

top

Step 5: Create the tier

Follow these steps to create and configure the tier. Cassatt Active Response prepopulates tier values with the parameters you set during the image capture. If necessary, you can change them during tier creation.

  1. On the left navigation pane, click Domain or Tiers to display the Tier List.
  2. On the Tier List page, click the New Tier button (on the right).
  3. On the Properties page, set the properties for the Oracle 10g RAC tier.
  4. On the SLA page, select the SLA type and set as needed.
  5. On the Requirements page, set the hardware requirements for the tier.
  6. On the networks page, select the network you'll be using as the primary network and choose the Primary radio button.
  7. Select the network you added to Cassatt Active Response for use as the Oracle private interconnect and choose the Dedicated radio button.
  8. On the IPs and Hostnames page, assign IP addresses or host names, or accept the default settings. You will use these during personalization.
  9. You can close the Creating Tier Progress page and continue with the next section.

top

Step 6: Personalize the image instances

On the tier page, select the Oracle 10g tier using the checkbox, then select Personalize from the actions dropdown menu.

Cassatt Active Response boots the maximum nodes for the tier.

Most items in this section are performed on each application node as root; a few are performed on each application node as the Oracle user, a few are performed on only one application node, and a few are performed on the control node. Pay close attention to be sure you perform each step on the correct node or nodes.

Prerequisites

  • The following OracleMetaLink problem report (available to registered MetaLink users):

    Doc ID: Note:316583.1
    Subject: vipca Fails Complaining that Interface is not Public
  • The following OracleMetaLink problem report (available to registered MetaLink users) can be helpful for troubleshooting Oracle cluster problems:

    Doc ID: Note:259301.1
    Subject: CRS and 10g Real Application Clusters
  • Solaris x86 with IPMP: The following OracleMetaLink problem report (available to registered MetaLink users) can be helpful to run VIPCA when using IP multipathing in Solaris x86:

    Doc ID: Note:283107.1
    Subject: Configuring Solaris IP Multipathing (IPMP) for the Oracle 10g VIP

Add VIP addresses

  1. Log into the control node as root.
  2. Make a backup copy of /cassatt/etc/hosts:

    cp -p /cassatt/etc/hosts /root/hosts.backup
  3. Add a line for each VIP address/hostname to the /cassatt/etc/hosts file on the control node. Use the addresses you reserved in the prerequisites.

    For the hostnames, append the suffix -vip to each node name, for example, nodename1-vip.

    In the file, add the VIP addresses/hostnames at the bottom of the first section, as shown in the example that follows. Do not put any comments after the hostname, because Cassatt controls any address with an appropriate comment following the node information. In the example, the additions are in bold; the generated lines represent the addresses/hostnames in the Cassatt Active Response domain, including those for the primary and private networks for the Oracle tier, which are shown in italics.

        4. Example /cassatt/etc/hosts file

        # Do not remove the following line, or various programs
        # that require network functionality will fail.
        127.0.0.1 localhost.localdomain localhost
        10.0.140.3 pea1.pea1.lab.msp.cassatt.com pea1
        # Following lines added manually for VIP connection in oracle
        10.0.140.12 nodename1-vip
        10.0.140.14 nodename2-vip
        # # cassatt_collage
        # Following lines generated by Cassatt Collage # cassatt_collage
        # Do not modify. # cassatt_collage
        # # cassatt_collage
        ...
        10.0.140.110 Pool0011 # cassatt_collage
        10.0.140.113 collage0014 # cassatt_collage
        10.0.140.107 collage0015 # cassatt_collage
        10.0.140.111 collage0016 # cassatt_collage
        10.0.140.112 nodename1 # cassatt_collage
        10.1.141.2 private1 # cassatt_collage
        10.0.140.114 nodename2 # cassatt_collage
        10.1.141.3 private2 # cassatt_collage
        10.0.143.254 PowerDevice1 # cassatt_collage
        ...
  4. Copy the /cassatt/etc/hosts file to /etc/hosts:

    cp -p /cassatt/etc/hosts /etc
  5. Update the dns server:

    /opt/cassatt/sudo/bm_named hosts

If you delete this tier, you must delete the VIP addresses manually from the control node:

/opt/cassatt/sudo/bm_named hostdel VIPaddress nodename1-vip.domain

Where VIPaddress is the VIP for the node, nodename1-vip is the VIP hostname of the node, and domain is the fully qualified domain name.

Check the network connectivity

Follow the instructions in Pre-Installation Tasks, Checking the Network Setup with CVU, in the Oracle installation document for your operating system.

If you are using a non-routable address, such as 10.x or 192.x, the vipca portion of this test will fail, as will the test as a whole. The test nonetheless provides valuable information. See the OracleMetaLink problem report listed in the prerequisites.

Set up the shared storage

Follow the instructions in Configuring Oracle Clusterware and Oracle Database Storage in the Oracle installation document for your operating system. Create mount points on your NAS for /quorum and /ora_data. Be sure to check your setup using CVU.

Prepare application nodes

On each application node:

  1. Create the /etc/fstab (Linux) or /etc/vfstab(Solaris x86) entries for /quorum and /ora_data.

    Refer to Checking the Network Requirements in Section 2, Pre-Installation Tasks, of the Oracle installation document for assistance with the network buffer size parameters rsize and wsize. Format as in these examples.

    Linux:

    name of the NAS:/mount point/apps/oracle/ora_data /ora_data nfs
    rw,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=312
    name of the NAS:/mount point/apps/oracle/quorum /quorum nfs tcp,noac 0 0

    Solaris x86:

    name of the NAS:/mount point/apps/oracle/ora_data - /ora_data nfs - yes
    rw,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,forcedirectio,vers=3
    name of the NAS:/mount point    /apps/oracle/quorum - /quorum nfs - yes
    rw,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,forcedirectio,vers=3
  2. Solaris x86 only: edit the /etc/vfstab entry for the root partition, as follows:

    control node IP:/image path/image name/image version/instances/node IP address - / nfs - - rw,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3
  3. Solaris x86 only: reboot the node.
  4. Edit /etc/ntp.conf to use the ntp server for your environment.
  5. Change the owner of /ora_data:

    chown oracle:oinstall /ora_data
  6. Mount the required file systems (quorum and ora_data).
  7. Set up the ssh configuration, as described in Configuring SSH on All Cluster Nodes of the Pre-Installation Tasks in the Oracle installation document for your operating system.
  8. Populate the node lists for the Oracle product and the Oracle cluster by running the Oracle installer on each node, as follows:
    1. From an X-Windows application, log into the application node as the oracle user.
    2. Change to the directory in the Oracle Clusterware home directory that contains the installer:

      cd $CRS_HOME/oui/bin
    3. Run the installer command, listing each cluster node:

      ./runInstaller -cfs -updateNodeList ORACLE_HOME=$CRS_HOME "CLUSTER_NODES={nodename1,nodename2,nodename3,nodename4}" "LOCAL_NODE=nodename1"

      Where nodenameN is the hostname of each node in the cluster.
    4. Change to the directory in the Oracle home directory that contains the installer:

      cd $ORACLE_HOME/oui/bin
    5. Run the installer command, listing each cluster node:

      ./runInstaller -cfs -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={nodename1,nodename2,nodename3,nodename4}" "LOCAL_NODE=nodename1"

      Where nodenameN is the hostname of each node in the cluster.

      If the command fails, run it again.
    6. Validate that the commands were entered correctly and that the nodes are correctly reflected in both the CRS_HOME and ORACLE_HOME sections of /var/opt/oracle/oraInventory/ContentsXML/inventory.xml.

Set up the quorum files

This step is run on only one node because the quorum files are on shared storage.

  1. Log in as root to the lowest-numbered application node.
  2. Remove any existing ocr and voting files:

    rm -f /quorum/voting.css; rm -f /quorum/ocr.dbf
  3. Initialize the ocr and voting files:

    dd if=/dev/zero of=/quorum/voting.css bs=8192 count=2560
    dd if=/dev/zero of=/quorum/ocr.dbf bs=8192 count=12800
    chmod 666 /quorum/*

Prepare the installation files

  1. As root on the lowest-numbered application node, bring in the Oracle environment:
    . ~oracle/.bashrc (Linux)
    OR
    . /home/oracle/.profile (Solaris x86)
  2. Edit $CRS_HOME/install/rootconfig and $CRS_HOME/install/paramfile.crs to have the correct node lists, for example:

    CRS_HOST_NAME_LIST=nodename1,1,nodename2,2,nodename3,3,
    nodename4,4
    CRS_NODE_NAME_LIST=nodename1,1,nodename2,2,nodename3,3,
    nodename4    ,4
    CRS_PRIVATE_NAME_LIST=private1,1,private2,2,private3,3,
    private4,4
    CRS_NODELIST=nodename1,nodename2,nodename3,nodename4
    CRS_NODEVIPS='nodename1/nodename1-vip/255.255.252.0/bond0,
    nodename2    /nodename2-vip/255.255.252.0/bond0,nodename3/
    nodename3    -vip/255.255.252.0/bond0,nodename4/
    nodename4-vip/255.255.252.0/bond0'

    Where nodenameN is the hostname of each node in the cluster and privateN is the private hostname of each node on the dedicated network.

    For Solaris x86, replace bond0 with the actual public interface name, such as bge0, ce0, or qfe0.
  3. In $CRS_HOME/install/rootconfig and $CRS_HOME/install/paramfile.crs, verify that the locations of the ocr and voting files are correct.
  4. Back up rootconfig to rootconfig.bak:

    cp -p rootconfig rootconfig.bak
  5. Make the rootconfig and paramfile.crs writable:

    chmod 777 rootconfig
    chmod 777 paramfile.crs
  6. Edit rootconfig to stop it from exiting on failure and to force the ocr to be rewritten, for example:

    else
    $ECHO "Failed to upgrade Oracle Cluster Registry configuration"
    # exit 1
    fi

    $ORACLE_HOME/bin/clscfg -install -force -nn $CRS_NODE_NAME_LIST -pn $CRS_PRIVATE_NAME_LIST
  7. Start Oracle CRS:

    cd $CRS_HOME
    ./root.sh

    You can ignore errors related to read-only file systems.
  8. Revert back to the original rootconfig, as follows:

    cd $CRS_HOME/install
    mv rootconfig rootconfig.rebuild_ocr
    mv rootconfig.bak rootconfig

Start Oracle on all cluster nodes

  1. As root on each additional node, load the Oracle environment, copy the rootconfig and paramfile.crs from the lowest-numbered node, then start Oracle:

    . ~oracle/.bashrc (Linux)
    OR
    . /home/oracle/.profile (Solaris x86)

    cd $CRS_HOME/install
    scp -p nodename1:$CRS_HOME/install/rootconfig .
    scp -p nodename1:$CRS_HOME/install/paramfile.crs .

    Where nodename1 is the first node in the cluster.

    cd $CRS_HOME;
    ./root.sh

    Make sure to wait for the script to complete on each node, displaying the line, "Local node checking complete" before moving on to the next node.
  2. After you start Oracle on the last node, vipca will run to configure the virtual IP addresses on all nodes. Unfortunately, in Linux, vipca fails on the last node, as described in OracleMetaLink problem report 316583.1. On Linux, you will need to run vipca manually on that node as root from an X-Windows application:

    . ~oracle/.bashrc

    cd $CRS_HOME/bin

    ./vipca

    In the vipca wizard, select the public interface, use a fully qualified domain name for the node VIP hostname, and use the correct netmask for your network.

Set up the listener information

  1. As root on each application node, bring in the Oracle environment:
    . ~oracle/.bashrc (Linux)
    OR
    . /home/oracle/.profile (Solaris x86)
  2. Change the owner of $CRS_HOME/crs; this prevents error CRS-0241, Invalid directory path:

    chown oracle:oinstall $CRS_HOME/crs
  3. From an X-Windows application, log into the lowest-numbered application node as the oracle user.
  4. Change to the directory that contains the tns information:

    cd $TNS_ADMIN
  5. If you are going to use an existing database, make a copy of the tnsnames.ora file:

    cp -p tnsnames.ora tnsnames.ora.bk
  6. Remove the tnsnames.ora file:

    rm -f listener.ora tnsnames.ora
  7. Run netca, taking all defaults.

Verify that all cluster services are running cleanly

  1. As root on each additional node, load the Oracle environment:

    . ~oracle/.bashrc (Linux)
    OR
    . /home/oracle/.profile (Solaris x86)

  2. Check the status of the cluster, for example:

    $CRS_HOME/bin/crs_stat –t

Create the database from an existing database or from scratch

Use dbca to create a new database as described in the Oracle documentation or import an existing database. Be sure to choose a cluster file system and use /ora_data as the common database location.

Verification

  1. As the oracle user on any application node, verify that the database is running:

    srvctl status database -d database instance

    Where database instance is your database instance.
  2. Verify SQLNET connectivity:

    sqlplus system/password@database instance
    select sysdate from dual;
  3. Set the database to start on boot by setting that option to "y" in /etc/oratab.
  4. Restart the node and verify that the database comes up properly.

When you are finished, return to the Controller, select the tier, and select "Personalization Complete" for the tier.

top

Step 7: Allocate nodes and activate the tier

Follow these steps to allocate nodes and activate the tier.

  1. From the Tiers table on the Domain page, select the checkbox for the Oracle tier.
  2. From the Tier Actions dropdown menu, select Allocate Nodes. Click Allocate Nodes on the confirmation page.

    Cassatt Active Response allocates nodes and displays the Tiers table.
  3. If the Oracle 10g RAC tier depends on another tier, you cannot activate the Oracle tier until after you create and activate the tier Oracle depends on.
  4. Again, select the checkbox for the Oracle 10g RAC tier.
  5. From the Tier Actions dropdown menu, select Activate. Click Activate on the confirmation page.

    Cassatt Active Response starts the operating system and the applications on each node in the tier.

This completes the procedure to create and deploy an Oracle 10g RAC image.

 

 
 
© Copyright 2008 Cassatt Corporation : All rights reserved. Privacy | Legal | Feedback