“This is the 9th day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021.”

Latest solution: Oracle Integration Base installation package + patch package

A, takeaway

This paper mainly introduces some pits of oracle 12201 RAC installation in Redhat7.6 system and ways to avoid pits. Absolute dry goods full, value for money, welcome to supplement and correction.

2. Environment introduction

OS: redhat 7.6 64-bit (3.10.0-957.el7.x86_64)

The ORACLE: 12 cr2 RAC

3. Guide to avoiding pits

Notes: The pits are performed by cVU check and root.sh in the Grid software installation procedure.

Here are the main points and main ideas of this article, simple and direct:

Pit 1: ASM Device Sharedness Check

Shared Storage Accessibility:/dev/asm_ocr … FAILED (PRVG-11506)

Refer to MOS documentation:

12.2: PRVG-0802: Storage type for path “/dev/mapper/asm011P1” could not be determined (Doc ID 2251322.1)

If CVU storage check fails for ASMLib paths, Apply patch 25784424.

 

Pit 2: CLSRSC-400: A system reboot is required to continue installing.

When root.sh was running at 14/19, it suddenly stopped and an error was reported **** clsrSC-400. I asked me to restart the host system. After the restart, the installation had stopped.

Refer to MOS documentation:

ALERT: Root. sh Fails With “clSRSC-400” While Installing GI 12.2.0.1 on RHEL or OL With RedHat Compatible Kernel (RHCK) 7.3 (Doc) Ids 2284463.1)

Not to mention solutions, MOS suggests patching with applyOneOffs: Interim Patch 25078431 is required before Installing 12.2 GI on Linux 7.3 (RedHat and OL7 with RHCK).

 

3: the pit kgfnGetConnDetails requires four parameters at/u01 / app / 12.2.0 / grid/lib/asmcmdbase PM line 5704.

When the root. Sh execution reaches 19/19, the root. Sh execution fails.

Refer to MOS documentation:

install.sh Hung And root.sh Is Failing At asmcmd lsdg –suppressheader While Installing A Zone With Clusterware (Doc ID 2414241.1)

ASMCMD Failing With “KGFNGETCONNDETAILS Requires 4 Parameters at <GI_HOME>/lib/ asmcmdBase.pm (Doc ID 2748316.1)

Solution:

/ usr/bin/make -f/u01 / app / 12.2.0 / grid/RDBMS/lib/ins_rdbms. Mk client_sharedlib libasmclntsh12. Ohso libasmperl12. Ohso ORACLE_HOME = / u01 / app / 12.2.0 / gridCopy the code

1. If the error occurs, run root.sh again after the preceding command. If you have seen this document in advance, perform method 2.

2.__When the root.sh message is displayed, run the preceding command on both nodes as root. Set the GRID_HOME path based on actual conditions

After all nodes are executed, run root.sh.

 

Finally, the solutions of pit 1 and 2 are summarized:

Refer to MOS documentation:

How to Apply a Grid Infrastructure Patch Before Grid Infrastructure Configuration (before root.sh or rootupgrade.sh or gridsetup.bat) is Executed (Doc ID 1410202.1)

Install the latest patch to Grid using _applyPSU__, which is supported by 12C, and fix the pit 1,2 bug: _

$su - grid -c "unzip /soft/ p6880880_122010_linux-x86-64. zip -d /u01/app/12.2.0/grid/" Notes: Select A 2. Check the OPatch patches version $su - the grid - c "/ u01 / app / 12.2.0 / grid/OPatch/OPatch version" OPatch version: 12.2.0.1.24 3. $su - grid -c "unzip /soft/ p32226491_122010_linux-x86-64. zip -d /soft" 4 Preparing the Home to Patch; Preparing the home to patch; Preparing the home to patch; Preparing the home to patch. Applying the patch /soft/32226491/... Successfully applied the patch. The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2021-04-01_04-18-54PM/installerPatchActions_2021-04-01_04-18-54PM.log Launching Oracle Grid Infrastructure Setup Wizard... $opatch lSPatches 26839277 $OPatch LSPatches 26839277 DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277) 32231681; ACFS JAN 2021 RELEASE UPDATE 12.2.0.1.210119 (32231681) 32253903; TOMCAT UPDATE 12.2.0.1.0(ID:RELEASE) (32253903) 31802727; OCT 2020 RELEASE UPDATE 12.2.0.1.201020 (31802727) 32228578; Database Jan 2021 Release Update: 12.2.0.1.210119 (32228578) OPatch succeeded.Copy the code

Please refer to the following section for details about environment configuration.

Four, the environment configuration points

1.Reverse Path Filtering (RP_filter)

For details on how to set rP_filter, please refer to the MOS documentation and official documentation:

RAC and Oracle Clusterware Best Practices and Starter Kit (Linux) (Doc ID 811306.1)

Docs.oracle.com/database/12…

For Linux Kernels 2.6.31 (including, for example, Oracle Linux and RedHat) and later versions, an error has been fixed in reverse path filtering. Because of this bug fix, interconnection packets may be blocked/dropped on multiple interconnection systems. To avoid this, Oracle RAC systems that use multiple nics for dedicated interconnection now require a specific setting of the RP_filter parameter. For example, if eth1 and eth2 are dedicated interconnect nics and eth0 is a public network NIC, set the parameters in /etc/sysctl.conf to set rP_filter of the private address to 2 (loose filtering) and rp_filter of the public address to 1 (strict filtering) :  net.ipv4.conf.eth2.rp_filter = 2 net.ipv4.conf.eth1.rp_filter = 2 net.ipv4.conf.eth0.rp_filter = 1Copy the code

2.Memlock&&HugePages

For information on how to set up standard HugePages memory, see the MOS documentation and official documentation:

What is Memlock and How to Calculate the Values for Memlock? (Doc ID 2511230.1)

Docs.oracle.com/database/12…

During the installation and deployment, the HugePages memory is usually disabled because it cannot be estimated that the application needs standard large pages. When the HugePages memory is disabled, the maximum locked memory limit should be set to at least 3145728 KB (3 GB).

cat <<EOF >>/etc/security/limits.conf
oracle   soft   memlock    3145728
oracle   hard   memlock    3145728
grid   soft   memlock    3145728
grid   hard   memlock    3145728
EOF
Copy the code

Some limitations on enabling large page memory:

A. Automatic Memory Management (AMM) is incompatible with HugePages. With AMM, the entire SGA memory is allocated by creating files under /dev/shm. HugePages are not reserved when Oracle database allocates SGA through AMM. To use HugePages on Oracle Database 12C, AMM must be disabled.

B. You must unset the MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters. For example, to unset the parameters of a database instance, use the command ALTER SYSTEM RESET.

C. Make sure HugePages are configured correctly, because if the application does not use too many HugePages, the system may run out of memory.

D. If the instance starts with insufficient HugePages and the initialization parameter use_LARge_pages is set to only, the database cannot start and the alert log message provides the necessary information about HugePages.

Given the risks and limitations, if you want to enable HugePages, follow these steps to see how to set up HugePages, assuming we now have 256GB of memory:

1. Run the $grep Huge /proc/meminfo 2 command to check whether the kernel supports HugePages. Using the root user to edit the/etc/security/limits memlock Settings in the file. Memory lock is specified in KB. When the HugePages memory is enabled, the maximum locked memory limit should be set to at least 90% of the current RAM; when the HugePages memory is disabled, the maximum locked memory limit should be set to at least 3145728 KB (3 GB). For example, if we are 256 GB OF RAM, add the following entries to increase the maximum memory-locked address space: (256 gb * 0.9 * 1024 * 1024 = 241591910.4 KB) cat < < EOF > > / etc/security/limits the conf oracle soft memlock 241591910 oracle hard memlock 241591910 grid soft memlock 241591910 grid hard memlock 241591910 EOF 3. After the memory lock is set, log in as oracle user and run the ulimit -l command to verify the new memory lock setting: $ulimit -l 241591910 4 Run the following command to display the value of the Hugepagesize variable: $grep Hugepagesize /proc/meminfo 5. A script provided with the official Oracle documentation calculates the recommended value for a large page configuration of the current shared memory segment: a. Create the shell script hugePages_settings. sh b. Add the following script content: #! /bin/bash # # hugepages_settings.sh # # Linux bash script to compute values for the # recommended HugePages/HugeTLB configuration # # Note: This script does calculation for all shared memory # segments available when the script is run, no matter it # is an Oracle RDBMS shared memory segment or not. # Check for the kernel version KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'` # Find out the HugePage size HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}` # Start from 1 pages to be on the safe side and guarantee 1 free HugePage NUM_PG=1 # Cumulative number of pages required to handle the running shared memory segments for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"` do MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q` if [ $MIN_PG -gt 0 ]; Then NUM_PG = ` echo "$$MIN_PG NUM_PG + + 1" | BC - q # ` fi done Finish with results case $KERN in '2.4') HUGETLB_POOL = ` echo "$NUM_PG*$HPG_SZ/1024" | bc -q`; echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;; '2.6' | '3.8') echo "it setting: the vm. Nr_hugepages = $NUM_PG";; *) echo "Unrecognized kernel version $KERN. Exiting." ;; Esac # End c. Grant shell script execution permission $chmod +x hugePages_settings. sh d Run the hugePages_settings. sh script to calculate the value of the hugePages configuration: $./ hugePages_settings. sh Notes: Before running this script, be sure to run all applications that use large pages. 6. Set the following kernel parameters, where value is the HugePages value that you determined in Step 5: # sysctl -w vm. nr_hugePages =value 7. To ensure that HugePages are assigned after the system restarts, add the following parameters to the /etc/sysctl.conf file, where value is the HugePages value you determined in Step 5: Vm. nr_hugePages =value Notes: If you cannot set HugePages allocation using nr_hugePages, the available memory may be fragmented. Restart the server for Hugepages assignments to take effect. 8. Run the following command to check the available large pages: $grep Huge /proc/meminfo 9 Restart the instance.Copy the code

3.Transparent HugePages&&NUMA

Transparent HugePages and NUMA together? Very simple, because they are configured in the same way, please refer to the MOS documentation and official documentation:

**ALERT: Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7, UEK2 and above (Doc ID 1557478.1) **

Docs.oracle.com/database/12…

Oracle’s advice on transparent large pages: If you are running RedHat/OEL 6/7, SLES 11/12, or UEK2 kernels, be sure to disable “transparent HugePages” to prevent performance issues and node/instance expulsion.

There are two ways to close transparent large pages:

A. Add the following line to /etc/rc.local and restart the server (although rc.local is not recommended, you can still do this on Redhat 7). This method is common to Linux6 and 7:

Under the root user: cat < < EOF > > / etc/rc. Local if the test - f/sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; Then the echo never > / sys/kernel/mm/transparent_hugepage defrag fi EOF Notes: redhat7 need to rc. Local execute permissions:  chmod +x /etc/rc.localCopy the code

B. Add the following to the kernel boot line of /etc/default/grub (symbolic link to /boot/grub/grub.conf) and then reboot the server (this is the preferred method), but only above linux7 is supported:

The same is true for disabling NUMA:

1. Add the shutdown command to the /etc/grub.conf file after quiet: sed -i 's/quiet/quiet transparent_hugepage=never numa=off/' /etc/default/grub 2. $grub2-mkconfig -o /etc/grub2.cfg b. CFG table $grub2-mkconfig -o /etc/grub2-efi. CFG Notes: Pay attention to the boot mode of the primary partition of the system. You can run the fdisk -l command to view the Disk label type of the primary partition. Restart the server. 4. Check whether $cat /proc/cmdline takes effectCopy the code

4.Disk I/O Scheduler

To set up the Disk I/O Scheduler, refer to the official documentation:

Docs.oracle.com/en/database…

To get the best performance out of Oracle ASM, Oracle recommends that you use the Deadline I/O Scheduler.

On each cluster node, type the following command to verify that the Deadline Disk I/O scheduler is configured for use:

# cat /sys/block/${ASM_DISK}/queue/scheduler
noop [deadline] cfq
Copy the code

In this example, the default disk I/O scheduler is Deadline, and ASM_DISK is the Oracle Automatic Storage Management (Oracle ASM) disk device.

If the output is null:

_Notes: _ In some virtual environments (VMS) and special devices (such as fast storage devices), the output of the above command may be empty. The operating system or VM bypasses kernel I/O scheduling and submits all I/O requests directly to the device. In this environment, do not change the I/O Scheduler Settings.

Next, let’s set up the Disk I/O Scheduler:

1. Use a text editor to create a UDEV rule file for Oracle ASM devices, add the following lines to the rule file and save:  # cat<<EOF >/etc/udev/rules.d/60-oracle-schedulers.rules ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline" EOF 2. If you are in an RAC cluster, you need to copy the rule files to all other nodes on the cluster. For example, $SCP 60-oracle-schedulers.rules root@node2: /etc/udev/rules.d/3. # udevadm control --reload-rules 4 Verify that the disk I/O scheduler is set to Deadline # cat /sys/block/${ASM_DISK}/queue/schedulerCopy the code

5.ORACLE_HOSTNAME Environment variables

Please refer to the official document:

Docs.oracle.com/en/database…

You must set the ORACLE_HOSTNAME environment variable to install the Oracle database on a multi-alias computer. A multi-alias computer is a computer resolved to multiple aliases.

A computer with multiple aliases is a computer registered with the naming service under a single IP address, but it resolves multiple aliases to that address. The naming service resolves any of these aliases to the same computer. Before installing the Oracle database on such a computer, set the Oracle installation owner environment variable ORACLE_HOSTNAME to the computer whose host name you want to use.

cat <<EOF >> /home/oracle/.bash_profile
$ ORACLE_HOSTNAME=somehost.example.com
$ export ORACLE_HOSTNAME
EOF
Copy the code

6.AVAHI daemon&&NOZEROCONF

For configuring AVAHI Daemon &&NOZEROCONF, refer to the MOS documentation and official documentation:

**RAC and Oracle Clusterware Best Practices and Starter Kit (Linux) (Doc ID 811306.1)    **

**CSSD Fails to Join the Cluster After Private Network Recovered if avahi Daemon is up and Running (Doc ID 1501093.1)    **

En.wikipedia.org/wiki/Zero-c…

1. Disable avahi-daemon service systemctl stop avahi-daemon.socket systemctl stop avahi-daemon.service systemctl disable avahi-daemon.service ps -ef|grep avahi-daemon 2.On Oracle Linux/Redhat Linux, "NOZEROCONF=yes" must be included in /etc/sysconfig/network echo "NOZEROCONF=yes" >> /etc/sysconfig/network 3.Once avahi  is disabled, restart the stack or reboot the node.Copy the code

7./dev/shm

For /dev/shm, please refer to the MOS documentation:

**Bug 25907259 – Linux:RHEL7: cvu reports /dev/shm not mounted when it is mounted (Doc ID 25907259.8)    **

**Bug 21441387 – CVU reports /dev/shm as NOT mounted when it is mounted (PRVE-0421) (Doc ID 21441387.8)    **

Rediscovery Notes
If /dev/shm mount check is performed on RHEL7
 
Workaround
ignore error messages
Copy the code

cat <<EOF >> /etc/fstab
tmpfs /dev/shm tmpfs size=4G 0 0
EOF
mount -o remount /dev/shm
Copy the code

_Notes: _Mos suggests that we ignore the prompt and add it to the fstab file manually.