메뉴 건너뛰기

Korea Oracle User Group

Install/Configuration

Oracle RAC RU rolling patching from 19.3 to 19.9.0.0.201020(Patch 31750108)

1. 패치에 포함된 README 탐독

2. 패치전 점검 사항

3. Grid Home 패치 수행

4. Oracle Home 패치 수행

5. Post-Installation

6. 적용된 사항 확인

7. 총평

1. 패치에 포함된 README 탐독

패치를 하기 전 항상 패치에 포함된 Readme 파일을 꼼꼼히 읽어 봐야 한다.

이번 패치의 경우 rolling 방식의 패치가 가능하다고 문서에 기재되어 있다.

 

0002.png

 

해당 패치에 대한 문서를 확인하려면

Doc 2694903.1 Oracle Database 19c RU/RUR Oct 2020 Known Issues 를 참조하시기 바랍니다.

 

패치에 포함된 readme.html 문서는 파일 첨부해 두도록 하겠습니다.

 

2. 패치전 점검 사항

1) opatch 버전 점검

이번 패치의 경우 opatch 버전이 최소 12.2.0.19 이상이여야 한다.

 

1
2
3
4
mprac1.localdomain@oracle:MPRAC1:/home/oracle> opatch version
OPatch Version: 12.2.0.1.17
 
OPatch succeeded.
cs

 

위와 같이 버전이 낮기 때문에 버전을 업그레이드 해야한다. 

아래 링크에서 받아서 설치하면 된다.

https://updates.oracle.com/ARULink/PatchDetails/process_form?patch_num=6880880

 

아래와 같이 Grid Home과 Oracle Home에 파일을 로드하고

Grid owner와 Oracle owner 계정으로 unzip 수행 후 버전을 확인하다.(2번 노드에서도 수행해야함)

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
--Zip file upload
mprac1@root:/root> cp p6880880_200000_Linux-x86-64.zip /u01/app/19.0.0/grid/
mprac1@root:/root> chown grid:oinstall /u01/app/19.0.0/grid/p6880880_200000_Linux-x86-64.zip 
mprac1@root:/root> cp p6880880_200000_Linux-x86-64.zip /u01/app/oracle/product/19.0.0/db_1/
mprac1@root:/root> chown oracle:oinstall /u01/app/oracle/product/19.0.0/db_1/p6880880_200000_Linux-x86-64.zip
 
--Grid Home opatch upgrade
mprac1@root:/root> cd /u01/app/19.0.0/grid
mprac1@root:/u01/app/19.0.0/grid> mv OPatch OPatch_bak
mprac1@root:/u01/app/19.0.0/grid> unzip p6880880_200000_Linux-x86-64.zip
...
...
mprac1@root:/u01/app/19.0.0/grid> chown -R grid:oinstall OPatch
mprac1@root:/u01/app/19.0.0/grid> su - grid
mprac1@grid:+ASM1:/home/grid> opatch version
OPatch Version: 12.2.0.1.23
 
OPatch succeeded.
 
--Oracle Home opatch upgrade
mprac1@root:/root> cd /u01/app/oracle/product/19.0.0/db_1/
mprac1@root:/u01/app/oracle/product/19.0.0/db_1> mv OPatch OPatch_bak
mprac1@root:/u01/app/oracle/product/19.0.0/db_1> unzip p6880880_200000_Linux-x86-64.zip 
...
...
mprac1@root:/u01/app/oracle/product/19.0.0/db_1> chown -R oracle:oinstall OPatch 
mprac1@root:/u01/app/oracle/product/19.0.0/db_1> su - oracle
mprac1@oracle:MPRAC1:/home/oracle> opatch version
OPatch Version: 12.2.0.1.23
 
OPatch succeeded.
cs

2) Oracle Inventory 정보 확인 및 저장

패치 수행 전에 현재 정보를 저장해 두자.

간혹 이슈 발생으로 확인이 필요해지는 경우가 생기기도 한다.(2번 노드에서도 체크해야 함)

 

1
2
3
4
5
6
--grid 계정에서 확인
mprac1@grid:+ASM1:/home/grid> opatch lsinventory -detail -oh $ORACLE_HOME > before_patch_lsinventory.txt
 
--oracle 계정에서 확인
mprac1@oracle:MPRAC1:/home/oracle> opatch lsinventory -detail -oh $ORACLE_HOME > before_patch_lsinventory.txt
 
cs

 

생성된 파일 before_patch_lsinventory.txt 내용을 확인해 둔다.

3) Opatch Conflict Check

패치 적용 전 항상 Conflict 체크를 해야한다.

만약 충돌 패치가 있다면 해결 방안을 만들고 해결하고 수행해야 한다.

 

먼저 패치 zip 파일을 unzip 한다.

 

1
2
3
4
5
6
--patch 파일 unzip
mprac1@root:/root> mv p31750108_190000_Linux-x86-64.zip /tmp/  
mprac1@root:/root> unzip /tmp/p31750108_190000_Linux-x86-64.zip -/tmp
...
...
mprac1@root:/root> chown -R grid:oinstall /tmp/31750108/
cs

 

패치 문서상으로는 아래와 같이 확인하라고 되어 있다.

 

1
2
3
4
5
6
7
8
9
For Grid Infrastructure Home, as home user:
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31771877
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31772784
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31773437
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31780966
For Database home, as home user:
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31771877
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/31750108/31772784
cs

 

그래서 Grid Home owner인 grid 계정에서 위 5개의 디렉토리로 체크를 해 보면 /tmp/31750108 체크에서 아래와 같이 에러가 발생한다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
mprac1@root:/root> su - grid
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31771877
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.23
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-30-31PM_1.log
 
Invoking prereq "checkconflictagainstohwithdetail"
 
Prereq "checkConflictAgainstOHWithDetail" passed.
 
OPatch succeeded.
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31772784
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.23
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-31-16PM_1.log
 
Invoking prereq "checkconflictagainstohwithdetail"
 
Prereq "checkConflictAgainstOHWithDetail" passed.
 
OPatch succeeded.
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31780966
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.23
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-32-27PM_1.log
 
Invoking prereq "checkconflictagainstohwithdetail"
 
Prereq "checkConflictAgainstOHWithDetail" passed.
 
OPatch succeeded.
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.23
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-33-12PM_1.log
 
This command doesn't support System Patch.
 
OPatch failed with error code 21
cs

 

이는 무시해도 되는 에러이다.

Readme 파일에서 체크할 필요가 없는 패치 디렉토리를 체크하라고 잘못 기재한 내용이다.

This command doesn't support System Patch 문구의 경우

오라클 메타링크에서 조회해 보면 잘못된 디렉토리 지정에 대한 에러라는 내용이 많다

 

로그 파일의 마지막 부분 로그를 확인하면 아래와 같으며

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[Dec 82020 1:33:13 PM] [INFO]     PREREQ session
[Dec 82020 1:33:13 PM] [INFO]     CUP_LOG: Trying to load HomeOperations object
[Dec 82020 1:33:13 PM] [INFO]     CUP_LOG: HomeOperations object created. CUP1.0 is enabled
[Dec 82020 1:33:13 PM] [INFO]     OPatch invoked as follows: 'prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/ -invPtrLoc /u01/app/19.0.0/grid/oraInst.loc '
[Dec 82020 1:33:13 PM] [INFO]     Runtime args: [-javaagent:/u01/app/19.0.0/grid/OPatch/jlib/oracle.opatch.classpath.jar, -Xmx3072m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/u01/app/19.0.0/grid/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=-DCommonLog.COMMAND_NAME=prereq, -DOPatch.ORACLE_HOME=/u01/app/19.0.0/grid, -DOPatch.DEBUG=false-DOPatch.MAKE=false-DOPatch.RUNNING_DIR=/u01/app/19.0.0/grid/OPatch, -DOPatch.MW_HOME=-DOPatch.WL_HOME=-DOPatch.COMMON_COMPONENTS_HOME=-DOPatch.FMW_COMPONENT_HOME=-DOPatch.OPATCH_CLASSPATH=-DOPatch.WEBLOGIC_CLASSPATH=-DOPatch.SKIP_OUI_VERSION_CHECK=-DOPatch.NEXTGEN_HOME_CHECK=false-DOPatch.PARALLEL_ON_FMW_OH=]
[Dec 82020 1:33:13 PM] [INFO]     Heap in use : 30 MB
                                    Total memory: 146 MB
                                    Free memory : 115 MB
                                    Max memory  : 2731 MB
[Dec 82020 1:33:13 PM] [INFO]     Oracle Home       : /u01/app/19.0.0/grid
                                    Central Inventory : /u01/app/oraInventory
                                       from           : /u01/app/19.0.0/grid/oraInst.loc
                                    OPatch version    : 12.2.0.1.23
                                    OUI version       : 12.2.0.7.0
                                    OUI location      : /u01/app/19.0.0/grid/oui
                                    Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_13-33-12PM_1.log
[Dec 82020 1:33:13 PM] [INFO]     Patch history file: /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch_history.txt
[Dec 82020 1:33:13 PM] [INFO]     [OPSR-TIME] Loading raw inventory
[Dec 82020 1:33:13 PM] [INFO]     [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 23 (MB)
[Dec 82020 1:33:13 PM] [INFO]     [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 23 (MB)
[Dec 82020 1:33:13 PM] [INFO]     [OPSR-TIME] Raw inventory loaded successfully
[Dec 82020 1:33:19 PM] [INFO]     This command doesn't support System Patch.
[Dec 8, 2020 1:33:19 PM] [INFO]     Finishing PrereqSession at Tue Dec 08 13:33:19 KST 2020
[Dec 8, 2020 1:33:19 PM] [INFO]     Stack Description: java.lang.RuntimeException: This command doesn't support System Patch.
                                        at oracle.opatch.PrereqSession.process(PrereqSession.java:184)
                                        at oracle.opatch.OPatchSession.process(OPatchSession.java:2780)
                                        at oracle.opatch.OPatch.process(OPatch.java:853)
                                        at oracle.opatch.OPatch.main(OPatch.java:910)
cs

 

점검 대상 디렉토리를 확인해 보면 딱히 점검할 파일이나 내용이 존재하지 않는다.

 

1
2
3
4
5
6
7
8
9
10
11
12
mprac1@oracle:MPRAC1:/home/oracle> ls -al /tmp/31750108/
total 136
drwxr-x---   7 grid oinstall    143 Oct 19 21:09 .
drwxrwxrwt. 32 root root       4096 Dec  8 13:49 ..
drwxr-x---   5 grid oinstall     81 Oct 19 21:10 31771877
drwxr-x---   5 grid oinstall     62 Oct 19 21:13 31772784
drwxr-x---   5 grid oinstall     62 Oct 19 21:09 31773437
drwxr-x---   4 grid oinstall     48 Oct 19 21:14 31780966
drwxr-x---   2 grid oinstall   4096 Oct 19 21:10 automation
-rw-rw-r--   1 grid oinstall   5054 Oct 20 06:36 bundle.xml
-rw-r--r--   1 grid oinstall 122266 Oct 20 06:20 README.html
-rw-r--r--   1 grid oinstall      0 Oct 19 21:10 README.txt
cs

 

이번에는 Oracle Home owner인 oracle 계정으로 Oracle Home 대상 패치에 대해서 패치 충돌 점검을 해 본다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
mprac1@grid:+ASM1:/home/grid> exit
logout
mprac1@root:/root> su - oracle
mprac1@oracle:MPRAC1:/home/oracle> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31771877
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/19.0.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.23
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatch/opatch2020-12-08_13-35-24PM_1.log
 
Invoking prereq "checkconflictagainstohwithdetail"
 
Prereq "checkConflictAgainstOHWithDetail" passed.
 
OPatch succeeded.
mprac1@oracle:MPRAC1:/home/oracle> opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /tmp/31750108/31772784
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/19.0.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.23
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatch/opatch2020-12-08_13-35-55PM_1.log
 
Invoking prereq "checkconflictagainstohwithdetail"
 
Prereq "checkConflictAgainstOHWithDetail" passed.
 
OPatch succeeded.
cs

 

위와 같이 충돌 이슈가 존재하지 않는다

이로서 Grid Home 과 Oracle Home 패치에 대한 패치 충돌은 없는 상태로 확인할 수 있다.

4) Opatch SystemSpace Check

패치에 필요한 공간이 적절하게 남아 있는지를 점검해 본다.

먼저 아래와 같은 내용의 patch_list_gihome.txt 파일을 생성한다.

 

1
2
3
4
5
6
mprac1@root:/root> cat /tmp/patch_list_gihome.txt
/tmp/31750108/31771877
/tmp/31750108/31772784
/tmp/31750108/31773437
/tmp/31750108/31780966
mprac1@root:/root> chown grid:oinstall /tmp/patch_list_gihome.txt
cs

 

grid 계정에서 아래와 같이 필요한 공간에 대해 점검해 본다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
mprac1@root:/root> su - grid
mprac1@grid:+ASM1:/home/grid> opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/19.0.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.0.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.23
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.0.0/grid/cfgtoollogs/opatch/opatch2020-12-08_15-32-56PM_1.log
 
Invoking prereq "checksystemspace"
 
Prereq "checkSystemSpace" passed.
 
OPatch succeeded.
cs

 

이슈 사항이 없는 것을 확인할 수 있다.

 

이제 oracle 계정으로 점검해 보자.

마찬가지로 아래 내용을 담은 patch_list_dbhome.txt 파일을 먼저 생성한다.

 

1
2
3
4
mprac1@root:/root> cat /tmp/patch_list_dbhome.txt
/tmp/31750108/31771877
/tmp/31750108/31772784
mprac1@root:/root> chown oracle:oinstall /tmp/patch_list_dbhome.txt 
cs

 

아래와 같이 점검해 보자

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
mprac1@root:/root> su - oracle
mprac1@oracle:MPRAC1:/home/oracle> opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt
Oracle Interim Patch Installer version 12.2.0.1.23
Copyright (c) 2020, Oracle Corporation.  All rights reserved.
 
PREREQ session
 
Oracle Home       : /u01/app/oracle/product/19.0.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.23
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatch/opatch2020-12-08_15-34-57PM_1.log
 
Invoking prereq "checksystemspace"
 
Prereq "checkSystemSpace" passed.
 
OPatch succeeded.
cs

 

위와 같이 이슈가 없다.

3. Grid Home 패치 수행(발생하는 이슈가 포함되어 있으므로 바로 따라하지 말고 내용 충분히 본 후 진행하기 바랍니다)

Readme 문서에 rolling fashion이 가능하다고 하여 opatchauto 를 사용하여

rolling 방식으로 진행했지만 에러가 발생하게 된다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
mprac1@root:/root> su - oracle
mprac1@oracle:MPRAC1:/home/oracle> srvctl stop instance -db MPRAC -instance MPRAC1 -stopoption immediate
mprac1@oracle:MPRAC1:/home/oracle> exit
logout
mprac1@root:/root> export PATH=$PATH:/u01/app/19.0.0/grid/OPatch
mprac1@root:/root> opatchauto apply /tmp/31750108
 
OPatchauto session is initiated at Tue Dec  8 16:13:14 2020
 
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-08_04-13-17PM.log.
 
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_04-14-03PM.log
The id for this session is 51ED
Shared home /u01/app/19.0.0/grid can only be patched in nonrolling mode.
 
OPATCHAUTO-72030: Execution mode invalid.
OPATCHAUTO-72030: Cannot execute in rolling mode, as CRS home is shared.
OPATCHAUTO-72030: Execute in non-rolling mode.
OPatchAuto failed.
 
OPatchauto session completed at Tue Dec  8 16:14:26 2020
Time taken to complete the session 1 minute, 12 seconds
 
 opatchauto failed with error code 42
cs

 

opatchauto의 경우 crs를 기동한 상태에서 진행하는 형태인데 이렇게 진행했을 때 에러가 발생해 

crs를 내려놓고 진행을 시도해 봤다. 하지만 역시나 같은 에러가 발생했다.

 

이는 Grid Home이 shared 상태라서 그렇다는 것인데

1, 2번 노드 모드 로컬 디스크에 Grid Home을 설치했기 때문에 shared 상태는 아니다.

아마도 이부분은 버그 같다는 생각이 든다.

rolling 방식으로 패치를 진행하면 위와 같은 에러가 지속적으로 발생한다

해당 원인에 대해 깊이있게 확인을 해 보지 못하고 nonrolling 방식으로 진행하기로 변경했다.

 

1번 노드의 crs만 내려놓고 nonrolling 모드로 진행한다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
mprac1@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME -nonrolling
 
OPatchauto session is initiated at Tue Dec  8 20:17:43 2020
 
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-08_08-17-48PM.log.
 
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_08-17-53PM.log
The id for this session is ERE4
 
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid
Patch applicability verified successfully on home /u01/app/19.0.0/grid
 
 
Executing patch validation checks on home /u01/app/19.0.0/grid
Patch validation checks successfully completed on home /u01/app/19.0.0/grid
 
 
Bringing down CRS service on home /u01/app/19.0.0/grid
Prepatch operation log file location: /u01/app/grid/crsdata/mprac1/crsconfig/crs_prepatch_apply_inplace_mprac1_2020-12-08_08-18-35PM.log
CRS service brought down successfully on home /u01/app/19.0.0/grid
 
 
Start applying binary patch on home /u01/app/19.0.0/grid
Binary patch applied successfully on home /u01/app/19.0.0/grid
 
 
Starting CRS service on home /u01/app/19.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/mprac1/crsconfig/crs_postpatch_apply_inplace_mprac1_2020-12-08_08-34-11PM.log
Failed to start CRS service on home /u01/app/19.0.0/grid
 
Execution of [GIStartupAction] patch action failed, check log for more details. Failures:
Patch Target : mprac1->/u01/app/19.0.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.0.0/grid, host: mprac1.
Command failed:  /u01/app/19.0.0/grid/perl/bin/perl -I/u01/app/19.0.0/grid/perl/lib -I/u01/app/19.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_mprac1/patchwork/crs/install -I/u01/app/19.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_mprac1/patchwork/xag /u01/app/19.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_mprac1/patchwork/crs/install/rootcrs.pl -postpatch -nonrolling
Command failure output: 
Using configuration parameter file: /u01/app/19.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_mprac1/patchwork/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/mprac1/crsconfig/crs_postpatch_apply_inplace_mprac1_2020-12-08_08-34-11PM.log
2020/12/08 20:34:32 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'mprac1'
CRS-2672: Attempting to start 'ora.evmd' on 'mprac1'
CRS-2676: Start of 'ora.mdnsd' on 'mprac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'mprac1'
CRS-2676: Start of 'ora.gpnpd' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'mprac1'
CRS-2676: Start of 'ora.gipcd' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'mprac1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'mprac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'mprac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'mprac1'
CRS-2676: Start of 'ora.diskmon' on 'mprac1' succeeded
CRS-2676: Start of 'ora.crf' on 'mprac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'mprac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'mprac1'
CRS-2676: Start of 'ora.ctssd' on 'mprac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'mprac1'
CRS-2676: Start of 'ora.asm' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'mprac1'
CRS-2676: Start of 'ora.storage' on 'mprac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'mprac1'
CRS-2676: Start of 'ora.crsd' on 'mprac1' succeeded
Cluster registry server error. Server error stack -
PROC-00053: The configured patch level, release patch level and the patch level on the other active cluster nodes are not all the same.
- end of server error stackCRS-2883: Resource 'ora.crsd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
PRVH-0116 : Path "/u01/app/19.0.0/grid/crs/install/cmdllroot.sh" with permissions "rw-r--r--" does not have execute permissions for the owner, file's group, and others on node "mprac1".
PRVG-2031 : Owner of file "/u01/app/19.0.0/grid/crs/install/cmdllroot.sh" did not match the expected value on node "mprac1". [Expected = "grid(1001)" ; Found = "root(0)"]
PRVG-2032 : Group of file "/u01/app/19.0.0/grid/crs/install/cmdllroot.sh" did not match the expected value on node "mprac1". [Expected = "oinstall(1001)" ; Found = "root(0)"]
CRS-4000: Command Start failed, or completed with errors.
2020/12/08 20:35:38 CLSRSC-117: Failed to start Oracle Clusterware stack from the Grid Infrastructure home /u01/app/19.0.0/grid 
 
After fixing the cause of failure Run opatchauto resume
 
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
 
OPatchauto session completed at Tue Dec  8 20:35:41 2020
Time taken to complete the session 17 minutes, 58 seconds
 
 opatchauto failed with error code 42
cs

 

위와 같이 패치에 실패하게 된다.

문제를 해결하고 opatchauto resume 으로 다시 진행하라는 내용이다.

 

문제는 위 내용 중 아래 내용을 보면 알 수 있다.

 

1
2
3
4
Cluster registry server error. Server error stack -
PROC-00053: The configured patch level, release patch level and the patch level on the other active cluster nodes are not all the same.
- end of server error stackCRS-2883: Resource 'ora.crsd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
cs

 

1번 노드의 패치가 끝나고 1번 노드의 crs를 올리려고 하니

현재 2번에 기동되어 있는 crs와 패치 레벨이 맞지 않는다는 내용이다.

그래서 2번 노드의 crs를 내리고 resume을 진행해 준다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
mprac1.localdomain@root:/root> opatchauto resume
 
OPatchauto session is initiated at Tue Dec  8 20:41:46 2020
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_08-41-50PM.log
Resuming existing session with id ERE4
 
Starting CRS service on home /u01/app/19.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/mprac1/crsconfig/crs_postpatch_apply_inplace_mprac1_2020-12-08_08-42-09PM.log
CRS service started successfully on home /u01/app/19.0.0/grid
 
OPatchAuto successful.
 
--------------------------------Summary--------------------------------
 
Patching is completed successfully. Please find the summary as follows:
 
Host:mprac1
CRS Home:/u01/app/19.0.0/grid
Version:19.0.0.0.0
Summary:
 
==Following patches were SUCCESSFULLY applied:
 
Patch: /tmp/31750108/31771877
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_20-21-59PM_1.log
 
Patch: /tmp/31750108/31772784
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_20-21-59PM_1.log
 
Patch: /tmp/31750108/31773437
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_20-21-59PM_1.log
 
Patch: /tmp/31750108/31780966
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_20-21-59PM_1.log
 
 
 
OPatchauto session completed at Tue Dec  8 20:47:54 2020
Time taken to complete the session 6 minutes, 10 seconds
cs

 

1번 노드의 Grid Home 패치가 성공적으로 완료됐다.

grid 계정으로 패치 인벤토리를 확인해 본다.

 

1
2
3
4
5
6
mprac1@grid:+ASM1:/home/grid> opatch lsinventory | grep "Patch description"
Patch description:  "TOMCAT RELEASE UPDATE 19.0.0.0.0 (31780966)"
Patch description:  "ACFS RELEASE UPDATE 19.9.0.0.0 (31773437)"
Patch description:  "OCW RELEASE UPDATE 19.9.0.0.0 (31772784)"
Patch description:  "Database Release Update : 19.9.0.0.201020 (31771877)"
mprac1@grid:+ASM1:/home/grid>
cs

 

위와 같이 잘 적용된 것을 확인할 수 있다.

현재 상태는 1번 노드의 crs는 기동되어 있는 상태이며 2번 노드의 crs는 내려가 있는 상태이다.

 

2번 노드의 Grid Home 패치하기 전에 1번 노드의 crs를 내리고

패치가 안된 2번 노드의 crs를 올려 놓은 상태로 만들어 놓아야 한다.

 

이렇게 하는 이유는 opatchauto의 경우 crs가 떠 있는 상태에서 진행하는 방식이라서 

2번 노드에서 opatchauto 명령어로 진행하면 2번 노드의 crs를 직접 올릴려고 하는데

앞서 발생한 에러와 마찬가지로 1, 2번 노드의 crs 패치 레벨이 맞지 않아

2번 노드 crs의 기동시 에러가 발생하여 패치 진행이 되지 않는다.

 

그래서 1번 노드의 crs를 내리고 2번 노드의 crs를 기동시켜 놓은 상태에서 opatchauto를 수행한다.

아래 내용은 위에 설명한 상황에서의 에러 내용이다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
mprac2@root:/root> export PATH=$PATH:/u01/app/19.0.0/grid/OPatch
mprac2@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME -nonrolling
 
OPatchauto session is initiated at Tue Dec  8 20:53:38 2020
 
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-08_08-53-47PM.log.
 
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_08-53-55PM.log
The id for this session is 8FKQ
OPATCHAUTO-72132: Grid is not running on the local host.
OPATCHAUTO-72132: Cannot start a new apply or rollback session when the local grid is not running.
OPATCHAUTO-72132: Please start grid service on the local host to start patching.
OPatchAuto failed.
 
OPatchauto session completed at Tue Dec  8 20:54:20 2020
Time taken to complete the session 0 minute, 43 seconds
 
 opatchauto failed with error code 42
cs

 

이제 1번 노드의 crs를 내리고 2번 노드의 crs를 올리고 진행해 보자

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
mprac2@root:/root> crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
mprac2@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME -nonrolling
 
OPatchauto session is initiated at Tue Dec  8 21:03:13 2020
 
System initialization log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-12-08_09-03-18PM.log.
 
Session log file is /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-12-08_09-03-51PM.log
The id for this session is S7SL
 
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/grid
Patch applicability verified successfully on home /u01/app/19.0.0/grid
 
 
Executing patch validation checks on home /u01/app/19.0.0/grid
Patch validation checks successfully completed on home /u01/app/19.0.0/grid
 
 
Bringing down CRS service on home /u01/app/19.0.0/grid
Prepatch operation log file location: /u01/app/grid/crsdata/mprac2/crsconfig/crs_prepatch_apply_inplace_mprac2_2020-12-08_09-04-36PM.log
CRS service brought down successfully on home /u01/app/19.0.0/grid
 
 
Start applying binary patch on home /u01/app/19.0.0/grid
Binary patch applied successfully on home /u01/app/19.0.0/grid
 
 
Starting CRS service on home /u01/app/19.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/mprac2/crsconfig/crs_postpatch_apply_inplace_mprac2_2020-12-08_09-20-11PM.log
CRS service started successfully on home /u01/app/19.0.0/grid
 
OPatchAuto successful.
 
--------------------------------Summary--------------------------------
 
Patching is completed successfully. Please find the summary as follows:
 
Host:mprac2
CRS Home:/u01/app/19.0.0/grid
Version:19.0.0.0.0
Summary:
 
==Following patches were SUCCESSFULLY applied:
 
Patch: /tmp/31750108/31771877
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_21-07-29PM_1.log
 
Patch: /tmp/31750108/31772784
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_21-07-29PM_1.log
 
Patch: /tmp/31750108/31773437
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_21-07-29PM_1.log
 
Patch: /tmp/31750108/31780966
Log: /u01/app/19.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-08_21-07-29PM_1.log
 
 
 
OPatchauto session completed at Tue Dec  8 21:26:10 2020
Time taken to complete the session 22 minutes, 57 seconds
cs

 

2번 노드 Grid Home도 정상적으로 패치가 완료되었다. 

4. Oracle Home 패치 수행(발생하는 이슈가 포함되어 있으므로 바로 따라하지 말고 내용 충분히 본 후 진행하기 바랍니다)

현재 2번 Grid Home 패치를 진행해 1번의 crs는 내려가 있고 2번 노드의 crs만 올라와 있는 상태이다.

현재 상태에서 1번 노드의 Oracle Home 패치를 진행해 본다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
mprac1@root:/root> echo $ORACLE_HOME
/u01/app/oracle/product/19.0.0/db_1
mprac1@root:/root> which opatch
/u01/app/oracle/product/19.0.0/db_1/OPatch/opatch
mprac1@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME
 
OPatchauto session is initiated at Wed Dec  9 10:18:08 2020
 
System initialization log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchautodb/systemconfig2020-12-09_10-18-22AM.log.
 
Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-12-09_10-18-29AM.log
The id for this session is TMKA
 
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Executing patch validation checks on home /u01/app/oracle/product/19.0.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/db_1
 
 
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
 
 
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
 
 
Performing prepatch operation on home /u01/app/oracle/product/19.0.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Start applying binary patch on home /u01/app/oracle/product/19.0.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Performing postpatch operation on home /u01/app/oracle/product/19.0.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Preparing home /u01/app/oracle/product/19.0.0/db_1 after database service restarted
No step execution required.........
 
 
Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/db_1
No step execution required.........
 
OPatchAuto successful.
 
--------------------------------Summary--------------------------------
 
Patching is completed successfully. Please find the summary as follows:
 
Host:mprac1
RAC Home:/u01/app/oracle/product/19.0.0/db_1
Version:19.0.0.0.0
Summary:
 
==Following patches were SKIPPED:
 
Patch: /tmp/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"
 
Patch: /tmp/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"
 
 
==Following patches were SUCCESSFULLY applied:
 
Patch: /tmp/31750108/31771877
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_10-19-13AM_1.log
 
Patch: /tmp/31750108/31772784
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_10-19-13AM_1.log
 
 
 
OPatchauto session completed at Wed Dec  9 10:27:58 2020
Time taken to complete the session 9 minutes, 52 seconds
cs

 

정상적으로 완료됐다.

 

2번 노드의 Oracle Home 패치를 진행해 본다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
mprac2@root:/root> opatchauto apply /tmp/31750108 -oh $ORACLE_HOME
 
OPatchauto session is initiated at Wed Dec  9 11:20:37 2020
 
System initialization log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchautodb/systemconfig2020-12-09_11-20-53AM.log.
 
Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-12-09_11-21-58AM.log
The id for this session is 1FAA
 
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Executing patch validation checks on home /u01/app/oracle/product/19.0.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/db_1
 
 
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/db_1
Successfully prepared home /u01/app/oracle/product/19.0.0/db_1 to bring down database service
 
 
Bringing down database service on home /u01/app/oracle/product/19.0.0/db_1
Database service successfully brought down on home /u01/app/oracle/product/19.0.0/db_1
 
 
Performing prepatch operation on home /u01/app/oracle/product/19.0.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Start applying binary patch on home /u01/app/oracle/product/19.0.0/db_1
Failed while applying binary patches on home /u01/app/oracle/product/19.0.0/db_1
 
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : mprac2->/u01/app/oracle/product/19.0.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19.0.0/db_1, host: mprac2.
Command failed:  /u01/app/oracle/product/19.0.0/db_1/OPatch/opatchauto  apply /tmp/31750108 -oh /u01/app/oracle/product/19.0.0/db_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.0.0/db_1/oraInst.loc -jre /u01/app/oracle/product/19.0.0/db_1/OPatch/jre -persistresult /u01/app/oracle/product/19.0.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_mprac2_rac_2.ser -analyzedresult /u01/app/oracle/product/19.0.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_mprac2_rac_2.ser
Command failure output: 
==Following patches FAILED in apply:
 
Patch: /tmp/31750108/31771877
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_11-22-52AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)' 
 
After fixing the cause of failure Run opatchauto resume
 
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
 
OPatchauto session completed at Wed Dec  9 11:32:30 2020
Time taken to complete the session 11 minutes, 56 seconds
 
 opatchauto failed with error code 42
cs

 

위의 내용과 같이 에러가 발생한다. 

 

/u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_11-22-52AM_1.log 로그 파일을 통해

내용을 확인해 보면 에러 내용은 아래와 같다.

 

1
2
3
4
5
6
[Dec 92020 1:54:32 PM] [INFO]     ENTERING METHOD: NApplyReportForPrereqs(OPatchPatch[] patches,OPatchNApplyOptions options)
[Dec 92020 1:54:32 PM] [INFO]     [OPSR-TIME] Loading cooked inventory
[Dec 92020 1:54:32 PM] [INFO]     [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 83 (MB)
[Dec 92020 1:54:32 PM] [INFO]     Stack Description: oracle.opatch.opatchsdk.OPatchException: Unable to create patchObject
                                    Possible causes are:
                                       ORACLE_HOME/inventory/oneoffs/31771877 is corrupted. PatchObject constructor: Input file "/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/31771877/etc/config/actions" or "/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/31771877/etc/config/inventory" does not exist.
cs

 

위 로그로 확인해 보니 /u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs 이 디렉토리 밑에 31771877 디렉토리가 존재하지 않았다.

1번 노드에서 scp를 통해서 내용을 아래와 같이 복사해 온다.

 

1
2
3
mprac1@oracle:MPRAC1:/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs> scp -r ./31771877 mprac2:/u01/app/oracle/product/19.0.0/db_1/inventory/oneoffs/
inventory.xml                                                                                                                                 100%  319KB  20.2MB/s   00:00    
actions.xml                                                                                                                                   100% 2372KB  95.4MB/s   00:00    
cs

 

2번 노드에서 다시 패치를 진행한다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
mprac2@root:/root> opatchauto resume
 
OPatchauto session is initiated at Wed Dec  9 14:06:27 2020
Session log file is /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/opatchauto2020-12-09_02-06-28PM.log
Resuming existing session with id 3QLX
 
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/db_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Executing patch validation checks on home /u01/app/oracle/product/19.0.0/db_1
Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0/db_1
 
 
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/db_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/db_1
Successfully prepared home /u01/app/oracle/product/19.0.0/db_1 to bring down database service
 
 
Bringing down database service on home /u01/app/oracle/product/19.0.0/db_1
Database service successfully brought down on home /u01/app/oracle/product/19.0.0/db_1
 
 
Performing prepatch operation on home /u01/app/oracle/product/19.0.0/db_1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Start applying binary patch on home /u01/app/oracle/product/19.0.0/db_1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Performing postpatch operation on home /u01/app/oracle/product/19.0.0/db_1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0/db_1
 
 
Starting database service on home /u01/app/oracle/product/19.0.0/db_1
Database service successfully started on home /u01/app/oracle/product/19.0.0/db_1
 
 
Preparing home /u01/app/oracle/product/19.0.0/db_1 after database service restarted
No step execution required.........
 
 
Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/db_1
SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/db_1
 
OPatchAuto successful.
 
--------------------------------Summary--------------------------------
 
Patching is completed successfully. Please find the summary as follows:
 
Host:mprac2
RAC Home:/u01/app/oracle/product/19.0.0/db_1
Version:19.0.0.0.0
Summary:
 
==Following patches were SKIPPED:
 
Patch: /tmp/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"
 
Patch: /tmp/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"
 
Patch: /tmp/31750108/31771877
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_14-06-45PM_1.log
Reason: /tmp/31750108/31771877 is not required to be applied to oracle home /u01/app/oracle/product/19.0.0/db_1
 
 
==Following patches were SUCCESSFULLY applied:
 
Patch: /tmp/31750108/31772784
Log: /u01/app/oracle/product/19.0.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-12-09_14-07-21PM_1.log
 
 
Patching session reported following warning(s): 
_________________________________________________
 
[WARNING] The database instance 'MPRAC2' from '/u01/app/oracle/product/19.0.0/db_1', in host'mprac2' is not running. SQL changes, if any,  will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.
 
OPatchauto session completed at Wed Dec  9 14:08:35 2020
Time taken to complete the session 2 minutes, 9 seconds
cs

 

위 WARNING의 경우는 datapatch를 진행하지 않아서 진행하라는 의미이다.

 

1번과 2번에 적용된 패치 정보를 인벤토리를 통해서 확인해 보자 

 

1
2
3
4
5
6
7
mprac1@oracle:MPRAC1:/home/oracle> opatch lsinventory | grep "Patch description"
Patch description:  "OCW RELEASE UPDATE 19.9.0.0.0 (31772784)"
Patch description:  "Database Release Update : 19.9.0.0.201020 (31771877)"
 
mprac2@oracle:MPRAC2:/home/oracle> opatch lsinventory | grep "Patch description"
Patch description:  "OCW RELEASE UPDATE 19.9.0.0.0 (31772784)"
Patch description:  "Database Release Update : 19.9.0.0.201020 (31771877)"
cs

 

위와 같이 잘 적용되어 있는 것을 확인할 수 있다. 

5. Post-Installation

1) datapatch 적용

먼저 인스턴스를 기동하고 pluggable DB를 모두 open 시켜준다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
mprac1@oracle:MPRAC1:/home/oracle> sqlss
 
SQL*Plus: Release 19.0.0.0.0 - Production on 수 12월 9 15:17:19 2020
Version 19.9.0.0.0
 
Copyright (c) 19822020, Oracle.  All rights reserved.
 
휴지 인스턴스에 접속되었습니다.
 
SQL> startup
ORACLE 인스턴스가 시작되었습니다.
 
Total System Global Area 3221224552 bytes
Fixed Size                  9141352 bytes
Variable Size             822083584 bytes
Database Buffers         2382364672 bytes
Redo Buffers                7634944 bytes
데이터베이스가 마운트되었습니다.
데이터베이스가 열렸습니다.
SQL> alter pluggable database all open;
 
플러그인할 수 있는 데이터베이스가 변경되었습니다.
 
SQL> quit
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0에서 분리되었습니다.
cs

 

datapatch를 수행한다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
mprac1@oracle:MPRAC1:/home/oracle> cd $ORACLE_HOME/OPatch
mprac1@oracle:MPRAC1:/u01/app/oracle/product/19.0.0/db_1/OPatch> ./datapatch -verbose
SQL Patching tool version 19.9.0.0.0 Production on Wed Dec  9 15:26:33 2020
Copyright (c) 20122020, Oracle.  All rights reserved.
 
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_846331_2020_12_09_15_26_33/sqlpatch_invocation.log
 
Connecting to database...OK
Gathering database info...done
 
Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)
 
Bootstrapping registry and package to current versions...done
Determining current state...done
 
Current state of interim SQL patches:
  No interim patches found
 
Current state of release update SQL patches:
  Binary registry:
    19.9.0.0.0 Release_Update 200930183249: Installed
  PDB CDB$ROOT:
    Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 20/12/07 15:34:33.745914
  PDB MPPDB:
    Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 20/12/07 15:56:03.693390
  PDB PDB$SEED:
    Applied 19.3.0.0.0 Release_Update 190410122720 successfully on 20/12/07 15:56:03.693390
 
Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED MPPDB
    No interim patches need to be rolled back
    Patch 31771877 (Database Release Update : 19.9.0.0.201020 (31771877)):
      Apply from 19.3.0.0.0 Release_Update 190410122720 to 19.9.0.0.0 Release_Update 200930183249
    No interim patches need to be applied
 
Installing patches...
Patch installation complete.  Total patches installed: 3
 
Validating logfiles...done
Patch 31771877 apply (pdb CDB$ROOT): WITH ERRORS (RETRYABLE)
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_MPRAC_CDBROOT_2020Dec09_15_31_37.log (errors)
  -> Error at line 475: script rdbms/admin/standard.sql
      - ORA-00060: 자원 대기중 교착 상태가 검출되었습니다
Patch 31771877 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_MPRAC_PDBSEED_2020Dec09_15_58_50.log (no errors)
Patch 31771877 apply (pdb MPPDB): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_MPRAC_MPPDB_2020Dec09_15_58_49.log (no errors)
 
Adding patches to retry installation queue...done
Retry installation queue:
  For the following PDBs: CDB$ROOT
    No interim patches need to be rolled back
    Patch 31771877 (Database Release Update : 19.9.0.0.201020 (31771877)):
      Apply from 19.3.0.0.0 Release_Update 190410122720 to 19.9.0.0.0 Release_Update 200930183249
    No interim patches need to be applied
 
Installing patches...
Patch installation complete.  Total patches installed: 1
 
Validating logfiles...done
Patch 31771877 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/31771877/23869227/31771877_apply_MPRAC_CDBROOT_2020Dec09_16_21_26.log (no errors)
SQL Patching tool complete on Wed Dec  9 16:39:08 2020
cs

 

위와 같이 정상적으로 완료되었다.

이제 invalid object 체크 및 컴파일을 위해 utlrp를 수행해 준다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
mprac1@oracle:MPRAC1:/home/oracle> cd $ORACLE_HOME/rdbms/admin
mprac1@oracle:MPRAC1:/u01/app/oracle/product/19.0.0/db_1/rdbms/admin> sqlss
 
SQL*Plus: Release 19.0.0.0.0 - Production on 수 12월 9 16:45:43 2020
Version 19.9.0.0.0
 
Copyright (c) 19822020, Oracle.  All rights reserved.
 
 
다음에 접속됨:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
 
SQL> @utlrp.sql 
cs

 

이제 모든 서비스를 정상적으로 기동한다.

6. 적용된 사항 확인

패치 적용된 부분들을 Query 해 본다.

 

1
2
3
4
5
mprac1@grid:+ASM1:/home/grid> crsctl query crs releasepatch
Oracle Clusterware release patch level is [376483838] and the complete list of patches [31771877 31772784 31773437 31780966 ] have been applied on the local node. The release patch string is [19.9.0.0.0].
 
mprac2@grid:+ASM2:/home/grid> crsctl query crs releasepatch
Oracle Clusterware release patch level is [376483838] and the complete list of patches [31771877 31772784 31773437 31780966 ] have been applied on the local node. The release patch string is [19.9.0.0.0].
cs

 

sqlplus로 조회해 본다.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
mprac1@oracle:MPRAC1:/home/oracle> sqlplus / as sysdba
 
SQL*Plus: Release 19.0.0.0.0 - Production on 수 12월 9 16:51:44 2020
Version 19.9.0.0.0
 
Copyright (c) 19822020, Oracle.  All rights reserved.
 
 
다음에 접속됨:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0
 
SQL> SELECT   CON_ID
        ,TO_CHAR (ACTION_TIME, 'YYYY-MM-DD')    AS ACTION_TIME
        ,PATCH_ID
        ,PATCH_TYPE
        ,ACTION
        ,DESCRIPTION
        ,SOURCE_VERSION
        ,TARGET_VERSION
FROM     CDB_REGISTRY_SQLPATCH
ORDER BY CON_ID, ACTION_TIME, PATCH_ID;
 
 
    CON_ID ACTION_TIM   PATCH_ID PATCH_TYPE ACTION     DESCRIPTION                                                  SOURCE_VER TARGET_VER
---------- ---------- ---------- ---------- ---------- ------------------------------------------------------------ ---------- ----------
         1 2020-12-07   29517242 RU         APPLY      Database Release Update : 19.3.0.0.190416 (29517242)         19.1.0.0.0 19.3.0.0.0
         1 2020-12-09   31771877 RU         APPLY      Database Release Update : 19.9.0.0.201020 (31771877)         19.3.0.0.0 19.9.0.0.0
         1 2020-12-09   31771877 RU         APPLY      Database Release Update : 19.9.0.0.201020 (31771877)         19.3.0.0.0 19.9.0.0.0
         2 2020-12-07   29517242 RU         APPLY      Database Release Update : 19.3.0.0.190416 (29517242)         19.1.0.0.0 19.3.0.0.0
         2 2020-12-09   31771877 RU         APPLY      Database Release Update : 19.9.0.0.201020 (31771877)         19.3.0.0.0 19.9.0.0.0
         3 2020-12-07   29517242 RU         APPLY      Database Release Update : 19.3.0.0.190416 (29517242)         19.1.0.0.0 19.3.0.0.0
         3 2020-12-09   31771877 RU         APPLY      Database Release Update : 19.9.0.0.201020 (31771877)         19.3.0.0.0 19.9.0.0.0
 
7 행이 선택되었습니다.
 
SQL> 
cs

 

7. 총평

여러 에러를 겪으면서 완료했는데 충분히 준비하고 진행한 상황에서도 에러를 겪게된다.

에러 중에는 Grid Home을 shared라고 오인식 하는 버그 같은 내용도 있고

readme 문서의 내용으로 진행해도 정상적으로 진행되지 않는 등 readme 문서가 부실하다는 느낌을 받게 된다. 

이전에도 경험하고 느낀 점이지만 오라클 패치의 readme 문서는 잘못 기재한 경우도 많고 

설치시에 내용을 세세히 다 다루지 않는 거 같다.

 

매번 느끼는 거지만 오라클의 경우 패치 적용하는 과정에 대해 문서가 패치를 진행하는 엔지니어에게나

패치를 고려하고 있는 사용자에게는 불친절하다라는 느낌을 받게 된다. 

이 부분은 오라클에서도 고쳐야할 부분인거 같다.

 

마지막으로 현재 작성한 내용이 길어졌는데 유저 입장에서 일일이 세세한 부분까지 확인하면서

읽을 수 있도록 하기 위해 패치 설치시에 발생한 내용을 다 넣어서 그렇게 되었다.ㅏ

이런 부분도 참고가 될 수 있기를 바란다.

 

번호 제목 글쓴이 날짜 조회 수
40 Oracle Database 19c Patch Update(단일 인스턴스 오라클 DB 19.3 에서 19.9로 RU Update) 명품관 2020.11.24 34923
39 New version of our Hands-On Lab available for download "Upgrade/Migrate/Consolidate to Oracle 12c" HOL 명품관 2015.12.22 8992
38 How to Enable/Disable Oracle Binary Option [1] Talros 2016.03.09 8620
37 19c RAC OJVM 패치작업 우뽕 2020.04.22 5209
» Oracle RAC RU rolling patching from 19.3 to 19.9.0.0.201020(Patch 31750108) [1] file 명품관 2020.12.08 5153
35 Oracle 19c RAC 설치 with Oracle Linux 8.2 - 2 (Grid Infrastructure) file 명품관 2020.11.19 4935
34 Oracle 19c RAC 설치 with Oracle Linux 8.2 - 1 (OS 및 스토리지 설정) 명품관 2020.11.14 4742
33 ASM 에 spfile 등록 및 삭제 하기 우뽕 2020.07.27 4432
32 오라클 21c RAC 설치 with Oracle Linux 8.2 - 1 (OS 및 스토리지 설정) 명품관 2021.12.01 2987
31 Oracle 19c RAC 설치 with Oracle Linux 8.2 - 5 (DBCA로 Database 생성) file 명품관 2020.11.19 2805
30 19c RAC 2Node 설치 -1 [ 환경 ,OS 세팅 ] file 우뽕 2020.04.14 2647
29 19c RAC - Manual Patch 적용방법 우뽕 2020.05.05 2334
28 Oracle EM(Enterprise Manager) 13.4 설치 및 구성 - 2(EM 설치) file 명품관 2020.11.27 2241
27 Oracle 19c RAC 설치 with Oracle Linux 8.2 - 3 (사용할 Disk Group 생성) file 명품관 2020.11.19 1478
26 DBCA/DBUA 사용시 마지막 단계 Finish 버튼 클릭 후 행 현상 발생 시 체크 사항 명품관 2015.12.17 1460
25 19c RAC 2Node 설치-8 [ Patch ] 우뽕 2020.04.22 1396
24 19c RAC 2Node 삭제 하기 file 우뽕 2020.04.22 1336
23 메뉴얼하게 DB 추가 작업 - GRID 리소스에 DB 추가 작업 우뽕 2020.07.27 1250
22 Oracle EM(Enterprise Manager) 13.4 설치 및 구성 - 1(Repository DB 설치) file 명품관 2020.11.23 1146
21 Automatic Storage Management (ASM) 사용시 고려해야 할 사항 [2] 우뽕 2020.02.06 1041
위로