Failed
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
rsysdev3 bi db
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
721 vmbmospatch_bi_rsysdev3_JAN2025_20250430_094120vmospatch_req
541 2025-04-30 09:43:04 UN 10
python owfe_req_details_v1.py -reqid 721
nohup python processor.py -wfid 541 -reqid 721 &
python owfe_force_comp_task.py 1616
nohup python processor.py -wfid 541 -reqid 721 -resume </dev/null >/dev/null 2>&1
&
+++++
Error
++++++
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/oci/included/x86_64/
repodata/repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
Trying other mirror.
failure: repodata/repomd.xml from ol7_ociyum_config: [Errno 256] No more mirrors to
try.
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/oci/included/x86_64/
repodata/repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
2025-04-30 09:50:21 [ERROR] (root) <MainThread> - STDOUT:
-phx
Loaded plugins: ulninfo, versionlock
2025-04-30 09:50:21 [ERROR] (root) <MainThread> - EXITED:
2025-04-30 09:50:21 [ERROR] (root) <MainThread> - <dba.rpc.Result exited: "1" cmd:
"sudo grep phx /etc/yum/vars/oc ...">
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
rsysdev3 rman db
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
722 vmbmospatch_rc_rsysdev3_JAN2025_20250430_094120vmospatch_req
541 2025-04-30 09:57:50 FA 10
python owfe_req_details_v1.py -reqid 722
nohup python processor.py -wfid 541 -reqid 722 &
python owfe_force_comp_task.py 7211
nohup python processor.py -wfid 541 -reqid 722 -resume </dev/null >/dev/null 2>&1
&
+++++
Error
++++++
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/addons/x86_64/repodata/
repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
Trying other mirror.
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/oci/included/x86_64/
repodata/repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
Trying other mirror.
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64/
repodata/repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
Trying other mirror.
Excluding 2 updates due to versionlock (use "yum versionlock status" to show them)
Resolving Dependencies
Transaction Summary
===================================================================================
===================================================================================
=======
Install 3 Packages
Upgrade 157 Packages
Remove 3 Packages
Total size: 644 M
Total download size: 74 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for ol7_ociyum_config
jre-8u361-linux-x64.rpm FAILED
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/oci/included/x86_64/
getPackage/jre-8u361-linux-x64.rpm: [Errno 14] curl#22 - "The requested URL
returned error: 503"
Trying other mirror.
Error downloading packages:
jre1.8-1.8.0_361-fcs.x86_64: [Errno 256] No more mirrors to try.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
rsysdev3 AQ db
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[root@rijms-aq11 tmp]# curl -I
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/addons/x86_64/repodata/
repomd.xml
HTTP/1.1 503 Service Unavailable
Server: squid/4.6
Mime-Version: 1.0
Date: Wed, 30 Apr 2025 10:42:09 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 1961
X-Squid-Error: ERR_DNS_FAIL 0
Vary: Accept-Language
Content-Language: en
curl: (56) Received HTTP code 503 from proxy after CONNECT
[root@rijms-aq11 tmp]#
looks like the yum repo url is not reachable from all of the rsysdev3 servers
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
rsysdev1 BI db
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1141 vmbmospatch_bi_rsysdev1_JAN2025_20250430_105730vmospatch_req
1001 2025-04-30 11:00:36 IN 10
---same 503 error
********** COMPLETED BY IMPLEMENTING THE BELOW FIX ON BOTH THE NODES **************
sudo yum-config-manager --disable ol7_UEKR5 ol7_addons ol7_ociyum_config
ol7_optional_latest
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
rsysdev1 AQ db
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
issue 1
+++++++++
2025-04-30 12:44:55 [ERROR] (shutdown_inst.py.rijms-
aq11.rsysdev1pod.dev1phx.oraclevcn.com.d101003x_phx1mg.rijms-
aq11.rsysdev1pod.dev1phx.oraclevcn.com) - Cannot run command
/u01/app/oracle/product/19.0.0.0/dbhome_1/bin/srvctl stop instance -db
d101003x_phx1mg -instance d101003x1 -o "NORMAL" -failover
2025-04-30 12:44:55 [ERROR] (shutdown_inst.py.rijms-
aq11.rsysdev1pod.dev1phx.oraclevcn.com.d101003x_phx1mg.rijms-
aq11.rsysdev1pod.dev1phx.oraclevcn.com) - STDERR:
2025-04-30 12:44:55 [ERROR] (shutdown_inst.py.rijms-
aq11.rsysdev1pod.dev1phx.oraclevcn.com.d101003x_phx1mg.rijms-
aq11.rsysdev1pod.dev1phx.oraclevcn.com) - STDOUT:
PRCD-1131 : Failed to stop database d101003x_phx1mg and its services on nodes
rijms-aq11
PRCR-1133 : Failed to stop database d101003x_phx1mg and its running services
PRCR-1132 : Failed to stop resources using a filter
CRS-2974: unable to act on resource 'ora.d101003x_phx1mg.db' on server 'rijms-aq11'
because that would require stopping or relocating resource
'ora.d101003x_phx1mg.d101003x_d101003aq1.paas.oracle.com.svc' but the appropriate
force flag was not specified
oracle@rijms-aq11 ~ d101003x1 >$srvctl status service -d $ORACLE_UNQNAME
Service d101003aq1_001_01 is running on instance(s) d101003x2
Service d101003aq1_admin is running on instance(s) d101003x2
Service d101003x_admin is running on instance(s) d101003x2
Service d101003x_d101003aq1.paas.oracle.com is running on instance(s)
d101003x1,d101003x2
ora.d101003x_phx1mg.d101003x_d101003aq1.paas.oracle.com.svc --> This service is
running in both the nodes and the automation failed to stop the instance
srvctl stop instance -d $ORACLE_UNQNAME -i $ORACLE_SID -failover
****** STOPPED THE SERVICE d101003x_d101003aq1.paas.oracle.com ON BOTH THE NODES
AND CONTINUED *********
issue 2
+++++++++
failure: repodata/repomd.xml from ol7_addons: [Errno 256] No more mirrors to try.
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/addons/x86_64/repodata/
repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
2025-04-30 01:21:37 [ERROR] (root) <MainThread> - STDOUT:
-phx
Loaded plugins: ulninfo, versionlock
2025-04-30 01:21:37 [ERROR] (root) <MainThread> - EXITED:
2025-04-30 01:21:37 [ERROR] (root) <MainThread> - <dba.rpc.Result exited: "1" cmd:
"sudo grep phx /etc/yum/vars/oc ...">
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++
Fixes
+++++++++++
sudo yum-config-manager --disable ol7_UEKR5
yum repolist enabled | grep ol7_UEKR5
sudo yum-config-manager --disable ol7_addons
yum repolist enabled | grep ol7_addons
sudo yum-config-manager --disable ol7_ociyum_config
yum repolist enabled | grep ol7_ociyum_config
sudo yum-config-manager --disable ol7_optional_latest
yum repolist enabled | grep ol7_optional_latest
sudo yum-config-manager --disable ol7_UEKR5 ol7_addons ol7_ociyum_config
ol7_optional_latest
yum repolist enabled
yum-config-manager ol7_UEKR5 | grep enabled
yum-config-manager --disable ol7_UEKR5
yum-config-manager ol7_addons | grep enabled
yum-config-manager --disable ol7_addons
yum-config-manager ol7_ociyum_config | grep enabled
yum-config-manager --disable ol7_ociyum_config
yum-config-manager ol7_optional_latest | grep enabled
yum-config-manager --disable ol7_optional_latest
These repos were causing repeated HTTP 503 - Service Unavailable errors during
patching
These repos were not required for the kernel version or the current patch scope on
our servers.
kernel version - 4.12
• ol7_UEKR5 is used only if we’re running or upgrading to UEK5, but we’re
on UEK4.
• ol7_ociyum_config and ol7_optional_latest are for OCI-specific or
optional components not relevant to our current patch cycle.
Repository ol7_UEKR4_archive is listed more than once in the configuration
Repository ol7_security_validation is listed more than once in the configuration
http://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/UEKR5/x86_64/repodata/
repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/addons/x86_64/repodata/
repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
Trying other mirror.
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/oci/included/x86_64/
repodata/repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
Trying other mirror.
https://yum.us-phoenix-1.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64/
repodata/repomd.xml: [Errno 14] curl#22 - "The requested URL returned error: 503"
Trying other mirror.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++
[root@ridb-yymby1 dbserver_patch_250417]# ./patchmgr -dbnodes ../dbs_group -
precheck -iso_repo /u01/patching/25.1.2.0.0.250213/p37471710_251000_Linux-x86-
64.zip -target_version 25.1.2.0.0.250213.1 -nomodify_at_prereq -
allow_non_signed_repo
***********************************************************************************
*************************
NOTE patchmgr release: 25.250417 (always check MOS 1553103.1 for the latest
release of dbserver.patch.zip)
NOTE
WARNING Do not interrupt the patchmgr session.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot database nodes during update or rollback.
WARNING Do not open logfiles in write mode and do not try to alter them.
***********************************************************************************
*************************
2025-05-02 09:58:17 +0000 :INFO : Checking hosts connectivity via
ICMP/ping
2025-05-02 09:58:18 +0000 :INFO : Hosts Reachable: [ridb-yymby2]
2025-05-02 09:58:18 +0000 :INFO : All hosts are reachable via ICMP/ping
2025-05-02 09:58:19 +0000 :Working: Verify SSH equivalence for the root user
to ridb-yymby2
2025-05-02 09:58:19 +0000 :SUCCESS: Verify SSH equivalence for the root user
to ridb-yymby2
2025-05-02 09:58:22 +0000 :Working: Initiate precheck on 1 node(s)
2025-05-02 09:58:24 +0000 :INFO : Preparing nodes: ridb-yymby2
2025-05-02 09:58:24 +0000 :Working: Check for enough free space on ridb-
yymby2 to transfer and unzip files.
2025-05-02 09:58:28 +0000 :SUCCESS: Check for enough free space on ridb-
yymby2 to transfer and unzip files.
2025-05-02 09:58:50 +0000 :Working: dbnodeupdate.sh running a precheck for
the conventional update on node(s).
2025-05-02 10:01:43 +0000 :INFO : Node ridb-yymby2 will not be Live
Updated.
2025-05-02 10:01:54 +0000 :SUCCESS: Initiate precheck on node(s).
2025-05-02 10:01:56 +0000 :SUCCESS: Completed run of command: ./patchmgr -
dbnodes ../dbs_group -precheck -iso_repo
/u01/patching/25.1.2.0.0.250213/p37471710_251000_Linux-x86-64.zip -target_version
25.1.2.0.0.250213.1 -nomodify_at_prereq -allow_non_signed_repo
2025-05-02 10:01:56 +0000 :INFO : Precheck performed on dbnode(s) in
file ../dbs_group: [ridb-yymby2]
2025-05-02 10:01:56 +0000 :INFO : Current image version on dbnode(s) is:
2025-05-02 10:01:56 +0000 :INFO : ridb-yymby2: 24.1.3.0.0.240809.1
2025-05-02 10:01:56 +0000 :INFO : For details, check the following files
in /u01/patching/22.1.32.0.0.250205/dbserver_patch_250417:
2025-05-02 10:01:56 +0000 :INFO : - <dbnode_name>_dbnodeupdate.log
2025-05-02 10:01:56 +0000 :INFO : - patchmgr.log
2025-05-02 10:01:56 +0000 :INFO : - patchmgr.trc
2025-05-02 10:01:56 +0000 :INFO : Exit status:0
2025-05-02 10:01:56 +0000 :INFO : Exiting.
[root@ridb-yymby1 dbserver_patch_250417]#