Table of Contents
Overview
- Flex ASM can be configured for Standard Cluster and for Flex cluster
- Changing the cluster mode requires cluster downtime.
- Oracle does not support changing an Oracle Flex Cluster to an Oracle Clusterware standard Cluster.
- Oracle Flex Cluster requires Grid Naming Service (GNS). Zone delegation is not required.
Verify ASM instance status
Check ASM cluster mode and state [grid@gract1 ~]$ asmcmd showclustermode ASM cluster : Flex mode enabled [grid@gract1 ~]$ asmcmd showclusterstate Normal [grid@gract1 ~/ASM]$ oifcfg getif eth1 192.168.1.0 global public eth2 192.168.2.0 global cluster_interconnect,asm --> Only 2 networks present : ASM traffic uses standard interconnect [grid@gract1 ~]$ olsnodes -n -s -a gract1 1 Active Hub gract2 2 Active Hub gract3 3 Active Hub --> Note: only HUP nodes with access to storage can act as an ASM FLEX instance ASM resource status: [grid@gract1 ~]$ crs | egrep 'STATE|asm|--' NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.asm ONLINE ONLINE gract1 STABLE ora.asm ONLINE ONLINE gract3 STABLE ora.asm ONLINE ONLINE gract2 STABLE $ srvctl status asm -detail ASM is running on gract3,gract2,gract1 ASM is enabled.
Testing ASM connection:
How many ASM connections do we expect ? [grid@gract2 ASM]$ srvctl status asm -detail ASM is running on gract2,gract1 ASM is enabled. Let's connect to the global +ASM service [grid@gract2 ASM]$ sqlplus sys/sys@gract1:/+ASM as sysasm @v SQL> select INSTANCE_NUMBER,INSTANCE_NAME,STATUS, HOST_NAME from v$instance; INSTANCE_NUMBER INSTANCE_N STATUS HOST_NAME --------------- ---------- ------------ ------------------------------ 1 +ASM1 STARTED gract1.example.com [grid@gract2 ASM]$ sqlplus sys/sys@gract2:/+ASM as sysasm @v SQL> select INSTANCE_NUMBER,INSTANCE_NAME,STATUS, HOST_NAME from v$instance; INSTANCE_NUMBER INSTANCE_N STATUS HOST_NAME --------------- ---------- ------------ ------------------------------ 2 +ASM2 STARTED gract2.example.com [grid@gract2 ASM]$ sqlplus sys/sys@gract3:/+ASM as sysasm ERROR: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor --> This error is expected as +ASM servcice is not running on gract3
ASM Network
- With Flex ASM Oracle 12c, a new type of network is called the ASM network.
- This network is used for communication between ASM and its clients and is accessible on all the nodes.
- All ASM clients in the cluster have access to one or ore ASM network.
- Also it is possible to configure single network can perform both function as a private and an ASM network (oifcfg getif, you should see cluster_interconnect,asm)
ASM Listeners
- To support FLex ASM, a set of ASM listeners are configured for every ASM network.
- Up to three ASM listener addresses are registred as remote listeners in each client database instance.
- All clients connections are load balanced across the entire set of ASM instances
ASM Listener Details
Note: ASM listeners only run on nodes with ASM instances [grid@gract2 ~]$ srvctl config asm -a ASM home: /u01/app/121/grid Password file: +DATA/orapwASM ASM listener: LISTENER ASM is enabled. ASM instance count: 2 Cluster ASM listener: ASMNET1LSNR_ASM Check clusterware resources [root@gract1 var]# crs | egrep 'asm|ASMNET|--|STATE_DETAILS' Rescource NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE gract1 STABLE ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE gract2 STABLE ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE gract3 STABLE Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.asm 1 ONLINE ONLINE gract1 Started,STABLE ora.asm 3 ONLINE ONLINE gract2 Started,STABLE a ASM listener details [grid@gract2 ~]$ lsnrctl services ASMNET1LSNR_ASM LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 07-AUG-2014 17:14:01 Copyright (c) 1991, 2013, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM))) Services Summary... Service "+ASM" has 2 instance(s). Instance "+ASM2", status READY, has 2 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.2.112)(PORT=1521))) "DEDICATED" established:1213 refused:0 state:ready LOCAL SERVER Instance "+ASM3", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.2.113)(PORT=1521))) The command completed successfully The ASM instance automatically registers with ASMListener at port 1521 using the asm or cluster_interconnect,asm network [grid@gract1 ASM]$ oifcfg getif -type cluster_interconnect,asm eth2 192.168.2.0 global cluster_interconnect,asm --> Instance "+ASM2" runs on gract2 serviced by ASMNET1LSNR_ASM listening on 192.168.2.112 Port 1521 Instance "+ASM3" runs on gract3 serviced by ASMNET1LSNR_ASM listening on 192.168.2.113 Port 1521 Check resource dependencies [root@gract1 var]# crsctl status resource ora.asm -f | grep -i DEP RELOCATE_BY_DEPENDENCY=0 START_DEPENDENCIES=weak(ora.LISTENER.lsnr) pullup(ora.ASMNET1LSNR_ASM.lsnr) hard(ora.ASMNET1LSNR_ASM.lsnr) STOP_DEPENDENCIES=hard(intermediate:ora.ASMNET1LSNR_ASM.lsnr) --> ASM instance starts and stops ASM listener
ADVM Proxy
- ASM Dynamic Volume Manager (ADVM) Proxy is a special Oracle instance.
- It enables ADVM to connect to Flex ASM and is required to run on the same node as ADVM and ACFS.
- It can be shutdown when ACFS is not running.
- The ADVM proxy instance has its ORACLE_SID set to +APX<node number>
ADVM Proxy details
$ my_crs_stat NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ Name Target State Server State ora.proxy_advm ONLINE ONLINE gract1 STABLE ora.proxy_advm ONLINE ONLINE gract2 STABLE ora.proxy_advm ONLINE ONLINE gract3 STABLE $ ps -elf | grep pmon | grep APX 0 S grid 4019 1 0 80 0 - 350587 semtim 07:53 ? 00:00:00 apx_pmon_+APX3 $ srvctl status asm -proxy ADVM proxy is running on node gract3,gract2,gract1 Stop/Start ADVM proxy server on a certain node: $ srvctl status asm -proxy ADVM proxy is running on node gract3,gract2,gract1 $ srvctl stop asm -proxy -node gract3 $ srvctl status asm -proxy ADVM proxy is running on node gract2,gract1 $ ps -elf | grep pmon | grep APX $ srvctl start asm -proxy -node gract3 $ ps -elf | grep pmon | grep APX 0 S grid 4967 1 0 80 0 - 350587 semtim 08:09 ? 00:00:00 apx_pmon_+APX3 $ srvctl status asm -proxy ADVM proxy is running on node gract3,gract2,gract1
Remove and add an ASM instance to our FLEX ASM cluster
Remove an ASM instance to our FLEX ASM cluster [grid@gract1 ~]$ srvctl modify asm -count 2 [grid@gract1 ~]$ crs | egrep 'STATE|asm|--' NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.asm ONLINE ONLINE gract3 STABLE ora.asm ONLINE ONLINE gract2 STABLE [grid@gract1 ~]$ srvctl status asm -detail ASM is running on gract3,gract2 ASM is enabled. Add an ASM instance to our FLEX ASM cluster [grid@gract1 ~]$ srvctl modify asm -count 3 [grid@gract1 ~]$ srvctl status asm -detail ASM is running on gract3,gract2 ASM is enabled. --> Here we need to startup the ASM instance manually [grid@gract1 ~]$ srvctl start asm [grid@gract1 ~]$ crs | egrep 'STATE|asm|--' NAME TARGET STATE SERVER STATE_DETAILS ------------------------- ---------- ---------- ------------ ------------------ ora.asm ONLINE ONLINE gract1 STABLE ora.asm ONLINE ONLINE gract3 STABLE ora.asm ONLINE ONLINE gract2 STABLE [grid@gract1 ~]$ srvctl status asm -detail ASM is running on gract3,gract2,gract1 ASM is enabled.
Some useful commands to administer Flex ASM
$ srvctl config asm ASM home: /u01/app/121/grid Password file: +DATA/orapwASM ASM listener: LISTENER ASM instance count: 3 Cluster ASM listener: ASMNET1LSNR_ASM $ srvctl status asm -detail ASM is running on gract3,gract2,gract1 ASM is enabled. $ srvctl status asm -proxy -detail ADVM proxy is running on node gract3,gract2,gract1 ADVM proxy is enabled $ crsctl status resource ora.asm -f | grep CARDINALITY= CARDINALITY=3
Monitor Flex ASM Connections:
SQL> select INST_ID,GROUP_NUMBER, INSTANCE_NAME, DB_NAME, INSTANCE_NAME||':'||DB_NAME client_id, STATUS from gv$asm_client;
INST_ID GROUP_NUMBER INSTANCE_N DB_NAME CLIENT_ID STATUS
---------- ------------ ---------- -------- -------------------- ------------
3 1 +ASM3 +ASM +ASM3:+ASM CONNECTED
3 1 gract1_1 gract1 gract1_1:gract1 CONNECTED
3 1 gract1_2 gract1 gract1_2:gract1 CONNECTED
3 0 +APX1 +APX +APX1:+APX CONNECTED
3 0 +APX2 +APX +APX2:+APX CONNECTED
3 0 +APX3 +APX +APX3:+APX CONNECTED
3 1 -MGMTDB _mgmtdb -MGMTDB:_mgmtdb CONNECTED
2 1 gract1_3 gract1 gract1_3:gract1 CONNECTED
Reduce/Increase ASM instance count
$ srvctl config asm ASM home: /u01/app/121/grid Password file: +DATA/orapwASM ASM listener: LISTENER ASM instance count: 2 Cluster ASM listener: ASMNET1LSNR_ASM $ srvctl status asm -detail ASM is running on gract3,gract2 $ srvctl modify asm -count 3 $ srvctl start asm -n gract1 $ srvctl status asm -detail ASM is running on gract3,gract2,gract1 ASM is enabled. $ srvctl config asm ASM home: /u01/app/121/grid Password file: +DATA/orapwASM ASM listener: LISTENER ASM instance count: 3 Cluster ASM listener: ASMNET1LSNR_ASM
Relocate ASM connections
Due to the BUG mentioned below I've tested ASM relocation feature against 12.1.0.2 Bug 18003355 : LNX64-12.1-ASM, EXECUTE "RELOCATE" COMMAND,HIT ORA-15344 for further details Setup testbase by reducing ASM instances to 2 in our 3-node cluster running 3 HUBs [oracle@gract1 ~]$ srvctl modify asm -count 2 [oracle@gract1 ~]$ srvctl status asm -detail ASM is running on gract2,gract1 ASM is enabled. --> ASM is running on gract1,gract2 SQL> @asm_cli INSTANCE_NAME -------------------- +ASM1 INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME CLIENT_ID STATUS ---------- ------------ -------------------- -------- -------------------- ------------ 1 2 cdbn1 cdbn cdbn1:cdbn CONNECTED 1 2 cdbn3 cdbn cdbn3:cdbn CONNECTED 2 2 cdbn2 cdbn cdbn2:cdbn CONNECTED cdbn2:cdbn -> connected to instance id 2 -> local ASM client cdbn3:cdbn -> connected to instance id 1 -> Remote ASM client cdbn1:cdbn -> connected to instance id 1 -> local ASM client What happens when we relocate an client ? 1) Relocate client cdbn2:cdbn when connected to INST_ID 1 SQL> alter system relocate client 'cdbn2:cdbn'; alter system relocate client 'cdbn2:cdbn' * ERROR at line 1: ORA-15344: client cdbn2:cdbn@+ASM1 not found INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME CLIENT_ID STATUS ---------- ------------ -------------------- -------- -------------------- ------------ 1 2 cdbn1 cdbn cdbn1:cdbn CONNECTED 1 2 cdbn3 cdbn cdbn3:cdbn CONNECTED 2 2 cdbn2 cdbn cdbn2:cdbn CONNECTED --> Error ORA-15344 . We need connect INST_ID 2 to relocate that client 'cdbn2:cdbn'; 2) Relocate the local client cdbn1:cdbn on INST_ID 1 SQL> alter system relocate client 'cdbn1:cdbn'; System altered. +ASM1 alert.log Mon Aug 11 14:19:10 2014 NOTE: relocating client cdbn1:cdbn:gract (reg:3580848907) NOTE: cleaned up ASM client cdbn1:cdbn:gract connection state Mon Aug 11 14:19:10 2014 NOTE: Flex client id 0x10005 [cdbn1:cdbn:gract] attempting to reconnect NOTE: registered owner id 0x10005 for cdbn1:cdbn:gract (reconnecting) NOTE: Flex client cdbn1:cdbn:gract re-registered, osid 17384, mbr 0x0, asmb 32283 (reg:1445977979) NOTE: client [cdbn1:cdbn:gract] declared 0 additional pending writes INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME CLIENT_ID STATUS ---------- ------------ -------------------- -------- -------------------- ------------ 1 2 cdbn1 cdbn cdbn1:cdbn CONNECTED 1 2 cdbn3 cdbn cdbn3:cdbn CONNECTED 2 2 cdbn2 cdbn cdbn2:cdbn CONNECTED --> ASM doesn't relocate the instance - instead it just recconnects to the local ASM instance 3) Relocate the Remote ASM client connection SQL> alter system relocate client 'cdbn3:cdbn'; ASM1+ alert.log Mon Aug 11 14:25:47 2014 NOTE: relocating client cdbn3:cdbn:gract (reg:959447971) NOTE: cleaned up ASM client cdbn3:cdbn:gract connection state Mon Aug 11 14:25:49 2014 NOTE: detected orphaned client id 0x10003. NOTE: client cdbn3:cdbn:gract id 0x10003 has reconnected to ASM inst +ASM2 (reg:309839056), or has been fenced INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME CLIENT_ID STATUS ---------- ------------ -------------------- -------- -------------------- ------------ 1 2 cdbn1 cdbn cdbn1:cdbn CONNECTED 2 2 cdbn2 cdbn cdbn2:cdbn CONNECTED 2 2 cdbn3 cdbn cdbn3:cdbn CONNECTED Note : The relocate not always worked ( sometimes I got a reconnect even for as REMOTE asn client ) In that case just repeat the the relocate command To relocate ASM instance back to INST_ID 1 connect to +ASM2 instance and run SQL> alter system relocate client 'cdbn3:cdbn'; Summary: Local ASM clients always tries to re-connect locally Remote ASM clients really fail over the remote note
Force a Flex ASM network connection by reducing ASM instance count
$ srvctl status asm -detail ASM is running on gract3,gract2,gract1 ASM is enabled. $ srvctl modify asm -count 2 $ srvctl status asm -detail ASM is running on gract3,gract2 ASM is enabled. --> ASM instance dropped on gract1 - Connected session still working SQL> select INSTANCE_NUMBER,INSTANCE_NAME,STATUS, HOST_NAME from v$instance; INSTANCE_NUMBER INSTANCE_N STATUS HOST_NAME --------------- ---------- ------------ ------------------------------ 1 gract1_1 OPEN gract1.example.com --> alert.log on gract1 reports that ASM is reconnecting to ASM instance +ASM3 Thu Oct 03 10:35:35 2013 NOTE: ASMB registering with ASM instance as client 0x10004 (reg:2698621261) NOTE: ASMB connected to ASM instance +ASM3 (Flex mode; client id 0x10004) NOTE: ASMB rebuilding ASM server state NOTE: ASMB rebuilt 1 (of 1) groups NOTE: ASMB rebuilt 17 (of 17) allocated files NOTE: fetching new locked extents from server NOTE: 0 locks established; 0 pending writes sent to server SUCCESS: ASMB reconnected & completed ASM server state SQL> select INST_ID,GROUP_NUMBER, INSTANCE_NAME, DB_NAME, INSTANCE_NAME||':'||DB_NAME client_id, STATUS from gv$asm_client where DB_NAME = 'gract1'; INST_ID GROUP_NUMBER INSTANCE_N DB_NAME CLIENT_ID STATUS ---------- ------------ ---------- -------- -------------------- ------------ 3 1 gract1_1 gract1 gract1_1:gract1 CONNECTED <-- ASM connection connected via Network 3 1 gract1_2 gract1 gract1_2:gract1 CONNECTED 2 1 gract1_3 gract1 gract1_3:gract1 CONNECTED
Force a Flex ASM network connection by killing ASM asmb process
Testing ASM instance crash Prepare a db session as user scott connecting to PDB1 using CDBN1 [oracle@gract1 ~]$ sqlplus scott/tiger@gract1/PDB1 SQL> select INSTANCE_NUMBER,INSTANCE_NAME,STATUS, HOST_NAME from v$instance; INSTANCE_NUMBER INSTANCE_N STATUS HOST_NAME --------------- ---------- ------------ ------------------------------ 1 cdbn1 OPEN gract1.example.com SQL> SQL> show con_name CON_NAME ------------------------------ PDB1 SQL> select user from dual; USER ------------------------------ SCOTT --> Don't close this session we will test whether this connection survives the ASM instance crash Connect to the local +ASM1 instance and check ASM client status SQL> @asm_cli INSTANCE_NAME -------------------- +ASM1 INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME CLIENT_ID STATUS ---------- ------------ -------------------- -------- -------------------- ------------ 1 2 cdbn1 cdbn cdbn1:cdbn CONNECTED 1 2 cdbn3 cdbn cdbn3:cdbn CONNECTED 2 2 cdbn2 cdbn cdbn2:cdbn CONNECTED +ASM1 instance serves a local client cdbn1:cdbn and a remote client : cdbn3:cdbn Lets kill +ASM1 instance [grid@gract1 trace]$ ps -elf | grep -i asm_asmb 0 S grid 32119 1 0 80 0 - 361252 pipe_w 09:37 ? 00:00:00 asm_asmb_+ASM1 [root@gract1 var]# kill -9 32119 Connect to +ASM2 instance and check ASM status INSTANCE_NAME -------------------- +ASM2 INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME CLIENT_ID STATUS ---------- ------------ -------------------- -------- -------------------- ------------ 2 2 cdbn1 cdbn cdbn1:cdbn CONNECTED 2 2 cdbn2 cdbn cdbn2:cdbn CONNECTED 2 2 cdbn3 cdbn cdbn3:cdbn CONNECTED --> Now we have 2 REMOTE ASM clients cdbn1:cdbn and cdbn3:cdbn All ASM operations are now serviced by INST_ID 2 ( +ASM2 ) Check SQL session status SQL> select user from dual; USER ------------------------------ SCOTT --> SQL session survived the ASM failover Note ASM instance is automatically restarted due to CW resources monitoring Check CW resources [root@gract1 var]# crs | egrep 'asm|---|INST' ------------------------- ---------- ---------- ------------ ------------------ Resource NAME INST TARGET STATE SERVER STATE_DETAILS --------------------------- ---- ------------ ------------ --------------- ----------------------------------------- ora.asm 1 ONLINE ONLINE gract1 Started,STABLE ora.asm 3 ONLINE ONLINE gract2 Started,STABLE Relocate ASM clients as needed SQL> alter system relocate client 'cdbn3:cdbn'; System altered. SQL> alter system relocate client 'cdbn1:cdbn'; System altered. SQL> @asm_cli INSTANCE_NAME -------------------- +ASM2 INST_ID GROUP_NUMBER INSTANCE_NAME DB_NAME CLIENT_ID STATUS ---------- ------------ -------------------- -------- -------------------- ------------ 1 2 cdbn1 cdbn cdbn1:cdbn CONNECTED 1 2 cdbn3 cdbn cdbn3:cdbn CONNECTED 2 2 cdbn2 cdbn cdbn2:cdbn CONNECTED
Reference
- http://www.oracleangels.com/2013/09/flex-asm-network-listeners-advm-proxy.html
- Bug 18003355 : LNX64-12.1-ASM, EXECUTE “RELOCATE” COMMAND,HIT ORA-15344 for further details
Thanks Helmut, this is good info.
Steve
Hi Helmut,
Is the output below, a correct output for a two node 12.1.0.2 RAC one node cluster installation with FLEX ASM option?
ora.asm
1 ONLINE ONLINE racnode1 STABLE
2 ONLINE ONLINE racnode2 Started,STABLE
3 OFFLINE OFFLINE STABLE
I wonder because of the difference in the status “Started,STABLE” and “STABLE” of the two nodes. I have an example of a three node RAC cluster (not RAC one node) where all three entries are in “Started,STABLE” status.
Best regards,
Evagelos
Really a good amount of info.
thanks for your post. I have a question here, you say that “Up to three ASM listener addresses are registred as remote listeners in each client database instance.”. So I understand that if you want a local database instance to use remote asm instance, you have to set the parameter scan_listener in database instance to something else ( may be ASMNET1LSNR_ASM ), not scan name as default in RAC 11gR2. Is this correct ?
If it is correct so how can database instance register with scan listener ?
If it is not correct so how can I configure local database instance connect to remote database instance for metadata transfer ?