Overview

• CRS mamnaged ACFS is designed for creating a shared ORACLE_HOME file systems
• You cannot store Oracle database files amd Grid Infrastructure Home files in ACFS
• Registery Managed ACFS can used to store following datatypes: BFILE, exports , ADR , .

ACFS on 11.2.0.4.3 with UEK 3.8.13 kernel – not yet supported

[grid@grac41 ~]$ /sbin/acfsutil registry
acfsutil registry: CLSU-00100: Operating System function: open64 failed with error data: 2
acfsutil registry: CLSU-00101: Operating System error message: No such file or directory
acfsutil registry: CLSU-00103: error location: OOF_1
acfsutil registry: CLSU-00104: additional error information: open64 (/dev/ofsctl)
acfsutil registry: ACFS-00502: Failed to communicate with the ACFS driver.  Verify the ACFS driver has been loaded

Check kernel version 
[root@grac41 OLD_DOCS]# uname -a
Linux grac41.example.com 3.8.13-35.1.2.el6uek.x86_64 #2 SMP Fri Jun 20 00:18:57 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[grid@grac41 ~]$ acfsroot enable
ACFS-9459: ADVM/ACFS is not supported on this OS version: '3.8.13-35.1.2.el6uek.x86_64'

[grid@grac41 ~]$  acfsroot version_check
ACFS-9459: ADVM/ACFS is not supported on this OS version: '3.8.13-35.1.2.el6uek.x86_64'
ACFS-9319: Unknown OS kernel version '3.8.13-35.1.2.el6uek.x86_64' detected.
ACFS-9320: Missing file: 'oracleadvm.ko'.
ACFS-9320: Missing file: 'oracleoks.ko'.
ACFS-9320: Missing file: 'oracleacfs.ko'.
ACFS-9317: No ADVM/ACFS distribution media detected at location: '/u01/app/11204/grid/install/usm/ADVM/ACFS is not supported on 3.8.13-35.1.2.el6uek.x86_64'

• Monitor Bug 19171929 : BLR BACKPORT OF BUG 16318126 ON TOP OF ACFS PSU 11.2.0.4.3 until patch becomes available

 

Verify a working combination : ACFS on 11.2.0.4.2 and UEK 2.6

Verify load status of ACFS driver and CRS patch level
[root@grac41 Desktop]# crsctl status resource ora.drivers.acfs -init
NAME=ora.drivers.acfs
TYPE=ora.drivers.acfs.type
TARGET=ONLINE
STATE=ONLINE on grac41
[root@grac41 Desktop]# lsmod | grep ora
oracleacfs           1980672  0
oracleadvm            243152  1
oracleoks             426774  2 oracleacfs,oracleadvm
oracleasm              53865  1
[root@grac41 Desktop]#  acfsdriverstate version
ACFS-9325:     Driver OS kernel version = 2.6.39-400.3.0.el6uek.x86_64(x86_64).
ACFS-9326:     Driver Oracle version = 140312.
[root@grac41 Desktop]# acfsdriverstate installed
ACFS-9203: true
[root@grac41 Desktop]#  acfsdriverstate installed
ACFS-9203: true
[root@grac41 Desktop]# acfsdriverstate supported
ACFS-9200: Supported
[root@grac41 Desktop]# crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
[grid@grac41 ~]$  $GRID_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure 11g                                       11.2.0.4.0
There are 1 product(s) installed in this Oracle Home.
Interim patches (3) :
Patch  18031668     : applied on Sun May 04 10:25:43 CEST 2014
Unique Patch ID:  17255543
Patch description:  "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Created on 20 Feb 2014, 05:15:58 hrs PST8PDT
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Patch  18031731     : applied on Sat May 03 17:01:53 CEST 2014
Unique Patch ID:  17253722
Patch description:  "ACFS Patch Set Update : 11.2.0.4.2 (18031731)"
Created on 17 Mar 2014, 06:35:22 hrs PST8PDT

Patch  18031740     : applied on Sat May 03 16:55:07 CEST 2014
Unique Patch ID:  17253722
Patch description:  "OCW Patch Set Update : 11.2.0.4.2 (18031740)"
Created on 19 Mar 2014, 09:06:37 hrs PST8PDT

Setup Registry Managed ACFS File System

Check DG attributes
ASMCMD> lsdg -g ACFS
Inst_ID  State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
1  MOUNTED  NORMAL  N         512   4096  1048576      2038     1764                0             882              0             N  ACFS/
2  MOUNTED  NORMAL  N         512   4096  1048576      2038     1764                0             882              0             N  ACFS/
3  MOUNTED  NORMAL  N         512   4096  1048576      2038     1764                0             882              0             N  ACFS/
ASMCMD> lsattr -G ACFS -l
Name                     Value
access_control.enabled   FALSE
access_control.umask     066
au_size                  1048576
cell.smart_scan_capable  FALSE
compatible.advm          11.2.0.0.0
compatible.asm           11.2.0.0.0
compatible.rdbms         11.2.0.0.0
disk_repair_time         3.6h
sector_size              512

Create an Oracle ASM volume and display volume details
ASMCMD> volcreate -G ACFS -s 100M acfs_vol1
ASMCMD> volinfo -a
Diskgroup Name: ACFS
Volume Name: ACFS_VOL1
Volume Device: /dev/asm/acfs_vol1-292
State: ENABLED
Size (MB): 128
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

ASMCMD> volinfo -G ACFS acfs_vol1
Diskgroup Name: ACFS
Volume Name: ACFS_VOL1
Volume Device: /dev/asm/acfs_vol1-292
State: ENABLED
Size (MB): 128
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

Automatically mount the filesystem after reboot by registering Mount points
[root@grac41 Desktop]# mkdir /u01/oradata/acfs
[root@grac43 rules.d]#  acfsutil registry -a /dev/asm/acfs_vol1-292  /u01/oradata/acfs
acfsutil registry: mount point /u01/oradata/acfs successfully added to Oracle Registry
[root@grac43 rules.d]#  acfsutil registry
Mount Object:
Device: /dev/asm/acfs_vol1-292
Mount Point: /u01/oradata/acfs
Disk Group: ACFS
Volume: ACFS_VOL1
Options: none
Nodes: all

Create file system and verify FS with acfsutil info fs
[root@grac43 rules.d]# mkfs -t acfs /dev/asm/acfs_vol1-292
mkfs.acfs: version                   = 11.2.0.4.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/acfs_vol1-292
mkfs.acfs: ACFS-01002: ACFS requires a minimum volume size of 200MB.
mkfs.acfs: ACFS-01004: /dev/asm/acfs_vol1-292 was not formatted.

Create a new 200 MByte volume 
ASMCMD> volcreate -G ACFS -s 200M acfs_vol2
[root@grac43 rules.d]# mkdir /u01/oradata/acfs2
[root@grac43 rules.d]# acfsutil registry -a /dev/asm/acfs_vol2-292  /u01/oradata/acfs2

Create file system
[root@grac41 Desktop]# mkfs -t acfs /dev/asm/acfs_vol2-292
mkfs.acfs: version                   = 11.2.0.4.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/acfs_vol2-292
mkfs.acfs: volume size               = 234881024
mkfs.acfs: Format complete.

Check File System
[root@grac41 Desktop]# /sbin/acfsutil info fs /u01/oradata/acfs2
/u01/oradata/acfs2
ACFS Version: 11.2.0.4.0
flags:        MountPoint,Available
mount time:   Wed Jul 23 15:57:17 2014
volumes:      1
total size:   234881024
total free:   114507776
primary volume: /dev/asm/acfs_vol2-292
label:
flags:                 Primary,Available,ADVM
on-disk version:       39.0
allocation unit:       4096
major, minor:          251, 149506
size:                  234881024
free:                  114507776
ADVM diskgroup         ACFS
ADVM resize increment: 33554432
ADVM redundancy:       mirror
ADVM stripe columns:   4
ADVM stripe width:     131072
number of snapshots:  0
snapshot space usage: 0
replication status: DISABLED

Related commands for AIX/Solaris
# mkfs -V acfs /dev/asm/acfs_vol2-292 (AIX)
# mkfs -F acfs /dev/asm/acfs_vol2-292 (Solaris)

Mounting/dismount ACFS  file system
# mount -t acfs /dev/asm/acfs_vol2-292  /u01/oradata/acfs2 (linux)
# mount -V acfs /dev/asm/acfs_vol2-292  /u01/oradata/acfs2 (AIX)
# mount -F acfs /dev/asm/acfs_vol2-292  /u01/oradata/acfs2 (Solaris)

Dismount a specific ACFS filesystem
#  umount   /dev/asm/acfs_vol2-292

Dismount all ACFS filesystems on the node
# umount -a -t acfs (Linux Only)
# unmountall -F acfs (Solaris, AIX)

- Before umounting an ACFs FS stop all process which still has open files with that FS
- Use lsof/fuser command to detect open files and user processes

Check and repair an Oracle ACFS file system.

[root@grac41 Desktop]# fsck -a -v -y -t acfs /dev/asm/acfs_vol2-292
fsck from util-linux-ng 2.17.2
version                   = 11.2.0.4.0
fsck.acfs: temporary directory '/usr/tmp'
fsck.acfs: current directory '/root/Desktop'
fsck.acfs: ACFS-00511: /dev/asm/acfs_vol2-292 is mounted on at least one node of the cluster.
fsck.acfs: ACFS-07656: unable to continue

Dismount ACFS FS an all nodes
[root@grac41 Desktop]# ssh grac41 umount /dev/asm/acfs_vol2-292
[root@grac41 Desktop]# ssh grac42 umount /dev/asm/acfs_vol2-292
[root@grac41 Desktop]# ssh grac43 umount /dev/asm/acfs_vol2-292

[root@grac41 Desktop]#  fsck -a -v -y -t acfs /dev/asm/acfs_vol2-292
fsck from util-linux-ng 2.17.2
version                   = 11.2.0.4.0
fsck.acfs: temporary directory '/usr/tmp'
fsck.acfs: current directory '/root/Desktop'
*****************************
********** Pass: 1 **********
*****************************
fsck.acfs: file system check starting for volume: /dev/asm/acfs_vol2-292
Oracle ASM Cluster File System (ACFS) On-Disk Structure Version: 39.0
Volume indicates the on-disk version is 39.0
Volume indicates the on-disk version is 39.0
ACFS file system created at: Wed Jul 23 15:22:38 2014
checking primary file system
fsck.acfs: Volume_Log recovery on node 1 not needed
fsck.acfs: Volume_Log recovery on node 2 not needed
fsck.acfs: Volume_Log recovery on node 3 not needed
fsck.acfs: checking directory:
for file:
ACFS Internal File: [ACFS Root Directory]
file identifier: 2 (0x2)    disk offset: 70656 (0x11400)
fsck.acfs: checking file '/.ACFS'
fsck.acfs: checking directory:
for file:
File: '/.ACFS'
file identifier: 7 (0x7)    disk offset: 73216 (0x11e00)
Parent Directory: [ACFS Root Directory]
file identifier: 2 (0x2)    disk offset: 70656 (0x11400)
fsck.acfs: checking file '/.ACFS/.fileid'
fsck.acfs: checking directory:
for file:
File: '/.ACFS/.fileid'
file identifier: 8 (0x8)    disk offset: 73728 (0x12000)
Parent Directory: /.ACFS
file identifier: 7 (0x7)    disk offset: 73216 (0x11e00)
fsck.acfs: checking file '/.ACFS/repl'
fsck.acfs: checking directory:
for file:
File: '/.ACFS/repl'
file identifier: 9 (0x9)    disk offset: 74240 (0x12200)
Parent Directory: /.ACFS
file identifier: 7 (0x7)    disk offset: 73216 (0x11e00)
fsck.acfs: checking file '/.ACFS/repl/staging'
fsck.acfs: checking directory:
for file:
File: '/.ACFS/repl/staging'
file identifier: 10 (0xa)    disk offset: 74752 (0x12400)
Parent Directory: /.ACFS/repl
file identifier: 9 (0x9)    disk offset: 74240 (0x12200)
fsck.acfs: checking file '/.ACFS/repl/ready'
fsck.acfs: checking directory:
for file:
File: '/.ACFS/repl/ready'
file identifier: 11 (0xb)    disk offset: 75264 (0x12600)
Parent Directory: /.ACFS/repl
file identifier: 9 (0x9)    disk offset: 74240 (0x12200)
fsck.acfs: checking file '/.ACFS/repl/processed'
fsck.acfs: checking directory:
for file:
File: '/.ACFS/repl/processed'
file identifier: 12 (0xc)    disk offset: 75776 (0x12800)
Parent Directory: /.ACFS/repl
file identifier: 9 (0x9)    disk offset: 74240 (0x12200)
fsck.acfs: checking file '/.ACFS/repl/logs'
fsck.acfs: checking directory:
for file:
File: '/.ACFS/repl/logs'
file identifier: 13 (0xd)    disk offset: 76288 (0x12a00)
Parent Directory: /.ACFS/repl
file identifier: 9 (0x9)    disk offset: 74240 (0x12200)
fsck.acfs: checking file '/.ACFS/snaps'
fsck.acfs: checking directory:
for file:
File: '/.ACFS/snaps'
file identifier: 14 (0xe)    disk offset: 76800 (0x12c00)
Parent Directory: /.ACFS
file identifier: 7 (0x7)    disk offset: 73216 (0x11e00)
fsck.acfs: checking file '/lost+found'
fsck.acfs: checking directory:
for file:
File: '/lost+found'
file identifier: 21 (0x15)    disk offset: 80384 (0x13a00)
Parent Directory: [ACFS Root Directory]
file identifier: 2 (0x2)    disk offset: 70656 (0x11400)
fsck.acfs: checking File_Entry_Table for unprocessed entries
fsck.acfs: checking File_Entry_Table entry: 24 (0x18) at disk offset: 81920 (0x14000)
fsck.acfs: checking File_Entry_Table entry: 25 (0x19) at disk offset: 82432 (0x14200)
fsck.acfs: checking File_Entry_Table entry: 26 (0x1a) at disk offset: 82944 (0x14400)
fsck.acfs: checking File_Entry_Table entry: 27 (0x1b) at disk offset: 83456 (0x14600)
fsck.acfs: checking File_Entry_Table entry: 28 (0x1c) at disk offset: 83968 (0x14800)
fsck.acfs: checking File_Entry_Table entry: 29 (0x1d) at disk offset: 84480 (0x14a00)
fsck.acfs: checking File_Entry_Table entry: 30 (0x1e) at disk offset: 84992 (0x14c00)
fsck.acfs: checking File_Entry_Table entry: 31 (0x1f) at disk offset: 85504 (0x14e00)
fsck.acfs: check for unprocessed File_Entry_Table entries complete
Files checked in primary file system: 100%
Checking if any files are orphaned...
Phase 1 Orphan check...
Phase 2 Orphan check...
0 orphans found
Checker completed with no errors.

Remount FS after fsck
# ssh grac41  mount -t acfs /dev/asm/acfs_vol2-292  /u01/oradata/acfs2
# ssh grac42  mount -t acfs /dev/asm/acfs_vol2-292  /u01/oradata/acfs2
# ssh grac43  mount -t acfs /dev/asm/acfs_vol2-292  /u01/oradata/acfs2

fsck commnads for AIX/Solaris
# fsck -a -v -y -V acfs /dev/asm/acfs_vol2-292 (AIX)
# fsck -a -v -y -F acfs /dev/asm/acfs_vol2-292 (Solaris)

Drop a Registered Managed ACFS File System

To unregister a file system from the Oracle Automatic Storage Management volume registry
#  acfsutil registry -d /dev/asm/acfs_vol1-292
#  acfsutil registry -d /dev/asm/acfs_vol2-292
acfsutil registry: successfully removed ACFS volume /dev/asm/acfs_vol2-292 from Oracle Registry

Remove the file system
# acfsutil rmfs /dev/asm/acfs_vol2-292
acfsutil rmfs: ACFS-00511: /dev/asm/acfs_vol2-292 is mounted on at least one node of the cluster.
acfsutil rmfs: ACFS-03014: file system not removed

Unmount first 
[root@grac41 Desktop]# ssh grac41 umount /dev/asm/acfs_vol2-292
[root@grac41 Desktop]# ssh grac42 umount /dev/asm/acfs_vol2-292
[root@grac41 Desktop]# ssh grac43 umount /dev/asm/acfs_vol2-292
# acfsutil rmfs /dev/asm/acfs_vol2-292
--> ACFS filesystem removed

Disable volumes
ASMCMD> voldisable -G ACFS ACFS_VOL2
ASMCMD> voldisable -G ACFS ACFS_VOL1

Delete  volumes
ASMCMD> voldelete -G ACFS ACFS_VOL1
ASMCMD> voldelete -G ACFS ACFS_VOL2

ACFS File System resize operation

[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 369098752 (352MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 402653184 (384MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 436207616 (416MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 469762048 (448MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 503316480 (480MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 536870912 (512MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 570425344 (544MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 603979776 (576MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: new file system size: 637534208 (608MB)
[root@grac41 Desktop]# acfsutil size +10M  /u01/oradata/acfs
acfsutil size: ACFS-03171: Insufficient contiguous free ASM Diskgroup space.  Check the ASM alert log.
--> Seems that number of resize operation for an ACFS FS has increased
In case you get a different error from above ( this error is space related ) run fsck to fix the problem

CRS managed ACFS File System

Create DG ( > 6 GByte )
--> Select this DG and select : Create ACFS for Database HOME
--> Run script acfs_script.sh to mount FS and setup the needed CW resources

How to dismount an  ACFS filesystem failing  with : device is busy error

Dismount an  ACFS filesystem fails with : device is busy error
[root@grac41 test]# mount -v | grep acfs
/dev/asm/acfs_vol1-377 on /u01/oradata/acfs type acfs (rw)

[root@grac41 test]# umount /u01/oradata/acfs
umount: /u01/oradata/acfs: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /u01/oradata/acfs: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

Using lsof and grep for fragements of out mount point ( /u01/oradata/acfs )
[root@grac41 test]# lsof | grep acfs
csh        9334    oracle  cwd       DIR         251,193025      4096                   44 /u01/oradata/acfs/oracle_user
lsof       9802      root  cwd       DIR         251,193025      4096                   42 /u01/oradata/acfs/test
grep       9803      root  cwd       DIR         251,193025      4096                   42 /u01/oradata/acfs/test
lsof       9804      root  cwd       DIR         251,193025      4096                   42 /u01/oradata/acfs/test
--> 2 processes have open directories from on our ACFS FS  /u01/oradata/acfs
    root has opened directoy /u01/oradata/acfs/test
    --> Verify an fix this 
        [root@grac41 test]# pwd
        /u01/oradata/acfs/test
        [root@grac41 test]# cd
        [root@grac41 ~]# pwd
        /root
   oracle has opened direcotoy /u01/oradata/acfs/oracle_user
   --> Here we just kill the process
       [root@grac41 ~]#  kill -9 9334

Dismount ACFS filesystem again
[root@grac41 ~]# umount  /u01/oradata/acfs 
  --> umount works now

Shell scripts to create/recreate an ACFS filesystem/Volumes

Run script in the following order :
 drop_vol.sh      - drop old volume group
 cr_acfs_dg.sql   - recrate ACFS DG 
 cr_vol.sh        - create new volume group and mount ACFS file system clusterwide
 info_vol.sh      - display DG and VG status

Drop volume 
[grid@grac41 ACFS]$ cat drop_vol.sh
#!/bin/bash
MP=/u01/oradata/acfs
VOLNAME=acfs_vol1
DG=ACFS
VOL=`asmcmd volinfo -G ACFS $VOLNAME | grep 'Volume Device' | awk ' // { print $3 } ' `

echo Mount Point: $MP - Volume Name: $VOLNAME - Volume: $VOL
asmcmd lsattr -G $DG -l
echo "*** Dismount Volume and check mount status *** "
sudo ssh grac41 umount  $MP
sudo ssh grac42 umount  $MP 
sudo ssh grac43 umount  $MP
sudo ssh grac41 /bin/hostname
sudo ssh grac41 mount | grep  $MP
sudo ssh grac42 /bin/hostname
sudo ssh grac42 mount | grep  $MP
sudo ssh grac43 /bin/hostname
sudo ssh grac43 mount | grep  $MP

echo "*** Disable and Delete Volumne ***"
asmcmd voldisable  -G $DG $VOLNAME
asmcmd voldelete  -G  $DG $VOLNAME

Create DG 
[grid@grac41 ACFS]$ cat  cr_acfs_dg.sql
set echo on 
! ssh grac41 asmcmd umount ACFS
! ssh grac42 asmcmd umount ACFS
! ssh grac43 asmcmd umount ACFS
! asmcmd lsdg --discovery -g | egrep 'State|ACFS'
drop diskgroup ACFS force including contents;
! dd if=/dev/zero of=/dev/asm_test_1G_disk1 bs=8k count=1000
! dd if=/dev/zero of=/dev/asm_test_1G_disk2 bs=8k count=1000
CREATE DISKGROUP ACFS NORMAL REDUNDANCY
  FAILGROUP FG1  DISK
    '/dev/asm_test_1G_disk1' NAME diska1
  FAILGROUP FG2  DISK
    '/dev/asm_test_1G_disk2' NAME diskb1
  ATTRIBUTE 'au_size'='4M',
    'compatible.asm' = '11.2', 
    'compatible.rdbms' = '11.2',
    'compatible.advm' = '11.2';

! asmcmd lsdg -g ACFS
! ssh grac42 asmcmd mount ACFS
! ssh grac43 asmcmd mount ACFS
! asmcmd lsdg -g ACFS

Create ACFS volume 
[grid@grac41 ACFS]$ cat cr_vol.sh 
#/bin/bash
MP=/u01/oradata/acfs
VOLNAME=acfs_vol1
DG=ACFS

echo "*** Check ASM dG ***" 
asmcmd lsdg -g $DG 
asmcmd lsattr -G $DG -l

echo "*** Create and Enable ACFS volume ***" 
asmcmd volcreate -G $DG  -s 200M  $VOLNAME
VOL=`asmcmd volinfo -G ACFS $VOLNAME | grep 'Volume Device' | awk ' // { print $3 } ' ` 
echo Mount Point: $MP - Volume Name: $VOLNAME - Volume: $VOL

asmcmd volinfo -G  $DG   $VOLNAME
asmcmd volinfo -G  $DG $VOLNAME  | grep 'Volume Device'

echo "*** Re-register ACFS filesystem and create ACFS filesystem ***" 
sudo acfsutil registry -d   $MP
sudo acfsutil registry -a  $VOL $MP
sudo acfsutil registry 
sudo mkfs  -t acfs  $VOL 

echo " ***  Mount ACFS FS  and check mount status ACFS file system status ***"
sudo ssh grac41 mount  -t acfs $VOL $MP 
sudo ssh grac42 mount  -t acfs $VOL $MP 
sudo ssh grac43 mount  -t acfs $VOL $MP 

sudo ssh grac41 /bin/hostname
sudo ssh grac41 mount | grep  $VOL 
sudo ssh grac42 /bin/hostname
sudo ssh grac42 mount | grep  $VOL 
sudo ssh grac43 /bin/hostname
sudo ssh grac43 mount | grep  $VOL 

sudo /sbin/acfsutil info fs $MP

Display volume and diskgroup status 
[grid@grac41 ACFS]$ cat info_vol.sh
#/bin/bash
MP=/u01/oradata/acfs
VOLNAME=acfs_vol1
DG=ACFS

VOL=`asmcmd volinfo -G ACFS $VOLNAME | grep 'Volume Device' | awk ' // { print $3 } ' ` 
echo Mount Point: $MP - Volume Name: $VOLNAME - Volume: $VOL

echo "*** Check ASM dG ***" 
asmcmd lsdg -g $DG 
asmcmd lsattr -G $DG -l

echo "*** Check Volume   ***" 
asmcmd volinfo -G  $DG   $VOLNAME
asmcmd volinfo -G  $DG $VOLNAME  | grep 'Volume Device'

echo "*** Is mount point already registered  ***" 
sudo acfsutil registry -l   $MP

echo "***  mount status  ***" 
sudo ssh grac41 /bin/hostname
sudo ssh grac41 mount | grep  $VOL 
sudo ssh grac42 /bin/hostname
sudo ssh grac42 mount | grep  $VOL 
sudo ssh grac43 /bin/hostname
sudo ssh grac43 mount | grep  $VOL 

echo "***  check filesystem  ***" 
sudo /sbin/acfsutil info fs $MP

Reference

• ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)
• ASM/ACFS 11.2: How To Create A General Purpose ACFS Filesystem (Registry Managed) On RAC Through the ASMCA Assistance. (Doc ID 1524195.1)
• How To Create A New ACFS Volume & Filesystem And Set The ACFS Filesystem Ownership To A Non-Grid/Oracle OS User (on Oracle Restart/SIHA/Standalone)? (Doc ID 1322383.1)
• ASM/ACFS 11.2: How To Create “An ACFS CRS Managed Database Home File System For An Oracle Home Database (RDBMS)” On RAC Through the ASMCA Assistance. (Doc ID 1484221.1)
• ASM/ACFS 11.2: Example/Demo About How To Convert A General Purpose ACFS Filesystem (Registry Managed) To CRS Managed On Release 11.2.0.X. (Doc ID 1523204.1)
• How To Setup 11.2.0.X ACFS On Windows (64-bit) On RAC. (Doc ID 1350188.1)
• Bug 16318126 : UEK3 SUPPORT FOR ACFS
• Bug 19171929 : BLR BACKPORT OF BUG 16318126 ON TOP OF ACFS PSU 11.2.0.4.3 (BLR #3880917)
• Bug 18522509 – 11.2.0.4.3 (Jul 2014) Database Patch Set Update (PSU) (Doc ID 18522509.8)

 

Check OS and CRS version

$GRID_HOME/OPatch/opatch lsinventory
Oracle Home       : /u01/app/121/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/121/grid/oraInst.loc
OPatch version    : 12.1.0.1.0
OUI version       : 12.1.0.1.0
Log file location : /u01/app/121/grid/cfgtoollogs/opatch/opatch2014-06-25_14-50-13PM_1.log

Lsinventory Output file location : /u01/app/121/grid/cfgtoollogs/opatch/lsinv/lsinventory2014-06-25_14-50-13PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1): 
Oracle Grid Infrastructure 12c                                       12.1.0.1.0
Patch level status of Cluster nodes :
 Patching Level              Nodes
 --------------              -----
 0                           gract3,gract2,gract1
# uname -a
Linux gract1.example.com 2.6.39-400.109.6.el6uek.x86_64 #1 SMP Wed Aug 28 09:56:40 PDT 2013 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/oracle-release
Oracle Linux Server release 6.4

Check whether related ACFS modules are already loaded
# lsmod | grep ora
oracleacfs           3053229  0 
oracleadvm            320180  2 
oracleoks             417171  2 oracleacfs,oracleadvm
oracleasm              53865  1 

You may check whether your kernel version is supported by reading
ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)

Create a new ACFS diskgroup

Use asmca and check driver status 
--> ADVM Driver status sould be : Loaded 

Create ASM diskgroup with HIGH redundany
Make 3 ASM disks available at OS using UDEV rules - Reboot 
All custer nodes should show the same results
# ls -l  /dev/acfs*
brw-rw----. 1 grid asmadmin 8,  81 Jun 26 08:12 /dev/acfsdisk1_1G
brw-rw----. 1 grid asmadmin 8,  97 Jun 26 08:12 /dev/acfsdisk2_1G
brw-rw----. 1 grid asmadmin 8, 113 Jun 26 08:12 /dev/acfsdisk3_1G

Invoke acma and create a diskgroup with following paramters 
Change your disk discorvery path to : /dev/asm*,/dev/acfs*
Redundancy : HIGH
Disk PATH:
/dev/acfsdisk1_1G  CANDIDATE
/dev/acfsdisk2_1G  CANDIDATE
/dev/acfsdisk3_1G  CANDIDATE

ASM       Compatibility : 12.1.0.0.0
Show Advanced Options and set 
   ASM       Compatibility : 12.1.0.0.0
   Database  Compatibility : 12.1.0.0.0
   ADVM      Compatibility : 12.1.0.0.0

Verify diskgroup status ( should be mounted on all nodes )

[grid@gract1 ~]$  asmcmd lsdg -g ACFS_DG1
Inst_ID  State    Type  Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      1  MOUNTED  HIGH  N         512   4096  1048576      3057     2643                0             881              0             N  ACFS_DG1/
      3  MOUNTED  HIGH  N         512   4096  1048576      3057     2643                0             881              0             N  ACFS_DG1/
      2  MOUNTED  HIGH  N         512   4096  1048576      3057     2643                0             881              0             N  ACFS_DG1/

[grid@gract1 ~]$  asmcmd lsdsk -g
Inst_ID  Path
      1  /dev/acfsdisk1_1G
      3  /dev/acfsdisk1_1G
      2  /dev/acfsdisk1_1G
      1  /dev/acfsdisk2_1G
      3  /dev/acfsdisk2_1G
      2  /dev/acfsdisk2_1G
      1  /dev/acfsdisk3_1G
      3  /dev/acfsdisk3_1G
      2  /dev/acfsdisk3_1G

Create and display the volume status 
grid@gract1 ~]$  asmcmd  volcreate -G ACFS_DG1 -s 800m acfs_vol1

[grid@gract1 ~]$ asmcmd volinfo -G ACFS_DG1 acfs_vol1
Diskgroup Name: ACFS_DG1
     Volume Name: ACFS_VOL1
     Volume Device: /dev/asm/acfs_vol1-443
     State: ENABLED
     Size (MB): 800
     Resize Unit (MB): 32
     Redundancy: HIGH
     Stripe Columns: 4
     Stripe Width (K): 128
     Usage: 
     Mountpath: 

Create an ACFS filesystem
#  mkfs -t acfs /dev/asm/acfs_vol1-443
mkfs.acfs: version                   = 12.1.0.1.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/acfs_vol1-443
mkfs.acfs: volume size               = 838860800
mkfs.acfs: Format complete.

Register a mount point

[root@gract1 rules.d]#  mkdir -p /u01/acfs/acfs-vol1 
[root@gract1 rules.d]# /sbin/acfsutil registry -a  /dev/asm/acfs_vol1-443 /u01/acfs/acfs-vol1
acfsutil registry: mount point /u01/acfs/acfs-vol1 successfully added to Oracle Registry

Mount and dismount an ACFS filesystem

Locate any ACFS mounted filesystem 
[root@gract1 Desktop]#  mount -t acfs
/dev/asm/acfs_vol1-443 on /u01/acfs/acfs-vol1 type acfs (rw)

Dismount an ACFS filesystem 
[root@gract1 Desktop]# umount /u01/acfs/acfs-vol1 
[root@gract1 Desktop]# mount -t acfs

Find ACFS mountpoints and mount ACFS fs
[root@gract1 Desktop]#   /sbin/acfsutil registry -l
Device : /dev/asm/acfs_vol1-443 : Mount Point : /u01/acfs/acfs-vol1 : Options : none : Nodes : all : Disk Group : ACFS_DG1 : Volume : ACFS_VOL1

[root@gract1 Desktop]# mount -t acfs /dev/asm/acfs_vol1-443  /u01/acfs/acfs-vol1
[root@gract1 Desktop]# mount -t acfs
/dev/asm/acfs_vol1-443 on /u01/acfs/acfs-vol1 type acfs (rw)

Use lsof to investigate umount problem : device busy

[root@gract1 Desktop]#  umount /u01/acfs/acfs-vol1 
umount: /u01/acfs/acfs-vol1: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /u01/acfs/acfs-vol1: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
[root@gract1 Desktop]# lsof | grep  /u01/acfs/acfs-vol1
vi        11553      root  cwd       DIR         251,226817      4096     2  /u01/acfs/acfs-vol1
vi        11553      root    3u      REG         251,226817     12222     44 /u01/acfs/acfs-vol1/.test.swp
bash      29897      root  cwd       DIR         251,226817      4096      2 /u01/acfs/acfs-vol1
--> root run bash shell and this shell has open directory /u01/acfs/acfs-vol1 
    root also runs vi from the same direactory and vi has opene a .swp file 

After stopping or killing these processes and the dismount should go through
[root@gract1 Desktop]# lsof | grep  /u01/acfs/acfs-vol1
[root@gract1 Desktop]# umount /u01/acfs/acfs-vol1 

Verify  ACFS setup

# mount
/dev/asm/acfs_vol1-443 on /u01/acfs/acfs-vol1 type acfs (rw)
Commands to verify your setup

[root@gract1 rules.d]# lsmod | grep ora
oracleacfs           3053229  2
oracleadvm            320180  8
oracleoks             417171  2 oracleacfs,oracleadvm
oracleasm              53865  1

$ asmcmd volinfo -G ACFS_DG1 acfs_vol1
Diskgroup Name: ACFS_DG1
Volume Name: ACFS_VOL1
Volume Device: /dev/asm/acfs_vol1-443
State: ENABLED
Size (MB): 800
Resize Unit (MB): 32
Redundancy: HIGH
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath: /u01/acfs/acfs-vol1

[root@gract1 rules.d]#  /sbin/acfsutil registry -l
Device : /dev/asm/acfs_vol1-443 : Mount Point : /u01/acfs/acfs-vol1 : Options : none : Nodes : all : Disk Group : ACFS_DG1 : Volume : ACFS_VOL1
help:  help

[root@gract1 rules.d]# asmcmd lsdg -g
Inst_ID  State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
1  MOUNTED  HIGH    N         512   4096  1048576      3057      159                0              53              0             N  ACFS_DG1/
3  MOUNTED  HIGH    N         512   4096  1048576      3057      159                0              53              0             N  ACFS_DG1/
2  MOUNTED  HIGH    N         512   4096  1048576      3057      159                0              53              0             N  ACFS_DG1/

[root@gract1 rules.d]# df -k
Filesystem                1K-blocks      Used Available Use% Mounted on
/dev/asm/acfs_vol1-443       819200    118852    700348  15% /u01/acfs/acfs-vol1

Reference

• http://oracleinaction.com/ora-15477-communicate-volume-driver/
• ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1)

Used Software

• Clusterware: 11.2.0.3.4
• OEL 6.3
• VirtualBox 4.2.14

ACFS provides snapshotting capability for the respective filesystem. This snapshot uses the First Copy-on-Write (FCOW) methodology to enable a consistent, version-based, online view of the source filesystem.Snapshots are immutable views of the source file system as it appeared at a specific point in time.Snapshots are initially a sparse filesystem, as the source filesystem files change, the before-image extent ofthat file is copied into the snapshot directory. The before-image granularity is an ACFS extent, thus if any byte in an extent is modified, the extent is COW’ed and any subsequent changes in that extent requires no action for the snap. Note that ACFS snapshots cannot be used on filesystems that house RMAN backups,archived logs, and Datapump dumpsets.

When snapshots are created they are automatically available under the snapped filesystem as a hidden sub-directory inside the source filesystem called .ACFS.  Thus no separate mount operation is needed and no separate file store needs to be maintained for the snapshots. The following example shows how to create  ACFS snapshots.

 

Check whether we already have created any snapshots:
$ /sbin/acfsutil info fs
/u01/app/oracle/acfsmount/acfs_vol1
    ACFS Version: 11.2.0.3.0
    flags:        MountPoint,Available
    mount time:   Mon Aug  5 15:12:14 2013
    volumes:      1
    total size:   838860800
    total free:   763121664
    primary volume: /dev/asm/acfs_vol1-140
        label:                 
        flags:                 Primary,Available,ADVM
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          251, 71681
        size:                  838860800
        free:                  763121664
        ADVM diskgroup         ACFS
        ADVM resize increment: 33554432
        ADVM redundancy:       mirror
        ADVM stripe columns:   4
        ADVM stripe width:     131072
    number of snapshots:  0
    snapshot space usage: 0
    replication status: DISABLED

As we don't have created any snapshots yet - let's create one and verify the status
$ /sbin/acfsutil snap create snap_001  /u01/app/oracle/acfsmount/acfs_vol1 
acfsutil snap create: Snapshot operation is complete.
$  /sbin/acfsutil info fs
/u01/app/oracle/acfsmount/acfs_vol1
    ACFS Version: 11.2.0.3.0
    flags:        MountPoint,Available
    mount time:   Mon Aug  5 15:12:14 2013
    volumes:      1
    total size:   838860800
    total free:   696012800
    primary volume: /dev/asm/acfs_vol1-140
        label:                 
        flags:                 Primary,Available,ADVM
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          251, 71681
        size:                  838860800
        free:                  696012800
        ADVM diskgroup         ACFS
        ADVM resize increment: 33554432
        ADVM redundancy:       mirror
        ADVM stripe columns:   4
        ADVM stripe width:     131072
    number of snapshots:  1
    snapshot space usage: 49152
    replication status: DISABLED

Copy 2 files to our ACFS file system and create a 2.nd snapshot
$ ls /u01/app/oracle/acfsmount/acfs_vol1
INFO  INFO2  
$ /sbin/acfsutil snap create snap_002 /u01/app/oracle/acfsmount/acfs_vol1
acfsutil snap create: Snapshot operation is complete.
Checkout .ACFS directory and verify that snap_002 directory displays INFO, INFO2 files 
# find  /u01/app/oracle/acfsmount/acfs_vol1/.ACFS
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/.fileid
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/repl
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/repl/staging
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/repl/ready
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/repl/processed
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/repl/logs
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps/snap_001
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps/snap_001/lost+found
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps/snap_002
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps/snap_002/lost+found
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps/snap_002/INFO
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps/snap_002/INFO2

Remove INFO2 file from current ACFS fs
$ rm /u01/app/oracle/acfsmount/acfs_vol1/INFO2
$ ls  /u01/app/oracle/acfsmount/acfs_vol1
INFO

Restore file from snap_002 
# find  /u01/app/oracle/acfsmount/acfs_vol1/.ACFS | grep INFO2
/u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps/snap_002/INFO2
# cp /u01/app/oracle/acfsmount/acfs_vol1/.ACFS/snaps/snap_002/INFO2  /u01/app/oracle/acfsmount/acfs_vol1
#  ls  /u01/app/oracle/acfsmount/acfs_vol1
INFO  INFO2 

 

Reference:

     http://www.oracle.com/technetwork/products/cloud-storage/acfs-technical-overview-514457.pdf

Used Software

• GRID: 11.2.0.3.4
• OEL 6.3
• VirtualBox 4.2.14

 

Create ASM diskgroup

Create the new ASM disks:
D:\VM> VBoxManage createhd --filename C:\VM\GRACE2\ASM\asm_1G_ACFS1.vdi --size 1024 --format VDI --variant Fixed
D:\VM> VBoxManage createhd --filename C:\VM\GRACE2\ASM\asm_1G_ACFS2.vdi --size 1024 --format VDI --variant Fixed

Shutdown grac1 and attach disk to VM: grac1 
D:\VM>VBoxManage storageattach grac1 --storagectl "SATA" --port 9  --device 0 --type hdd --medium C:\VM\GRACE2\ASM\asm_1G_ACFS1.vdi 
D:\VM>VBoxManage storageattach grac1 --storagectl "SATA" --port 10  --device 0 --type hdd --medium  C:\VM\GRACE2\ASM\asm_1G_ACFS2.vdi 

Change disk type to sharable disks
D:\VM>VBoxManage modifyhd C:\VM\GRACE2\ASM\asm_1G_ACFS1.vdi  --type shareable
D:\VM>VBoxManage modifyhd C:\VM\GRACE2\ASM\asm_1G_ACFS2.vdi  --type shareable
After reboot check /dev ( we should have now 2 newly create disk devices )
# ls -l /dev/sdh /dev/sdi
brw-rw---- 1 root disk 8, 112 Aug  4 13:52 /dev/sdh
brw-rw---- 1 root disk 8, 128 Aug  4 13:49 /dev/sdi


Create new partitions ( sample for /dev/sdh ) 
# fdisk  /dev/sdh
Command (m for help): p
Disk /dev/sdh: 1073 MB, 1073741824 bytes 255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130): 
Using default value 130
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

Create ASM disks and check their status:
# /etc/init.d/oracleasm createdisk acfs_data1 /dev/sdh
Marking disk "acfs_data1" as an ASM disk:                  [FAILED]
--> Reboot VM 
# /etc/init.d/oracleasm createdisk acfs_data /dev/sdh1
Marking disk "acfs_data" as an ASM disk:                   [  OK  ]
[root@grac1 Desktop]# /etc/init.d/oracleasm createdisk acfs_data2 /dev/sdi1
Marking disk "acfs_data2" as an ASM disk:                  [  OK  ]
# /etc/init.d/oracleasm listdisks
ACFS_DATA
ACFS_DATA2

Create ASM diskgroup using asmca
--> ADVM Compatibility is greyed and not selectable
Check OS version and kernel release
# uname -r
2.6.39-300.17.2.el6uek.x86_64
# cat /etc/oracle-release 
Oracle Linux Server release 6.3
# lsmod | grep ora
oracleasm              53352  1
See: Bug 12983005  Linux: ADVM/ACFS is not supported on OS version '2.6.39-100.7.1.el6uek.x86_64'
--> Missing driver for oracleacfs, oracleadvm , oracleoks
Fix : Install clusterware patch 11.2.0.3.3 or higher
For a detailed 11.2.0.3.4 patch install please read the following link.

After patch install we should see the following driver loaded on both nodes: 
# lsmod | grep ora
oracleacfs           1844281  0 
oracleadvm            231722  0 
oracleoks             329652  2 
oracleasm              53352  1

Use asmca to create ASM diskgroup by selecting disks 
  asm_1G_ACFS1.vdi
  asm_1G_ACFS2.vdi
Use advanced configuration to set
  ASM       Compatibility : 11.2.0.0.0
  ADVM      Compatibility : 11.2.0.0.0

 

Create  new ACFS volume in out ACFS diskgroup

Trying to use 1G space for our volume  from our ACFS diskgroup
ASMCMD> volcreate -G ACFS -s 1G ACFS_VOL1
ORA-15032: not all alterations performed
ORA-15041: diskgroup "ACFS" space exhausted (DBD ERROR: OCIStmtExecute)
Create a first disk volume using 900 MBbyte from our 1G ACFS diskgroup
ASMCMD> volcreate -G ACFS -s 900M ACFS_VOL1

Check our ACFS disk volume
$ asmcmd volinfo -a
Diskgroup Name: ACFS
     Volume Name: ACFS_VOL1
     Volume Device: /dev/asm/acfs_vol1-140
     State: ENABLED
     Size (MB): 928
     Resize Unit (MB): 32
     Redundancy: MIRROR
     Stripe Columns: 4
     Stripe Width (K): 128
     Usage: 
     Mountpath: 

Check the related linux device
$ ls -l /dev/asm
total 0
brwxrwx--- 1 root asmadmin 251, 71681 Aug  4 19:53 acfs_vol1-140
Create ACFS file system
# mkfs -t acfs /dev/asm/acfs_vol1-140
mkfs.acfs: version                   = 11.2.0.3.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/acfs_vol1-140
mkfs.acfs: volume size               = 973078528
mkfs.acfs: Format complete.

On grac1 create mount points an mount the filesystem
# mkdir -p /u01/app/oracle/acfsmount/acfs_vol1
# mount -t acfs /dev/asm/acfs_vol1-140 /u01/app/oracle/acfsmount/acfs_vol1
# df /dev/asm/acfs_vol1-140
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/asm/acfs_vol1-140
                        950272     39192    911080   5% /u01/app/oracle/acfsmount/acfs_vol1

On grac2:
# mkdir -p /u01/app/oracle/acfsmount/acfs_vol1
#  mount -t acfs /dev/asm/acfs_vol1-140 /u01/app/oracle/acfsmount/acfs_vol1
mount.acfs: CLSU-00100: Operating System function: open64 failed with error data: 2
mount.acfs: CLSU-00101: Operating System error message: No such file or directory
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/acfs_vol1-140)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/acfs_vol1-140. Verify the volume exists.

Checking mount status with asmca -> diskgroup ACFS only mount by grac1 !
Trying to mount by right clicking the entry in asmca -> get error unsuffiecent number of disks 
Cecking ASM disks status 
# /etc/init.d/oracleasm  listdisks
DATA1
DATA2
DATA3
OCR1
OCR2
OCR3
--> ACFS disks missing on our VBox grac2 image
Shutdown grac2 and attach disks to grac2 Vbox image
D:\VM> VBoxManage storageattach grac2 --storagectl "SATA" --port 9  --device 0 --type hdd --medium C:\VM\GRACE2\ASM\asm_1G_ACFS1.vdi 
D:\VM> VBoxManage storageattach grac2 --storagectl "SATA" --port 10  --device 0 --type hdd --medium  C:\VM\GRACE2\ASM\asm_1G_ACFS2.vdi 
Reboot and check ACFS disks
# /etc/init.d/oracleasm  listdisks
ACFS_DATA
ACFS_DATA2
DATA1
...
# mount -t acfs /dev/asm/acfs_vol1-140 /u01/app/oracle/acfsmount/acfs_vol1
mount.acfs: CLSU-00100: Operating System function: open64 failed with error data: 2
mount.acfs: CLSU-00101: Operating System error message: No such file or directory
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/ofsctl)
mount.acfs: ACFS-00502: Failed to communicate with the ACFS driver.  Verify the ACFS driver has been loaded.
Checking ACFS driver 
#  lsmod | grep ora
oracleasm              53352  1 
Manually load acfs kernel driver and check again
$ $GRID_HOME/bin/acfsload start -s
[root@grac2 Desktop]#  lsmod | grep ora
oracleacfs           1844281  0 
oracleadvm            231722  0 
oracleoks             329652  2 oracleacfs,oracleadvm
oracleasm              53352  1 
# mount -t acfs /dev/asm/acfs_vol1-140 /u01/app/oracle/acfsmount/acfs_vol1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/acfs_vol1-140)
mount.acfs: ACFS-02 017: Failed to open volume /dev/asm/acfs_vol1-140. Verify the volume exists.

Check Linux device volume info and diskgroup on grac2:
# ls /dev/asm/acfs_vol1-140
ls: cannot access /dev/asm/acfs_vol1-140: No such file or directory
$ asmcmd volinfo -a
no volumes found
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576     15342    10125             5114            2505              0             N  DATA/
MOUNTED  NORMAL  N         512   4096  1048576      6141     5217             2047            1585              0             Y  OCR/
Diskgroup ACFS not mounted - check clusterware alert.log 
ORA-15041: diskgroup "ACFS" space exhausted
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/grac2/agent/crsd/oraagent_grid/oraagent_grid.log".
CRS-2674: Start of 'ora.ACFS.dg' on 'grac2' failed
/u01/app/11203/grid/log/grac2/agent/crsd/oraagent_grid/oraagent_grid.log
2013-08-04 20:56:01.107: [ora.ACFS.dg][536860416] {1:24621:491} [start] ORA-15032: not all alterations performed
ORA-15202: cannot create additional ASM internal change segment
ORA-15041: diskgroup "ACFS" space exhausted

Check ASM logfile  
ERROR: diskgroup ACFS was not mounted
ORA-15032: not all alterations performed
ORA-15202: cannot create additional ASM internal change segment
ORA-15041: diskgroup "ACFS" space exhausted
ERROR: ALTER DISKGROUP ACFS MOUNT  /* asm agent *//* {1:24621:491} */

Use amdu to check whether this diskgroup is already mounted to a different disk group
$ amdu -diskstring '/dev/oracleasm/disks/ACFS*' -dump 'ACFS'
amdu_2013_08_04_21_10_37/
AMDU-00204: Disk N0001 is in currently mounted diskgroup ACFS
AMDU-00201: Disk N0001: '/dev/oracleasm/disks/ACFS_DATA'
AMDU-00204: Disk N0002 is in currently mounted diskgroup ACFS
AMDU-00201: Disk N0002: '/dev/oracleasm/disks/ACFS_DATA2'
--> looks ok  - but could not see the diskgroup ACFS with asmcmd lsdg
Fix : Reboot that node 

After reboot diskgroup is available but mount still fails with  ACFS-00502
$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576      2038       36                0              18              0             N  ACFS/
# mount -t acfs /dev/asm/acfs_vol1-140 /u01/app/oracle/acfsmount/acfs_vol1
mount.acfs: ACFS-00502: Failed to communicate with the ACFS driver.  Verify the ACFS driver has been loaded.
Manually mount ACFS file system:
Recreate volumegroup with smaller size ( seems we need about 100 MByte for each node )
# umount /dev/asm/acfs_vol1-140 
$ asmcmd voldelete -G ACFS ACFS_VOL1
$ asmcmd volcreate -G ACFS -s 800M ACFS_VOL1
$ asmcmd volinfo -a
Diskgroup Name: ACFS
     Volume Name: ACFS_VOL1
     Volume Device: /dev/asm/acfs_vol1-140
     State: ENABLED
     Size (MB): 800
     Resize Unit (MB): 32
     Redundancy: MIRROR
     Stripe Columns: 4
     Stripe Width (K): 128
     Usage: 
     Mountpath: 
#  mkfs -t acfs /dev/asm/acfs_vol1-140
On grac1 mount ACFS filesystem:
# mount -t acfs /dev/asm/acfs_vol1-140 /u01/app/oracle/acfsmount/acfs_vol1
On grac1 run asmca -> ACFS filesystem mount on form grac1 -> press right mouse button -> select Mount on all Notes
Now volumen should be mounted on both Nodes grac1,grac2
On grac2 check ASM logfile and verify successfull mount:
SUCCESS: diskgroup ACFS was mounted
SUCCESS: ALTER DISKGROUP ACFS MOUNT  /* asm agent *//* {1:12727:327} */
Mon Aug 05 08:53:28 2013
NOTE: diskgroup resource ora.ACFS.dg is updated
$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576      2038      244                0             122              0             N  ACFS/
# lsmod | grep ora
oracleacfs           1844281  0 
oracleadvm            231722  1 
oracleoks             329652  2 oracleacfs,oracleadvm
oracleasm              53352  1
Enable volume on grac2 and verify that ACFS filesytem can be mounted
$ asmcmd volinfo -a
Diskgroup Name: ACFS
     Volume Name: ACFS_VOL1
     Volume Device: /dev/asm/acfs_vol1-140
     State: DISABLED
     Size (MB): 800
     Resize Unit (MB): 32
     Redundancy: MIRROR
     Stripe Columns: 4
     Stripe Width (K): 128
     Usage: ACFS
     Mountpath: /u01/app/oracle/acfsm/dev/asm/acfs_vol1-140ount/acfs_vol1 
$ ls -l /dev/asm/acfs_vol1-140
  ls: cannot access /dev/asm/acfs_vol1-140: No such file or directory
$ asmcmd volenable -G  ACFS ACFS_VOL1
$ ls -l /dev/asm/acfs_vol1-140
  brwxrwx--- 1 root asmadmin 251, 71681 Aug  5 09:04 
# mount -t acfs /dev/asm/acfs_vol1-140 /u01/app/oracle/acfsmount/acfs_vol1
# df /dev/asm/acfs_vol1-140
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/asm/acfs_vol1-140
                        819200     73964    745236  10% /u01/app/oracle/acfsmount/acfs_vol1

Register mount volume and mount point
# acfsutil registry -a /dev/asm/acfs_vol1-140 /u01/app/oracle/acfsmount/acfs_vol1
acfsutil registry: mount point /u01/app/oracle/acfsmount/acfs_vol1 successfully added to Oracle Registry
#  acfsutil registry -l
Device : /dev/asm/acfs_vol1-140 : Mount Point : /u01/app/oracle/acfsmount/acfs_vol1 : Options : none : Nodes : all : Disk Group : ACFS : Volume : ACFS_VOL1
Reboot grac2 and check wheter ACFS gets mounted automatically 

Automatically load ACFS driver after reboot
--> /etc/init.d/acfsload
#!/bin/sh
# chkconfig: 2345 30 21
# description: Load Oracle ACFS drivers at system boot
/u01/app/11203/grid/bin/acfsload start -s

# chmod u+x /etc/init.d/acfsload
# chkconfig --add acfsload
--> reboot and check ACFS driver state 
$ acfsdriverstate version
ACFS-9325:     Driver OS kernel version = 2.6.39-300.21.1.el6uek.x86_64(x86_64).
ACFS-9326:     Driver Oracle version = RELEASE.
$ acfsdriverstate loaded
ACFS-9203: true

To make the ACFS file system automatically mounted on reboot run : 
# /u01/app/11203/grid/bin/acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9118: oracleacfs.ko driver in use - cannot unload.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9118: oracleacfs.ko driver in use - cannot unload.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.
# /u01/app/11203/grid/bin/acfsroot enable
ACFS-9376: Adding ADVM/ACFS drivers resource succeeded.
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'grac1'
CRS-2676: Start of 'ora.drivers.acfs' on 'grac1' succeeded
ACFS-9380: Starting ADVM/ACFS drivers resource succeeded.
ACFS-9368: Adding ACFS registry resource succeeded.
CRS-2672: Attempting to start 'ora.registry.acfs' on 'grac2'
CRS-2672: Attempting to start 'ora.registry.acfs' on 'grac1'
CRS-2676: Start of 'ora.registry.acfs' on 'grac1' succeeded
CRS-2676: Start of 'ora.registry.acfs' on 'grac2' succeeded
ACFS-9372: Starting ACFS registry resource succeeded.

Verify that these resources are ONLINE now:
$ my_crs_stat | grep -i acfs
ora.ACFS.dg                    ONLINE     ONLINE          grac1         
ora.ACFS.dg                    ONLINE     ONLINE          grac2         
ora.registry.acfs              ONLINE     ONLINE          grac1         
ora.registry.acfs              ONLINE     ONLINE          grac2     

Reference

•  “ora.drivers.acfs” Resource Was Not Configured Therefore RAC ACFS Filesystem Is Not Mounting During The Reboot. (Doc ID 1486208.1)
• Bug 14503558 : ACFS FILESYSTEMS ARE NOT BEING MOUNTED AFTER REBOOT THE RAC NODES ON LINUX
• .Bug 12983005  Linux: ADVM/ACFS is not supported on OS version ‘2.6.39-100.7.1.el6uek.x86_64’

 

Troubleshooting tips

• Check ACFS driver : # lsmod | grep ora
• Check readiness of your ASM disk:  # dd if=/dev/oracleasm/disks/ACFS_DATA of=/dev/nulls bs=1M
• Check your ASM disks:  # /etc/init.d/oracleasm listdisks
• Check that your diskgroup is availabe on all nodes: $ asmcmd lsdg
• Check that your volumes are ready :  $ asmcmd volinfo -a
• Check your registry settings : #  acfsutil registry -l
• Check whether ASM disks are mounted by a different group: $ amdu -diskstring ‘/dev/oracleasm/disks/ACFS*’ -dump ‘ACFS’
• Check that resource ora.registry.acfs is ONLINE: $ my_crs_stat | grep -i acfs