Oracle/Red Hat/NetApp

Mon Aug 20 16:52:41 UTC 2007

We have several Oracle instances 10.2 (some RAC, some not, but all are having the same issue) all running on Red Hat 4.5.  The binaries and data files are all on the NetApp and mounted via NFS.

Sporadically when we shut the database down, we are unable to restart it.  

/u1/app/oracle/admin/prodrac/bdump/prodrac1_dbw0_10023.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

ORACLE_HOME = /u1/app/oracle/product/10.2.0

System name:   Linux

Node name:       orarac1

Release:               2.6.9-55.0.2.ELsmp

Version:               #1 SMP Tue Jun 12 17:58:20 EDT 2007

Machine:             x86_64

Instance name: prodrac1

Redo thread mounted by this instance: 1

Oracle process number: 8

Unix process pid: 10023, image: oracle at orarac1 (DBW0)

*** SERVICE NAME:() 2007-08-18 02:05:27.878

*** SESSION ID:(1194.1) 2007-08-18 02:05:27.878

ORA-01157: cannot identify/lock data file 1 - see DBWR trace file

ORA-01110: data file 1: '/u1/oradata/prod/system/system01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 2 - see DBWR trace file

ORA-01110: data file 2: '/u1/oradata/prod/temp/undotbs01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 3 - see DBWR trace file

ORA-01110: data file 3: '/u1/oradata/prod/system/sysaux01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 4 - see DBWR trace file

ORA-01110: data file 4: '/u1/oradata/prod/table/users01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 5 - see DBWR trace file

ORA-01110: data file 5: '/u1/oradata/prod/temp/undotbs02.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 6 - see DBWR trace file

ORA-01110: data file 6: '/u1/oradata/prod/temp/undotbs03.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 7 - see DBWR trace file

ORA-01110: data file 7: '/u1/oradata/prod/table/esis_codes_data01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 8 - see DBWR trace file

ORA-01110: data file 8: '/u1/oradata/prod/index/esis_codes_indx01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 9 - see DBWR trace file

ORA-01110: data file 9: '/u1/oradata/prod/index/esis_large_indx.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 10 - see DBWR trace file

ORA-01110: data file 10: '/u1/oradata/prod/index/esis_temp_indx.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 11 - see DBWR trace file

ORA-01110: data file 11: '/u1/oradata/prod/index/esis_indx.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 12 - see DBWR trace file

ORA-01110: data file 12: '/u1/oradata/prod/table/school_connects.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 13 - see DBWR trace file

ORA-01110: data file 13: '/u1/oradata/prod/table/esis_temp_data01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 14 - see DBWR trace file

ORA-01110: data file 14: '/u1/oradata/prod/table/esis_large_data01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 15 - see DBWR trace file

ORA-01110: data file 15: '/u1/oradata/prod/table/esis_data01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

ORA-01157: cannot identify/lock data file 16 - see DBWR trace file

ORA-01110: data file 16: '/u1/oradata/prod/table/tools01.dbf'

ORA-27086: unable to lock file - already in use

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: 8

The work around that we have found is to copy all of the files off of that mount point and then move them back.  We even tried stopping everything and umounting/remounting the file system, but it still has the same errors.

Oracle and Netapp are both pointing to other as the root of the problem.  But to me it sounds like an NFS locking issue.  Has anyone else experienced this, or have any suggestions on a better work around that having to copy the data files off and move them back?

Thanks 

-Troy