SAP Knowledge Base Article - Preview

2884418 - HANA Indexserver crash SIGNAL 6 (SIGABRT) - Log I/O Error Detected

Symptom

The indexserver crashes with SIGNAL 6 (SIGABRT) and error messages similar to the below indications can be seen in the logfiles at the time of the crash:

/var/log/messages:

xxxxxx kernel: sd 1:0:0:21: Parameters changed
xxxxxx kernel: sd 17:0:1:21: reservation conflict
xxxxxx kernel: sd 17:0:0:20: reservation conflict
xxxxxx kernel: XFS (dm-15): metadata I/O error in "xlog_iodone" at daddr 0x2002f78a len 64 error 52
xxxxxx kernel: XFS (dm-14): metadata I/O error in "xlog_iodone" at daddr 0x113488e32 len 64 error 52
xxxxxx kernel: XFS (dm-15): xfs_do_force_shutdown(0x2) called from line 1271 of file fs/xfs/xfs_log.c. Return address = 00000000537fefdc
xxxxxx kernel: XFS (dm-15): Log I/O Error Detected. Shutting down filesystem
xxxxxx kernel: XFS (dm-14): xfs_do_force_shutdown(0x2) called from line 1271 of file fs/xfs/xfs_log.c. Return address = 00000000537fefdc
xxxxxx kernel: XFS (dm-15): Please unmount the filesystem and rectify the problem(s)
xxxxxx kernel: XFS (dm-14): Log I/O Error Detected. Shutting down filesystem
xxxxxx kernel: XFS (dm-14): Please unmount the filesystem and rectify the problem(s)

++++++++++++++++++++++++++++

xxxxxx kernel: [4187157.293752] blk_update_request: I/O error, dev dm-12, sector 1736195056
xxxxxx kernel: [4187157.293759] blk_update_request: I/O error, dev dm-12, sector 1335519232
xxxxxx kernel: [4187157.293777] XFS (dm-6): metadata I/O error: block 0x7f6d6b10 ("xlog_iodone") error 5 numblks 64
xxxxxx kernel: [4187157.293783] XFS (dm-6): xfs_do_force_shutdown(0x2) called from line 1219 of file ../fs/xfs/xfs_log.c. Return address = 0xffffffffa06b7efd
xxxxxx kernel: [4187157.293886] XFS (dm-6): Log I/O Error Detected. Shutting down filesystem
xxxxxx kernel: [4187157.293888] XFS (dm-6): Please umount the filesystem and rectify the problem(s)
xxxxxx kernel: [4187157.293892] XFS (dm-6): xfs_log_force: error -5 returned.
xxxxxx kernel: [4187157.293924] Buffer I/O error on dev dm-6, logical block 217016456, lost async page write

++++++++++++++++++++++++++++

xxxxxx kernel: attempt to access beyond end of device
xxxxxx kernel: dm-21: rw=4096, want=17179869184, limit=9234179687
xxxxxx kernel: XFS (dm-21): last sector read failed

Crashdump

indexserver_xxxxxx.33003.crashdump.xxxxxx.trc
[CRASH_STACK] Stacktrace of crash: (xxxxxx Local)
----> Pending exceptions (possible root cause) <----
--- Pending assertion:
exception 1: no.2000008 (Basis/IO/FileAccess/impl/LocalFileCompletionThread.cpp:320) TID: 163221
Error during asynchronous file transfer (io_getevents), rc=5: Input/output error; $fileCallback$=[W] , buffer= 0x00007efd5f5bb000, offset= 1488687988736, size= 0/65536, file= "<root>/datavolume_0000.dat" ((open, mode= RW, access= rw-rw-r--, flags= ASYNC|DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/data/xxxxxx/mnt00001/hdb00003.00003/" (access= rw-rw-r--, flags= AUTOCREATE_PATH, usage= DATA, fs= xfs, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=256,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 10 (10/10)}

++++++++++++++++++++++++++++

indexserver_xxxxxx.32003.crashdump.xxxxxx.trc
[CRASH_STACK] Stacktrace of crash: (xxxxxx Local)
----> Pending exceptions (possible root cause) <----
--- Pending assertion:
exception 1: no.2000008 (Basis/IO/FileAccess/impl/LocalFileCompletionThread.cpp:320) TID: 94301
Error during asynchronous file transfer (io_getevents), rc=52: Invalid exchange; $fileCallback$=[W] , buffer= 0x00007f299c194000, offset= 62750720, size= 0/4096, file= "<root>/datavolume_0000.dat" ((open, mode= RW, access= rw-rw-r--, flags= ASYNC|DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/data/xxxxxx/mnt00001/hdb00003.00003/" (access= rw-rw-r--, flags= AUTOCREATE_PATH, usage= DATA, fs= xfs, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=128,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 10 (10/10)}

++++++++++++++++++++++++++++

[STACK_SHORT] Stacktrace of crash:

1: 0x00007fe1046e8dd9 in FileAccess::LocalFileIOControlBlock::reportError(char const*, unsigned long, char const*, int)+0x35 at LocalFileCallback.cpp:347 (libhdbbasis.so)
2: 0x00007fe1046edfd9 in FileAccess::LocalFileCompletionThread::run(void*&)+0xe05 at LocalFileCompletionThread.cpp:320 (libhdbbasis.so)
3: 0x00007fe1047334e8 in Execution::Thread::staticMainImp(Execution::Thread*)+0x534 at Thread.cpp:529 (libhdbbasis.so)
4: 0x00007fe104722798 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x164 at ThreadInterposition.cpp:633 (libhdbbasis.so)
5: 0x00007fe103eb42de in start_thread+0xfa (libpthread.so.0)
6: 0x00007fe1026b1a63 in __GI___clone+0x40 (libc.so.6)
0: gsignal + 0x10f
1: _crashImpl(char const*, int, char const*, rte::const_exception_ptr) + 0x13a
2: Basis::crashImpl(char const*, int, ltt::bad_alloc const&) + 0x10
3: Basis::crashImpl(char const*, int, ltt::exception const&) + 0xe
4: Basis::asyncCrashImpl(char const*, int, ltt::exception const&) + 0x9
5: PageAccess::PageFlushCallback::transferError(PageAccess::PageIOCallback::Status const&, ltt::exception const&) + 0x37
6: PageAccess::PageIOCallback::transferError(FileAccess::FileCallback::Status const&, ltt::exception const&) + 0x24
7: FileAccess::LocalFileIOControlBlock::reportError(char const*, unsigned long, char const*, int) + 0x1b0
8: FileAccess::LocalFileCompletionThread::run(void*&) + 0xe09
9: Execution::Thread::staticMainImp(Execution::Thread*) + 0x538
10: Execution::pthreadFunctionWrapper(Execution::Thread*) + 0x168
11: start_thread + 0xfe
12: __GI___clone + 0x40


Read more...

Environment

  • Linux
  • Any hardware
  • Any application

Product

SAP HANA, platform edition 2.0

Keywords

hp, hewlett packard, dell, lenovo, cisco, fujitsu, hitachi, huawei, nec, super micro computer, vce, , KBA , HAN-DB , SAP HANA Database , Problem

About this page

This is a preview of a SAP Knowledge Base Article. Click more to access the full version on SAP ONE Support launchpad (Login required).

Search for additional results

Visit SAP Support Portal's SAP Notes and KBA Search.