SAP Knowledge Base Article - Preview

2730077 - Indexserver Crash with Error during asynchronous file transfer

Symptom

  • You observe SAP HANA indexserver crash with SIGNAL 6 (SIGABRT);
  • The file indexserver_<hostname>.<port>.crashdump.YYYYMMDD-HHMMSS.ffffff.trc contains in the CRASH_STACK session an exception similar to the following:
    • Crash Stack 1:
      [CRASH_STACK] Stacktrace of crash: (2020-07-19 23:59:59 728 Local)
      ----> Pending exceptions (possible root cause) <----
      --- Pending assertion:
      exception 1: no.2000008 (Basis/IO/FileAccess/impl/LocalFileCompletionThread.cpp:320) TID: 000000
      Error during asynchronous file transfer (io_getevents), rc=5: Input/output error; $fileCallback$=[W] , buffer= 0x00007ff1b8aa1000, offset= 4266209280, size= 0/4194304, file= "<root>/logsegment_000_00000002.dat" ((open, mode= RW, access= rw-rw-r--, flags= ASYNC|DIRECT|LOCK|LAZY_OPEN), factory= (root= "/hana/log/<SID>/mnt00005/hdb00008.00003/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= LOG, fs= xfs, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=64,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 10 (10/10)}
      1: 0x00007ff34876c1f9 in FileAccess::LocalFileIOControlBlock::reportError(char const*, unsigned long, char const*, int)+0x35 at LocalFileCallback.cpp:347 (libhdbbasis.so)
      2: 0x00007ff34876ef99 in FileAccess::LocalFileCompletionThread::run(void*&)+0xe05 at LocalFileCompletionThread.cpp:320 (libhdbbasis.so)
      3: 0x00007ff3487b4b48 in Execution::Thread::staticMainImp(Execution::Thread*)+0x534 at Thread.cpp:529 (libhdbbasis.so)
      4: 0x00007ff3487a479d in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x169 at ThreadInterposition.cpp:636 (libhdbbasis.so)
      5: 0x00007ff347f30dd5 in start_thread+0xc1 (libpthread.so.0)
      6: 0x00007ff3467a502d in __clone+0x69 (libc.so.6)
    • Crash Stack 2:
      [CRASH_STACK] Stacktrace of crash: (2020-07-04 03:25:34 823 Local)
      ----> Pending exceptions (possible root cause) <----
      --- Pending assertion:
      exception 1: no.2000008 (Basis/IO/FileAccess/impl/LocalFileCompletionThread.cpp:290) TID: 00000
      Error during asynchronous file transfer (io_getevents), rc=5: Input/output error; $fileCallback$=[W] , buffer= 0x00002abf71cfc000, offset= 651657216, size= 0/36864, file= "<root>/logsegment_000_00001104.dat" ((open, mode= RW, access= rw-rw-r--, flags= DIRECT|MUST_EXIST|LOCK|LAZY_OPEN), factory= (root="/sapmnt/log/PB0/mnt00002/hdb00005.00003/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= LOG, fs= nfs4, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=64,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 10 (10/10)}
      exception throw location:
      1: 0x00002abb3acc1efe in FileAccess::LocalFileIOControlBlock::reportError(char const*, unsigned long, char const*, int)+0x4a at LocalFileCallback.cpp:335 (libhdbbasis.so)
      2: 0x00002abb3acc21dc in FileAccess::LocalFileIOControlBlock::handleReturnCode(char const*, unsigned long, char const*, int)+0x168 at LocalFileCallback.cpp:130 (libhdbbasis.so)
      3: 0x00002abb3acc33dc in FileAccess::LocalFileCompletionThread::requestSuccessful(io_event const&, FileAccess::LocalFileIOControlBlock*)+0x68 at LocalFileCompletionThread.cpp:290 (libhdbbasis.so)
      4: 0x00002abb3acc3719 in FileAccess::LocalFileCompletionThread::getNextFinishedRequests()+0x165 at LocalFileCompletionThread.cpp:264 (libhdbbasis.so)
      5: 0x00002abb3acc396d in FileAccess::LocalFileCompletionThread::getNextFinishedRequest()+0x49 at LocalFileCompletionThread.cpp:198 (libhdbbasis.so)
      6: 0x00002abb3acc4428 in FileAccess::LocalFileCompletionThread::run(void*&)+0x34 at LocalFileCompletionThread.cpp:96 (libhdbbasis.so)
      7: 0x00002abb3ac77f26 in Execution::Thread::staticMainImp(void**)+0x412 at Thread.cpp:537 (libhdbbasis.so)
      8: 0x00002abb3ac79ad8 in Execution::Thread::staticMain(void*)+0x34 at ThreadMain.cpp:26 (libhdbbasis.so)
    • Crash Stack 3:
      [CRASH_STACK] Stacktrace of crash: (2020-07-04 08:25:34 870 Local)
      ----> Pending exceptions (possible root cause) <----
      --- Pending assertion:
      exception 1: no.2000008 (Basis/IO/FileAccess/impl/LocalFileCompletionThread.cpp:320) TID: 94301
      Error during asynchronous file transfer (io_getevents), rc=52: Invalid exchange; $fileCallback$=[W] , buffer= 0x00007f299c194000, offset= 62750720, size= 0/4096, file= "<root>/datavolume_0000.dat" ((open, mode= RW, access= rw-rw-r--, flags= ASYNC|DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/data/<SID>>/mnt00001/hdb00003.00003/" (access= rw-rw-r--, flags= AUTOCREATE_PATH, usage= DATA, fs= xfs, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=128,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 10 (10/10)}
      $res$=
      LogicalPage@lpcb=0x00007f27a8c14600 [0x640ab99L@DefaultPageAccess, FlushPPageNo=0x600000003bd8P [0x3bd8000,4k,0], ConvPPageNo=0x600000003bd8P [0x3bd8000,4k,0]][Manage=1/ExternalCached=0/Disp=Shortterm/InternalShortInRecovery=0/Mod=0/Del=0/isToBeDel=0/Loading=0/needsBarrierForFlush=0/useBarrierForFlush=0/isInFlushQueue=0/isInIO=1/Ref=1/IterRef=0/Check=2], Page= dictionary={block={logical={page={savepoint=1493317,version=1,page_type=UnifiedTableDictionary,checksum_algo=CRC32,delete_mark=bytes(1)[
      0x00000000: 23 #
      ],converter_type=Default,size=4k,checksum=3108899865,flush_counter=0,reserved=0},owner=0xfe02da26be,pageno=0x640ab99L,nextpageno=0x640ab98L},capacity=4096,used=4096,index=5,reserved=0},blocks={blocks={dummy=0}}}
      exception throw location:
      1: 0x00007fe1046e8dd9 in FileAccess::LocalFileIOControlBlock::reportError(char const*, unsigned long, char const*, int)+0x35 at LocalFileCallback.cpp:347 (libhdbbasis.so)
      2: 0x00007fe1046edfd9 in FileAccess::LocalFileCompletionThread::run(void*&)+0xe05 at LocalFileCompletionThread.cpp:320 (libhdbbasis.so)
      3: 0x00007fe1047334e8 in Execution::Thread::staticMainImp(Execution::Thread*)+0x534 at Thread.cpp:529 (libhdbbasis.so)
      4: 0x00007fe104722798 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x164 at ThreadInterposition.cpp:633 (libhdbbasis.so)
      5: 0x00007fe103eb42de in start_thread+0xfa (libpthread.so.0)
      6: 0x00007fe1026b1a63 in __GI___clone+0x40 (libc.so.6)


Read more...

Environment

SAP HANA, platform edition

Product

SAP HANA, platform edition all versions

Keywords

crash_stack, crashed, short_stack, short, crashing, restarted, down, shutdown, exited, failed, hardware, log io, I/O, reservation conflict, FAILED, failover, kernel, xfs, nfs4, fs, dm, dm, rc=5, rc=52, LocalFileCompletionThread, SIGABRT, SIGNAL6, signals, error detected , KBA , HAN-DB-BAC , SAP HANA Backup & Recovery , HAN-DB-ENG , SAP HANA DB Engines , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , HAN-DB-PER , SAP HANA Database Persistence , Problem

About this page

This is a preview of a SAP Knowledge Base Article. Click more to access the full version on SAP ONE Support launchpad (Login required).

Search for additional results

Visit SAP Support Portal's SAP Notes and KBA Search.