SAP Knowledge Base Article - Preview

2185955 - SAP HANA startup hangs due to orphan nameserver process

Symptom

The startup of SAP HANA hangs.

One name server trace file contains errors like:

Responder.cpp(00568) : can't listen on port <ip_address>:3<inst_id>01: host unknown
Responder.cpp(00568) : can't listen on port <ip_address>:3<inst_id>02: host unknown

In scale-out landscapes other name server trace files contain errors like the following when trying to contact the above nameserver:

Channel.cpp(00362): active channel <id> from <id> to <ip_address>:3<inst_id>01: reading failed with timeout error; timeout=60000ms elapsed
BufferedIO.cpp(01151) : erroneous channel <id> from <id> to <id_address>:30101: read from channel failed; resetting buffer
30524[thr=137308]: TopologyReplicator at
 1: 0x00007fbf57f787d4 in Execution::ContextFunctions::dumpInfo(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool, bool, bool, bool, bool)+0x2c0 at ContextFunctions.cpp:629 (libhdbbasis.so)
 2: 0x00007fbf598d0db4 in TrexNet::BufferedIO::fillBuffer(bool)+0x450 at BufferedIO.cpp:1152 (libhdbbasement.so)
 3: 0x00007fbf598d4baa in TrexNet::BufferedIO::peekBuffer()+0x46 at BufferedIO.cpp:1326 (libhdbbasement.so)
 4: 0x00007fbf598d4d52 in TrexNet::BufferedNetInput::peek()+0x30 at BufferedIO.cpp:4331 (libhdbbasement.so)
 5: 0x000000000041e5fa in TrexNet::Request::in()+0x36 at Request.h:392 (hdbnameserver)
 6: 0x00007fbf5937f430 in NameServer::TNSInfo::sendRequestTo(NameServer::Request const&, NameServer::Response&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, unsigned short, NameServer::TrexNetRequestHolder*)+0x110 at TNSClient.cpp:782 (libhdbcsbase.so)
 7: 0x00007fbf5937ff5a in NameServer::TNSClient::sendRequestTo(NameServer::Request const&, NameServer::Response&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, unsigned short, NameServer::TrexNetRequestHolder*)+0x26 at TNSClient.cpp:1021 (libhdbcsbase.so)
 8: 0x00007fbf5af98c18 in NameServer::TopologyReplicator::processRequests()+0x6d4 at Replicator.cpp:477 (libhdbns.so)
 9: 0x00007fbf5af99795 in NameServer::TopologyReplicator::run(void*)+0x61 at Replicator.cpp:410 (libhdbns.so)
10: 0x00007fbf599261f2 in TrexThreads::PoolThread::run()+0x850 at PoolThread.cpp:265 (libhdbbasement.so)
11: 0x00007fbf59927d58 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:124 (libhdbbasement.so)
12: 0x00007fbf57fc4ccf in Execution::Thread::staticMainImp(void**)+0x98b at Thread.cpp:475 (libhdbbasis.so)
13: 0x00007fbf57fc521d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:545 (libhdbbasis.so)
[137308]{-1}[-1/-1] 2015-06-27 04:28:12.312881 e TNS              TNSClient.cpp(00800) : sendRequest storetrees to <host_name>:30101 failed with NetException. data=
TopologyReplicator(<host>:30<inst_id>1) req <id> failed with error 5521, discarding request: method setstarting
TopologyReplicator(<host>:30<inst_id>1) req <id> failed with error 5521, discarding request: method storetrees

A thread of type "Assign" may be visible waiting for 'open data+log volume <id> / local tables'.

In the call stacks one thread may be visible waiting for a request sent to the nameserver:

8977[thr=137398]: Assign at
 1: 0x00007fa116dbebc6 in __GI___poll+0x62 (libc.so.6)
 2: 0x00007fa13555836d in SiPoll+0x19 at siux.cpp:3554 (libhdbbasement.so)
 3: 0x00007fa1355592ce in SiPeek+0x7a at siux.cpp:3657 (libhdbbasement.so)
 4: 0x00007fa135520cb0 in NiIPeek(NITAB*, unsigned char, int)+0x100 at nixxi.cpp:5619 (libhdbbasement.so)
 5: 0x00007fa135521481 in NiIRead(NITAB*, unsigned char*, unsigned int, int, unsigned char, unsigned int*)+0xf0 at nixxi.cpp:4819 (libhdbbasement.so)
 6: 0x00007fa1350cb5b8 in comm::read(void*, void*, int, int)+0x1d4 at commlib.cpp:1754 (libhdbbasement.so)
 7: 0x00007fa135167167 in TrexNet::Channel::receive(void*, int, int&)+0x2d3 at Channel.cpp:349 (libhdbbasement.so)
 8: 0x00007fa1351509ea in TrexNet::BufferedIO::fillBuffer(bool)+0x86 at BufferedIO.cpp:1144 (libhdbbasement.so)
 9: 0x00007fa135154baa in TrexNet::BufferedIO::peekBuffer()+0x46 at BufferedIO.cpp:1326 (libhdbbasement.so)
10: 0x00007fa135154d52 in TrexNet::BufferedNetInput::peek()+0x30 at BufferedIO.cpp:4331 (libhdbbasement.so)
11: 0x000000000046876a in TrexNet::Request::in()+0x36 at Request.h:392 (hdbindexserver)
12: 0x00007fa134bff430 in NameServer::TNSInfo::sendRequestTo(NameServer::Request const&, NameServer::Response&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, unsigned short, NameServer::TrexNetRequestHolder*)+0x110 at TNSClient.cpp:782 (libhdbcsbase.so)
13: 0x00007fa134c0149f in NameServer::TNSInfo::sendRequest(NameServer::Request const&, NameServer::Response&)+0x10b at TNSClient.cpp:711 (libhdbcsbase.so)
14: 0x00007fa134c01c4d in NameServer::TNSInfo::processRequest(NameServer::Request const&, NameServer::Response&)+0xe9 at TNSClient.cpp:673 (libhdbcsbase.so)
15: 0x00007fa134c02131 in NameServer::TNSClient::processRequest(NameServer::Request const&, NameServer::Response&)+0x40 at TNSClient.cpp:1045 (libhdbcsbase.so)
16: 0x00007fa134c2e5ab in NameServer::TNSClient::deleteIndex(TrexBase::IndexName const&)+0x167 at TNSClient.cpp:1730 (libhdbcsbase.so)
17: 0x00007fa13741fc20 in TRexAPI::DDL::DropTable::handle()+0x100 at DDL_DropTable.cpp:109 (libhdbcsapi.so)
18: 0x00007fa13741aaba in TRexAPI::DDLRequest::execute()+0x276 at DDLRequest.cpp:110 (libhdbcsapi.so)
19: 0x00007fa13741ffe0 in TRexAPI::DDL::DropTable::execute()+0x100 at DDL_DropTable.cpp:64 (libhdbcsapi.so)
20: 0x00007fa136f3884c in TRexAPI::TRexApiAdmin::deleteIndex(bool, bool)+0x1f8 at TRexApiAdmin.cpp:607 (libhdbcsapi.so)
21: 0x00007fa11a43d957 in ptime::TrexMD::deleteIndex(bool, bool) const+0x33 at trex_metadata.cc:287 (libhdbcswrapper.so)
22: 0x00007fa11c09b82f in ptime::QueryExecutor::drop_trex_table(ptime::ClassPtr<ptime::TableInfo>, bool, bool)+0xa2b at qx_trex.cc:6573 (libhdbrskernel.so)
23: 0x00007fa11bfb8949 in ptime::QueryExecutor::drop_table(ptime::qp_drop_table const*, ptime::ClassPtr<ptime::TableInfo>, ptime::ClassPtr<ptime::TableInfo>)+0xee5 at qx_table.cc:3193 (libhdbrskernel.so)
24: 0x00007fa11bfb9129 in ptime::QueryExecutor::drop_table(ptime::qp_drop_table const*, ptime::ClassPtr<ptime::TableInfo>)+0xd5 at qx_table.cc:3039 (libhdbrskernel.so)
25: 0x00007fa11bf9571e in ptime::QueryExecutor::execute(ptime::Transaction&, char const*, ptime::qp_parse_tree const*, ptime::Query::Plan*, ptime::Query::result_t*)+0x119a at qx.cc:234 (libhdbrskernel.so)
26: 0x00007fa11c9a72d0 in ptime::Query::execute(ptime::Transaction&, char const*, ptime::Query::param_t*, ptime::Query::Plan*, ptime::Query::result_t*, ptime::Statement*, bool)+0x290 at query.cc:522 (libhdbrskernel.so)
27: 0x000000000044e572 in TRexAPI::deleteTable(Newdb::Transaction&, TrexTrace::Tracer&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&)+0xa0 at TREXIndexServer.cpp:1155 (hdbindexserver)
28: 0x00000000004513e2 in TRexAPI::TREXIndexServer::createTableBoePortalSearch(NameServer::TNSClient&)+0x240 at TREXIndexServer.cpp:1274 (hdbindexserver)
29: 0x000000000045956d in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0x489 at TREXIndexServer.cpp:895 (hdbindexserver)
30: 0x000000000048b938 in TRexAPI::AssignThread::run(void*)+0x34 at TREXIndexServer.cpp:408 (hdbindexserver)
31: 0x00007fa1351a61f2 in TrexThreads::PoolThread::run()+0x850 at PoolThread.cpp:265 (libhdbbasement.so)
32: 0x00007fa1351a7d58 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:124 (libhdbbasement.so)
33: 0x00007fa12a6d0ccf in Execution::Thread::staticMainImp(void**)+0x98b at Thread.cpp:475 (libhdbbasis.so)
34: 0x00007fa12a6d121d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:545 (libhdbbasis.so)

Read more...

Environment

SAP HANA

Product

SAP HANA, platform edition all versions

Keywords

old nameserver process running; , KBA , HAN-DB , SAP HANA Database , Problem

About this page

This is a preview of a SAP Knowledge Base Article. Click more to access the full version on SAP ONE Support launchpad (Login required).

Search for additional results

Visit SAP Support Portal's SAP Notes and KBA Search.