Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segfault in TLS #1696

Closed
EricTheOne opened this issue May 13, 2015 · 9 comments
Closed

segfault in TLS #1696

EricTheOne opened this issue May 13, 2015 · 9 comments
Labels
tls Issues and PRs related to the tls subsystem.

Comments

@EricTheOne
Copy link

Reproduces 100% of the time within a minute of running the server. Server opens multiple TLS client connections (as a client).

I'll gladly help with testing a fix or sharing some of the backtrace data, but cannot share server code or specific scenario.

Please see nodejs/node-v0.x-archive#8780, I'm not 100% sure but I think it may be the same issue. I'm unable to reproduce it in node 0.12.2 (server is running fine, the segfault doesn't happen).

Backtrace (commit 7693705):

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000069eed0 in SSL_read ()
(gdb) backtrace
#0  0x000000000069eed0 in SSL_read ()
#1  0x0000000000d40a39 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d4117e in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000ce109b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00002307b4e060bb in ?? ()
#7  0x00003c26fce690f9 in ?? ()
#8  0x00002307b4e06001 in ?? ()
#9  0x00007ffe6b7e47f0 in ?? ()
#10 0x00007ffe6b7e4850 in ?? ()
#11 0x00002307b4e99739 in ?? ()
#12 0x00002bb54efbee61 in ?? ()
#13 0x00003c26fce6b669 in ?? ()
...

dmesg:

segfault at 30 ip 000000000069eed0 sp 00007ffe6b7e41a8 error 4 in iojs[400000+d0e000]

Although nodejs/node-v0.x-archive#8780 contains a suggested fix, I don't see that it was applied in node.js 0.12.2, so either something else fixed it or the situation is different.

Naively applying the fix suggested in 8780 on io.js 2.0.1 (18d457b) doesn't seem to help. Instead I'm failing on an assertion.

"Fix" (tls_wrap.cc):

if (wrap->enc_out_) {
    NodeBIO::FromBIO(wrap->enc_out_)->Read(nullptr, wrap->write_size_);
    // Ensure that the progress will be made and `InvokeQueued` will be called.
    wrap->ClearIn();
}

Backtrace with fix applied:

Program terminated with signal SIGABRT, Aborted.
#0  0x00007fc17f768cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  0x00007fc17f768cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fc17f76c0d8 in __GI_abort () at abort.c:89
#2  0x00007fc17f761b86 in __assert_fail_base (fmt=0x7fc17f8b2830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xf7d1f9 "(bio->ptr) != (nullptr)", file=file@entry=0xf7d1e0 "../src/node_crypto_bio.h", line=line@entry=77, 
    function=function@entry=0xf7e080 <_ZZN4node7NodeBIO7FromBIOEP6bio_stE19__PRETTY_FUNCTION__> "static node::NodeBIO* node::NodeBIO::FromBIO(BIO*)") at assert.c:92
#3  0x00007fc17f761c32 in __GI___assert_fail (assertion=0xf7d1f9 "(bio->ptr) != (nullptr)", file=0xf7d1e0 "../src/node_crypto_bio.h", line=77, 
    function=0xf7e080 <_ZZN4node7NodeBIO7FromBIOEP6bio_stE19__PRETTY_FUNCTION__> "static node::NodeBIO* node::NodeBIO::FromBIO(BIO*)") at assert.c:101
#4  0x0000000000673989 in node::NodeBIO::FromBIO(bio_st*) [clone .isra.22] [clone .part.23] ()
#5  0x0000000000d409bf in node::TLSWrap::EncOutCb(node::WriteWrap*, int) ()
#6  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#7  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#8  0x00003993517060bb in ?? ()
#9  0x00007fff1da78730 in ?? ()
#10 0x0000399351706001 in ?? ()
#11 0x00007fff1da78710 in ?? ()
#12 0x00007fff1da78778 in ?? ()
#13 0x0000399351a8292c in ?? ()
#14 0x00001507279beb79 in ?? ()
#15 0x0000000000000000 in ?? ()
@EricTheOne
Copy link
Author

@indutny

@mscdex mscdex added the tls Issues and PRs related to the tls subsystem. label May 13, 2015
indutny added a commit to indutny/io.js that referenced this issue May 14, 2015
When loading session, OCSP response, SNI, always check that the
`self._handle` is present. If it is not - the socket was closed - and we
should emit the error instead of throwing an uncaught exception.

Fix: nodejs/node-v0.x-archive#8780
Fix: nodejs#1696
@indutny
Copy link
Member

indutny commented May 14, 2015

May I ask you to give a try to the following patch #1702 ?

@EricTheOne
Copy link
Author

@indutny gladly and thanks for the quick response. Applied the patch on io.js master (4e2f999), still getting segfault:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000069eed0 in SSL_read ()
(gdb) backtrace
#0  0x000000000069eed0 in SSL_read ()
#1  0x0000000000d40ae9 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d4122e in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000ce109b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x0000000000850002 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087721b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00000abcd52060bb in ?? ()
#7  0x00000abcd5206001 in ?? ()
#8  0x00007fffb979c580 in ?? ()
#9  0x00007fffb979c5d0 in ?? ()
#10 0x00000abcd55a6f1a in ?? ()
#11 0x000030f334efeea9 in ?? ()
#12 0x000015aab64b2489 in ?? ()
#13 0x00001bd0488c0841 in ?? ()
#14 0x000030f334efeea9 in ?? ()
#15 0x00000b493ed87e89 in ?? ()
#16 0x00001bd04889a081 in ?? ()
#17 0x00007fffb979c6a8 in ?? ()
#18 0x00000abcd5296468 in ?? ()
#19 0x000015aab64b2489 in ?? ()
#20 0x00001bd0488addf1 in ?? ()
#21 0x00000abcd54c8824 in ?? ()
#22 0x00000abcd54c8824 in ?? ()
#23 0x00003d23e40be3c1 in ?? ()
#24 0x00002f4a2ebf7a29 in ?? ()
#25 0x00000a9d72d66921 in ?? ()
#26 0x000015aab64b2489 in ?? ()
#27 0x000011d9d60bb679 in ?? ()
#28 0x00000a9d72dab0a9 in ?? ()
#29 0x00000a9d72d8dcf1 in ?? ()
#30 0x00007fff00000001 in ?? ()
#31 0x00000abc00000002 in ?? ()
#32 0x00007fffb979c700 in ?? ()
#33 0x000011d9d60bb631 in ?? ()
#34 0x00000a9d72d04141 in ?? ()
#35 0x00000a9d72d04161 in ?? ()
#36 0x00000b493ed87e89 in ?? ()
#37 0x00000a9d72d04101 in ?? ()
#38 0x00001bd0488c06f9 in ?? ()
#39 0x00000a9d72d04161 in ?? ()
#40 0x00000a9d72dc9181 in ?? ()
#41 0x000011d9d607ccf9 in ?? ()
#42 0x00000a9d72dc9181 in ?? ()
#43 0x000011d9d607ccf9 in ?? ()
#44 0x00007fffb979c6e0 in ?? ()
#45 0x00000abcd5214b35 in ?? ()
#46 0x000012917282dcf1 in ?? ()
#47 0x00001bd0488addf1 in ?? ()
#48 0x0000000200000000 in ?? ()
#49 0x00000a9d72dc9181 in ?? ()
#50 0x0000000a00000000 in ?? ()
#51 0x00007fffb979c778 in ?? ()
#52 0x00000abcd5621a9f in ?? ()
#53 0x000015aab64b2489 in ?? ()
#54 0x000012917282dcf1 in ?? ()
#55 0x00001bd0488addf1 in ?? ()
#56 0x00000a9d72d04101 in ?? ()
#57 0x00007fffb979c758 in ?? ()
#58 0x000030f334e1c389 in ?? ()
#59 0x00007fffb979c800 in ?? ()
#60 0x0000000000ac2c48 in v8::internal::Isolate::RunMicrotasks() ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

@EricTheOne
Copy link
Author

By the way, as I was testing the patch npm seemed to fail installing four different modules.
I'm running configure && make && make install on the io.js repo, anything I'm missing to get npm to work?

> [email protected] install /.../node_modules/heapdump
> node-gyp rebuild

gyp WARN install got an error, rolling back install
gyp ERR! configure error 
gyp ERR! stack Error: 404 status code downloading tarball
gyp ERR! stack     at Request.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:246:14)
gyp ERR! stack     at emitOne (events.js:82:20)
gyp ERR! stack     at Request.emit (events.js:169:7)
gyp ERR! stack     at Request.onRequestResponse (/usr/local/lib/node_modules/npm/node_modules/request/request.js:1156:10)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! stack     at ClientRequest.emit (events.js:169:7)
gyp ERR! stack     at HTTPParser.parserOnIncomingClient (_http_client.js:414:21)
gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:304:20)
gyp ERR! stack     at emitOne (events.js:77:13)
gyp ERR! System Linux 3.xx.x-xx-generic
gyp ERR! command "/usr/local/bin/iojs" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /.../node_modules/heapdump
gyp ERR! node -v v2.0.2
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok 


indutny added a commit to indutny/io.js that referenced this issue May 18, 2015
* Destroy `SSL*` and friends on a next tick to make sure that we are not
  doing it in one of the OpenSSL callbacks
* Add more checks to the C++ methods that might be invoked during
  destructor's pending queue cleanup

Fix: nodejs/node-v0.x-archive#8780
Fix: nodejs#1696
@EricTheOne
Copy link
Author

@indutny I've tested the changes on 2.1.0 as well, still segfaults:

#0  0x000000000069f060 in SSL_read ()
#1  0x0000000000d31499 in node::TLSWrap::ClearOut() ()
#2  0x0000000000d31bce in node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) ()
#3  0x0000000000cd285b in node::JSStream::ReadBuffer(v8::FunctionCallbackInfo<v8::Value> const&) ()
#4  0x000000000084d342 in v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#5  0x000000000087457b in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()

Any way I can help with more info?

@indutny
Copy link
Member

indutny commented May 26, 2015

@EricTheOne I commented about it on other issue. I really need a test case that reproduces the problem. It doesn't look like I fully understand the problem atm.

indutny added a commit to indutny/io.js that referenced this issue May 29, 2015
When loading session, OCSP response, SNI, always check that the
`self._handle` is present. If it is not - the socket was closed - and we
should emit the error instead of throwing an uncaught exception.

Fix: nodejs/node-v0.x-archive#8780
Fix: nodejs#1696
indutny added a commit to indutny/io.js that referenced this issue May 29, 2015
* Destroy `SSL*` and friends on a next tick to make sure that we are not
  doing it in one of the OpenSSL callbacks
* Add more checks to the C++ methods that might be invoked during
  destructor's pending queue cleanup

Fix: nodejs/node-v0.x-archive#8780
Fix: nodejs#1696
indutny added a commit to indutny/io.js that referenced this issue Jun 1, 2015
When loading session, OCSP response, SNI, always check that the
`self._handle` is present. If it is not - the socket was closed - and we
should emit the error instead of throwing an uncaught exception.

Fix: nodejs/node-v0.x-archive#8780
Fix: nodejs#1696
indutny added a commit to indutny/io.js that referenced this issue Jun 1, 2015
* Destroy `SSL*` and friends on a next tick to make sure that we are not
  doing it in one of the OpenSSL callbacks
* Add more checks to the C++ methods that might be invoked during
  destructor's pending queue cleanup

Fix: nodejs/node-v0.x-archive#8780
Fix: nodejs#1696
indutny added a commit that referenced this issue Jun 4, 2015
When loading session, OCSP response, SNI, always check that the
`self._handle` is present. If it is not - the socket was closed - and we
should emit the error instead of throwing an uncaught exception.

Fix: nodejs/node-v0.x-archive#8780
Fix: #1696
PR-URL: #1702
Reviewed-By: Trevor Norris <[email protected]>
indutny added a commit that referenced this issue Jun 4, 2015
* Destroy `SSL*` and friends on a next tick to make sure that we are not
  doing it in one of the OpenSSL callbacks
* Add more checks to the C++ methods that might be invoked during
  destructor's pending queue cleanup

Fix: nodejs/node-v0.x-archive#8780
Fix: #1696
PR-URL: #1702
Reviewed-By: Trevor Norris <[email protected]>
@indutny
Copy link
Member

indutny commented Jun 6, 2015

@EricTheOne PTAL at #1910, I think it should fix the issue for you ;)

indutny added a commit to indutny/io.js that referenced this issue Jun 30, 2015
Queued write requests should be invoked on handle close, otherwise the
"consumer" might be already destroyed when the write callbacks of the
"consumed" handle will be invoked.

Fix: nodejs#1696
indutny added a commit that referenced this issue Jul 1, 2015
Queued write requests should be invoked on handle close, otherwise the
"consumer" might be already destroyed when the write callbacks of the
"consumed" handle will be invoked. Same applies to the shutdown
requests.

Make sure to "move" away socket from server to not break the
`connections` counter in `net.js`. Otherwise it might not call `close`
callback, or call it too early.

Fix: #1696
PR-URL: #1910
Reviewed-By: Trevor Norris <[email protected]>
@Fishrock123
Copy link
Contributor

Ala #1910 (comment), @EricTheOne did that patch fix this issue? :)

mscdex pushed a commit to mscdex/io.js that referenced this issue Jul 9, 2015
Queued write requests should be invoked on handle close, otherwise the
"consumer" might be already destroyed when the write callbacks of the
"consumed" handle will be invoked. Same applies to the shutdown
requests.

Make sure to "move" away socket from server to not break the
`connections` counter in `net.js`. Otherwise it might not call `close`
callback, or call it too early.

Fix: nodejs#1696
PR-URL: nodejs#1910
Reviewed-By: Trevor Norris <[email protected]>
@Fishrock123
Copy link
Contributor

I'm going to assume it did. Please re-open if it didn't.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tls Issues and PRs related to the tls subsystem.
Projects
None yet
4 participants