Thank you for noticing the issue and filing the PR.

Regarding your concern, I think that for completeness we need to wait for the write to complete.

Otherwise, it is my understanding is that this assert would fail if the user of the socket does not immediately close the connection. Note that we do not require the user to close the connection immediately if TLS handshake fails with an alert.

FWIW, the principles behind are:

• there can be at most one write in-flight
  • this means that it is preferable to wait for the write completion of the last TLS handshake message before telling the application that the handshake is complete
• the only way to cancel a write is to close the socket
  • we could close the socket internally, but that is not what we are doing now
• only the user can instruct the socket to close

Does that make sense to you?

A Go test case:

package main

import (
	"crypto/tls"
	"log"
	"os"
)

func main() {
	cfg := &tls.Config{
		InsecureSkipVerify: true,
		CipherSuites:       []uint16{tls.TLS_RSA_WITH_RC4_128_SHA}, // an unsupported cipher by the server
	}

	c, err := tls.Dial("tcp", os.Args[1], cfg)
	if err != nil {
		log.Fatal(err)
	}
	c.Close()
}

If you save it to a file like tls.go, you can run it with go run tls.go <ip>:<port>

Against another web server, the output is:

2018/10/05 08:57:46 remote error: tls: handshake failure

Against h2o:

2018/10/05 12:56:34 EOF

@kazuho thanks for the feedback will take a look at the potentially problematic path (libh2o mostly I take).

@joeshaw thanks! openssl s_client also fails to provide a reason for the failure if there is merely a TCP closure (as expected).

Interestingly, I do not see the error when using master branch (currently at 27f04d8) with openssl 1.1.0d.

I see the alert being sent when the TLS connection is being closed, specifically in the following call trace:

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 4.1
    frame #0: 0x0000000100112ed2 h2o`write_core(fd=18, bufs=0x00007fff5fbfeb80, bufcnt=0x00007fff5fbfeb78, first_buf_written=0x00007fff5fbfeb60) at evloop.c.h:167
  * frame #1: 0x000000010010fedf h2o`do_write(_sock=0x0000000127604f30, bufs=0x0000000126a53010, bufcnt=1, cb=(h2o`shutdown_ssl at socket.c:361)) at evloop.c.h:275
    frame #2: 0x0000000100110335 h2o`flush_pending_ssl(sock=0x0000000127604f30, cb=(h2o`shutdown_ssl at socket.c:361)) at socket.c:306
    frame #3: 0x000000010010f479 h2o`shutdown_ssl(sock=0x0000000127604f30, err=0x0000000000000000) at socket.c:396
    frame #4: 0x000000010010e46a h2o`h2o_socket_close(sock=0x0000000127604f30) at socket.c:464
    frame #5: 0x0000000100178b66 h2o`on_ssl_handshake_complete(sock=0x0000000127604f30, err="ssl handshake failure") at util.c:346
    frame #6: 0x00000001001142c7 h2o`on_handshake_complete(sock=0x0000000127604f30, err="ssl handshake failure") at socket.c:1019
    frame #7: 0x0000000100111c66 h2o`proceed_handshake(sock=0x0000000127604f30, err="ssl handshake failure") at socket.c:1177
    frame #8: 0x0000000100113386 h2o`read_on_ready(sock=0x0000000127604f30) at evloop.c.h:248
    frame #9: 0x000000010011315d h2o`run_socket(sock=0x0000000127604f30) at evloop.c.h:517
    frame #10: 0x000000010010e9f8 h2o`run_pending(loop=0x00000001215cef50) at evloop.c.h:555
    frame #11: 0x000000010010e4d9 h2o`h2o_evloop_run(loop=0x00000001215cef50, max_wait=2147483647) at evloop.c.h:602
    frame #12: 0x000000010018069a h2o`run_loop(_thread_index=0x0000000000000000) at main.c:1661
    frame #13: 0x000000010017f92e h2o`main(argc=0, argv=0x00007fff5fbff568) at main.c:2263
    frame #14: 0x00007fff91093235 libdyld.dylib`start + 1

Apparently, the behavior of openssl has changed somewhere between 1.1.0d and 1.1.0g.

Previously, SSL_shutdown was emitting a TLS alert that contains the handshake error number when the TLS handshake failed due to some reason. The function returned 0 in such case. However, the same function in recent versions of OpenSSL seem to not emit anything in such case, and returns -1.

Current approach of H2O is to repeatedly call SSL_shutdown until it returns -1, when h2o_socket_close is called, expecting that the first call to SSL_shutdown emits the correct error and the subsequent call will return -1. The approach seems very wired.

The PR changes the approach to emit the alert when handshake fails. It also calls SSL_shutdown when h2o_socket_close is called. The call to SSL_shutdown is a bit scary, but we are at least not sending two alerts for the time being. And even if we do, it would not cause any interoperability issue, because the client will handle the first alert and terminate the TLS handshake.

To conclude; let's apply the fix, add tests, and revisit the issue when something breaks.