Why not just set the socket stream to non-blocking mode before calling SSL_accept(), and then block on something like select() with a timeout if SSL_accept() returns SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE? Alternatively, you can block on select() before calling SSL_accept(). Either should work. That way you can at least bound the time the connection is blocked due to the DoS like behavior/attack.

Bear in mind that SSL/TLS is record-oriented, meaning you must loop until the full record is read. SSL_pending() can help in such cases.

You can put the socket in non-blocking mode, then you ll get case SSL_ERROR_WANT_READ, or SSL_ERROR_WANT_WRITE from SSL_accept. You can then sleep a little, and try SSL_accept again. After some timeout value, you can quit and close the ssl and socket handles.

Note that this will affect all SSL operations, which means you ll need to do a similiar loop for all your read/write/shutdown calls. Basically, any call that can return WANT_READ or WANT_WRITE.

If you don t like the idea of polling, you can use select to figure out if you have data available on the socket...but that can get a bit complicated.

You can also try putting the socket back into blocking mode after the SSL_accept loop, and continue with your application.

I think code below might help others to solve the issue. It is not fully tested take it as inspiration.

  //Nonblocking SSL accept based on ACE/ace/SSL/SSL_SOCK_Acceptor.cpp
  SSL_CTX* ctx;
  ctx = initServerCTX(); // initialize SSL
  loadCertificates(ctx, certificate, privateKey); // load certs

  ...

  SSL* ssl = SSL_new(ctx); /* get new SSL state with context */
  SSL_set_fd(ssl, fd); /* set connection socket to SSL state */

  int flags = fcntl(fd, F_GETFL, 0);
  if (flags < 0)
  {
    printf("fcntl: F_GETFL 
");
    return false;
  }
  if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0)
  {
    printf("fcntl: F_SETFL 
");
    return false;
  }

  int status = -1;
  struct timeval tv, tvRestore;
  tv.tv_sec = 2;
  tv.tv_usec = 0;
  tvRestore = tv;

  fd_set writeFdSet;
  fd_set readFdSet;

  do
  {
    tv = tvRestore;
    FD_ZERO(&writeFdSet);
    FD_ZERO(&readFdSet);

    status = ::SSL_accept(ssl);
    switch (::SSL_get_error(ssl, status))
    {
    case SSL_ERROR_NONE:
      status = 0; // To tell caller about success
      break; // Done

    case SSL_ERROR_WANT_WRITE:
      FD_SET(fd, &writeFdSet);
      status = 1; // Wait for more activity
      break;

    case SSL_ERROR_WANT_READ:
      FD_SET(fd, &readFdSet);
      status = 1; // Wait for more activity
      break;

    case SSL_ERROR_ZERO_RETURN:
    case SSL_ERROR_SYSCALL:
      // The peer has notified us that it is shutting down via
      // the SSL "close_notify" message so we need to
      // shutdown, too.
      printf("Peer closed connection during SSL handshake,status:%d", status);
      status = -1;
      break;
    default:
      printf("Unexpected error during SSL handshake,status:%d", status);
      status = -1;
      break;
    }

    if (status == 1)
    {
      // Must have at least one handle to wait for at this point.
      status = select(fd + 1, &readFdSet, &writeFdSet, NULL, &tv);

      // 0 is timeout, so we re done.
      // -1 is error, so we re done.
      // Could be both handles set (same handle in both masks) so
      // set to 1.
      if (status >= 1)
      {
        status = 1;
      }
      else // Timeout or failure
      {
        printf("SSL handshake - peer timeout or failure");
        status = -1;
      }
    }

  }
  while (status == 1 && !SSL_is_init_finished(ssl));

  flags = fcntl(fd, F_GETFL, 0);
  if (flags < 0)
  {
    printf("fcntl: F_GETFL 
");
    return false;
  }
  if (fcntl(fd, F_SETFL, flags & (~O_NONBLOCK)) < 0)
  {
    printf("fcntl: F_SETFL 
");
    return false;
  }


  return (status >= 0);