Connections can get stuck when upgrading to SSL

[cody-friedrichsen]

Environment

• hackney: 4.7.2
• Erlang/OTP: 26

Description

I observed a behavior in an integration environment where the default connection pool become fully occupied by connections which were stuck attempting to upgrade to SSL. Within hackney_con:connected there is a call to ssl:connect/2 which results in utilizing the default timeout of infinity. If an issue occurs during the TLS handshake, the connection process remains in a waiting state and attempts to utilize the connection time out repeatedly as the call to hackney:maybe_register_h2/6 must pull the protocol from the state of the pooled connection which times out.

Is it possible to implement a concept of tls upgrade timeout or tls handshake timeout which can be optionally passed to an ssl:connect/3 call in the upgrade to TLS path?

More Details

Hackney Connection Supervisor has 4 linked processes in the pool and appears in a healthy state

[{registered_name,hackney_conn_sup},
 {current_function,{gen_server,loop,7}},
 {initial_call,{proc_lib,init_p,5}},
 {status,waiting},
 {message_queue_len,0},
 {links,[<0.663122.0>,<0.663147.0>,<0.663279.0>,<0.663134.0>,
         <0.1568.0>]},
 {dictionary,[{'$ancestors',[hackney_sup,<0.1567.0>]},
              {'$initial_call',{supervisor,hackney_conn_sup,1}}]},
 {trap_exit,true},
 {error_handler,error_handler},
 {priority,normal},
 {group_leader,<0.1566.0>},
 {total_heap_size,2208},
 {heap_size,1598},
 {stack_size,11},
 {reductions,865131},
 {garbage_collection,[{max_heap_size,#{error_logger => true,include_shared_binaries => false,
                                       kill => true,size => 0}},
                      {min_bin_vheap_size,46422},
                      {min_heap_size,233},
                      {fullsweep_after,65535},
                      {minor_gcs,35}]},
 {suspending,[]}]

One of the connections in the pool is inspected and is found in the waiting state with two messages in the queue. Connection is waiting on ssl handshake to complete

23> process_info(<0.663122.0>).
[{current_function,{gen,do_call,4}},
 {initial_call,{proc_lib,init_p,5}},
 {status,waiting},
 {message_queue_len,2},
 {links,[<0.1570.0>]},
 {dictionary,[{'$ancestors',[hackney_conn_sup,hackney_sup,
                             <0.1567.0>]},
              {'$initial_call',{hackney_conn,init,1}}]},
 {trap_exit,true},
 {error_handler,error_handler},
 {priority,normal},
 {group_leader,<0.1566.0>},
 {total_heap_size,3203},
 {heap_size,1598},
 {stack_size,47},
 {reductions,14839},
 {garbage_collection,[{max_heap_size,#{error_logger => true,include_shared_binaries => false,
                                       kill => true,size => 0}},
                      {min_bin_vheap_size,46422},
                      {min_heap_size,233},
                      {fullsweep_after,65535},
                      {minor_gcs,9}]},
 {suspending,[]}]

26> process_info(<0.663122.0>, [current_stacktrace]).
[{current_stacktrace,[{gen,do_call,4,
                           [{file,"gen.erl"},{line,240}]},
                      {gen_statem,call,3,[{file,"gen_statem.erl"},{line,923}]},
                      {ssl_gen_statem,call,2,
                                      [{file,"ssl_gen_statem.erl"},{line,1372}]},
                      {ssl_gen_statem,handshake,2,
                                      [{file,"ssl_gen_statem.erl"},{line,253}]},
                      {ssl_gen_statem,connect,8,
                                      [{file,"ssl_gen_statem.erl"},{line,222}]},
                      {ssl,connect,3,[{file,"ssl.erl"},{line,637}]},
                      {hackney_conn,connected,3,
                                    [{file,"/codebuild/output/src3735438855/src/mirror/_build/default/lib/hackney/src/hackney_conn.erl"},
                                     {line,909}]},
                      {gen_statem,loop_state_callback,11,
                                  [{file,"gen_statem.erl"},{line,1395}]}]}]

At this point in time I am not exactly sure what failure occurred to cause the handshake attempt to fail in this way so I can not reliably reproduce this issue. Regardless of the failure scenario I believe a timeout in this location would allow configuration of some safety rails against whatever kind of failures could happen.