net/http hangs (timeouts) since 9.2.7.0 #6078
Description
the issue was reported at MinIO's tracker but it seems to be a JRuby bug (given that it also occurs against other similar S3-like backends e.g. Ceph). esp. since MRI works as well as JRuby 9.2.6.0 (JRuby 9.2.7.0 seems to be the first to exhibit problems)
reproducer is a bit demanding, one needs MinIO setup + aws gem (might try dissecting later) :
gem install aws-sdk -v 2.11.436
docker run -p 9000:9000 minio/minio server /data
visit http://localhost:9000/minio/ and create a bucket: 'test'
require 'aws-sdk'
file_path = ARGV[0]
bucket = ARGV[1] || 'test'
admin = password = 'minioadmin'
aws_options_hash = {
endpoint: 'http://localhost:9000',
access_key_id: admin,
secret_access_key: password,
region: 'us-east-1',
force_path_style: true,
http_wire_trace: true
}
bucket = Aws::S3::Bucket.new(bucket, aws_options_hash)
obj = bucket.object(File.basename(file_path))
obj.upload_file(file_path, {})NOTE: the issue reproduces with bigger files that are split into multipart upload, reproduction is not 100% (one needs 2-3 tries) :
jruby -v aws_minio.rb lib/target/jruby-stdlib-9.2.10.0-SNAPSHOT.jar test
initially thought this might be related to a stdlib update but there was none in 9.2.7.0
(only in 9.2.6.0 - which is working: 7b964fd#diff-07931640fbf96ee2b8fc616794b0a314)
also noticed the net/http patches aws does are not there in their 3.x line, but I tried disabling their patching and the issue still occurs.
connection(s) will tend to hang and raise timeouts (aws seems to do auto-retry by default) :
read 0 bytes
Conn keep-alive
Conn close because of error Net::ReadTimeout
Conn close because of error Net::ReadTimeout
Conn close because of error Net::ReadTimeout
Conn close because of error Net::ReadTimeout
opening connection to localhost:9000...
opening connection to localhost:9000...
here's a failure stack using JRuby 9.2.9.0 :
```
2020-02-18T15:49:55.472+01:00 [main] INFO TraceType : Exception raised: Net::ReadTimeout : Net::ReadTimeout
2020-02-18T15:49:55.473+01:00 [main] INFO TraceType : Backtrace generated:
rbuf_fill at /opt/local/rvm/rubies/jruby-9.2.9.0/lib/ruby/stdlib/net/protocol.rb:181
readuntil at /opt/local/rvm/rubies/jruby-9.2.9.0/lib/ruby/stdlib/net/protocol.rb:157
readline at /opt/local/rvm/rubies/jruby-9.2.9.0/lib/ruby/stdlib/net/protocol.rb:167
read_status_line at /opt/local/rvm/rubies/jruby-9.2.9.0/lib/ruby/stdlib/net/http/response.rb:40
read_new at /opt/local/rvm/rubies/jruby-9.2.9.0/lib/ruby/stdlib/net/http/response.rb:29
new_transport_request at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/patches.rb:29
catch at org/jruby/RubyKernel.java:1193
new_transport_request at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/patches.rb:26
request at /opt/local/rvm/rubies/jruby-9.2.9.0/lib/ruby/stdlib/net/http.rb:1474
request at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/connection_pool.rb:330
transmit at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/handler.rb:72
session at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/handler.rb:121
session_for at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/connection_pool.rb:96
session at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/handler.rb:119
transmit at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/handler.rb:71
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/net_http/handler.rb:45
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/plugins/content_length.rb:12
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_request_signer.rb:88
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_request_signer.rb:23
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_host_id.rb:14
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/xml/error_handler.rb:8
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_request_signer.rb:65
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_redirects.rb:15
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/retry_errors.rb:108
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_dualstack.rb:32
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_accelerate.rb:49
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_md5s.rb:31
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_iad_regional_endpoint.rb:31
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/rest/handler.rb:7
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/user_agent.rb:12
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/plugins/endpoint.rb:41
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/param_validator.rb:21
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/plugins/raise_response_errors.rb:14
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_dualstack.rb:24
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/s3_accelerate.rb:34
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/idempotency_token.rb:18
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/aws-sdk-core/plugins/param_converter.rb:20
call at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/plugins/response_target.rb:21
send_request at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/request.rb:70
put_object at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-core-2.11.436/lib/seahorse/client/base.rb:207
put_object at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-resources-2.11.436/lib/aws-sdk-resources/services/s3/file_uploader.rb:42
open_file at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-resources-2.11.436/lib/aws-sdk-resources/services/s3/file_uploader.rb:49
put_object at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-resources-2.11.436/lib/aws-sdk-resources/services/s3/file_uploader.rb:41
upload at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-resources-2.11.436/lib/aws-sdk-resources/services/s3/file_uploader.rb:34
upload_file at /opt/local/rvm/gems/jruby-9.2.9.0@logstash/gems/aws-sdk-resources-2.11.436/lib/aws-sdk-resources/services/s3/object.rb:252
```