httpx 1.5.0 has been released. ``` HTTPX.get("https://gitlab.com/honeyryderchuck/httpx <https://gitlab.com/honeyryderchuck/httpx>") ``` HTTPX is an HTTP client library for the Ruby programming language. Among its features, it supports: * HTTP/2 and HTTP/1.x protocol versions * Concurrent requests by default * Simple and chainable API * Proxy Support (HTTP(S), CONNECT tunnel, Socks4/4a/5) * Simple Timeout System * Lightweight by default (require what you need) And also: * Compression (gzip, deflate, brotli) * Streaming Requests * Authentication (Basic Auth, Digest Auth, AWS Sigv4) * Expect 100-continue * Multipart Requests * Cookies * HTTP/2 Server Push * H2C Upgrade * Automatic follow redirects * International Domain Names * GRPC * Circuit breaker * WebDAV * SSRF Filter * Response caching * HTTP/2 bidirectional streaming * QUERY HTTP verb * Datadog integration * Faraday integration * Webmock integration * Sentry integration Here are the updates since the last release: # 1.5.0 ## Features ### `:stream_bidi` plugin The `:stream_bidi` plugin enables bidirectional streaming support (an HTTP/2 only feature!). It builds on top of the `:stream` plugin, and uses its block-based syntax to process incoming frames, while allowing the user to pipe more data to the request (from the same, or another thread/fiber). ```ruby http = HTTPX.plugin(:stream_bidi) request = http.build_request( "POST", "https://your-origin.com/stream", headers: { "content-type" => "application/x-ndjson" }, body: ["{\"message\":\"started\"}\n"] ) chunks = [] response = http.request(request, stream: true) Thread.start do response.each do |chunk| handle_data(chunk) end end # now send data... request << "{\"message\":\"foo\"}\n" request << "{\"message\":\"bar\"}\n" # ... ``` You can read more about it in https://honeyryderchuck.gitlab.io/httpx/wiki/Stream-Bidi ### `:query` plugin The `:query` plugin adds public methods supporting the `QUERY` HTTP verb: ```ruby http = HTTPX.plugin(:query) http.query("https://example.com/gquery", body: "foo=bar") # QUERY /gquery .... ``` You can read more about it in https://honeyryderchuck.gitlab.io/httpx/wiki/Query this functionality was added as a plugin for explicit opt-in, as it's experimental (RFC for the new HTTP verb is still in draft). ### `:response_cache` plugin filesystem based store The `:response_cache` plugin supports setting the filesystem as the response cache store (instead of just storing them in memory, which is the default `:store`). ```ruby # cache store in the filesystem, writes to the temporary directory from the OS http = HTTPX.plugin(:response_cache, response_cache_store: :file_store) # if you want a separate location http = HTTPX.plugin(:response_cache).with(response_cache_store: HTTPX::Plugins::ResponseCache::FileStore.new("/path/to/dir")) ``` You can read more about it in https://honeyryderchuck.gitlab.io/httpx/wiki/Response-Cache#:file_store ### `:close_on_fork` option A new option `:close_on_fork` can be used to ensure that a session object which may have open connections will not leak them in case the process is forked (this can be the case of `:persistent` plugin enabled sessions which have add usage before fork): ```ruby http = HTTPX.plugin(:persistent, close_on_fork: true) # http may have open connections here fork do # http has no connections here end ``` You can read more about it in https://honeyryderchuck.gitlab.io/httpx/wiki/Connection-Pools#Fork-Safety . ### `:debug_redact` option The `:debug_redact` option will, when enabled, replace parts of the debug logs (enabled via `:debug` and `:debug_level` options) which may contain sensitive information, with the `"[REDACTED]"` placeholder. You can read more about it in https://honeyryderchuck.gitlab.io/httpx/wiki/Debugging . ### `:max_connections` pool option A new `:max_connections` pool option (settable under `:pool_options`) can be used to defined the maximum number **overall** of connections for a pool ("in-transit" or "at-rest"); this complements, and supersedes when used, the already existing `:max_connections_per_origin`, which does the same per connection origin. ```ruby HTTPX.with(pool_options: { max_connections: 100 }) ``` You can read more about it in https://honeyryderchuck.gitlab.io/httpx/wiki/Connection-Pools . ### Subplugins An enhancement to the plugins architecture, it allows plugins to define submodules ("subplugins") which are loaded if another plugin is in use, or is loaded afterwards. You can read more about it in https://honeyryderchuck.gitlab.io/httpx/wiki/Custom-Plugins#Subplugins . ## Improvements * `:persistent` plugin: several improvements around reconnections of failure: * reconnections will only happen for "connection broken" errors (and will discard reconnection on timeouts) * reconnections won't exhaust retries * `:response_cache` plugin: several improements: * return cached response if not stale, send conditional request otherwise (it was always doing the latter). * consider immutable (i.e. `"Cache-Control: immutable"`) responses as never stale. * `:datadog` adapter: decorate spans with more tags (header, kind, component, etc...) * timers operations have been improved to use more efficient algorithms and reduce object creation. ## Bugfixes * ensure that setting request timeouts happens before the request is buffered (the latter could trigger a state transition required by the former). * `:response_cache` plugin: fix `"Vary"` header handling by supporting a new plugin option, `:supported_vary_headers`, which defines which headers are taken into account for cache key calculation. * fixed query string encoded value when passed an empty hash to the `:query` param and the URL already contains query string. * `:callbacks` plugin: ensure the callbacks from a session are copied when a new session is derived from it (via a `.plugin` call, for example). * `:callbacks` plugin: errors raised from hostname resolution should bubble up to user code. * fixed connection coalescing selector monitoring in cases where the coalescable connecton is cloned, while other branches were simplified. * clear the connection write buffer in corner cases where the remaining bytes may be interpreted as GOAWAY handshake frame (and may cause unintended writes to connections already identified as broken). * remove idle connections from the selector when an error happens before the state changes (this may happen if the thread is interrupted during name resolution). ## Chore `httpx` makes extensive use of features introduced in ruby 3.4, such as `Module#set_temporary_name` for otherwise plugin-generated anonymous classes (improves debugging and issue reporting), or `String#append_as_bytes` for a small but non-negligible perf boost in buffer operations. It falls back to the previous behaviour when used with ruby 3.3 or lower. Also, and in preparation for the incoming ruby 3.5 release, dependency of the `cgi` gem (which will be removed from stdlib) was removed. # 1.4.4 ## Improvements * `:stream` plugin: response will now be partially buffered in order to i.e. inspect response status or headers on the response body without buffering the full response * this fixes an issue in the `down` gem integration when used with the `:max_size` option. * do not unnecessarily probe for connection liveness if no more requests are inflight, including failed ones. * when using persistent connections, do not probe for liveness right after reconnecting after a keep alive timeout. ## Bugfixes * `:persistent` plugin: do not exhaust retry attempts when probing for (and failing) connection liveness. * since the introduction of per-session connection pools, and consequentially due to the possibility of multiple inactive connections for the same origin being in the pool, which may have been terminated by the peer server, requests would fail before being able to establish a new connection. * prevent retrying to connect the TCP socket object when an SSLSocket object is already in place and connecting. # 1.4.3 ## Bugfixes * `webmock` adapter: reassign headers to signature after callbacks are called (these may change the headers before virtual send). * do not close request (and its body) right after sending, instead only on response close * prevents retries from failing under the `:retries` plugin * fixes issue when using `faraday-multipart` request bodies * retry request with HTTP/1 when receiving an HTTP/2 GOAWAY frame with `HTTP_1_1_REQUIRED` error code. * fix wrong method call on HTTP/2 PING frame with unrecognized code. * fix EOFError issues on connection termination for long running connections which may have already been terminated by peer and were wrongly trying to complete the HTTP/2 termination handshake. # 1.4.2 ## Bugfixes * faraday: use default reason when none is matched by Net::HTTP::STATUS_CODES * native resolver: keep sending DNS queries if the socket is available, to avoid busy loops on select * native resolver fixes for Happy Eyeballs v2 * do not apply resolution delay if the IPv4 IP was not resolved via DNS * ignore ALIAS if DNS response carries IP answers * do not try to query for names already awaiting answer from the resolver * make sure all types of errors are propagated to connections * make sure next candidate is picked up if receiving NX_DOMAIN_NOT_FOUND error from resolver * raise error happening before any request is flushed to respective connections (avoids loop on non-actionable selector termination). * fix "NoMethodError: undefined method `after' for nil:NilClass", happening for requests flushed into persistent connections which errored, and were retried in a different connection before triggering the timeout callbacks from the previously-closed connection. ## Chore * Refactor of timers to allow for explicit and more performant single timer interval cancellation. * default log message restructured to include info about process, thread and caller. # 1.4.1 ## Bugfixes * several `datadog` integration bugfixes * only load the `datadog` integration when the `datadog` sdk is loaded (and not other gems that may define the `Datadog` module, like `dogstatsd`) * do not trace if datadog integration is loaded but disabled * distributed headers are now sent along (when the configuration is enabled, which it is by default) * fix for handling multiple `GOAWAY` frames coming from the server (node.js servers seem to send multiple frames on connection timeout) * fix regression for when a url is used with `httpx` which is not `http://` or `https://` (should raise `HTTPX::UnsupportedSchemaError`) * worked around `IO.copy_stream` which was emitting incorrect bytes for HTTP/2 requests which bodies larger than the maximum supported frame size. * multipart requests: make sure that a body declared as `Pathname` is opened for reading in binary mode. * `webmock` integration: ensure that request events are emitted (such as plugins and integrations relying in it, such as `datadog` and the OTel integration) * native resolver: do not propagate successful name resolutions for connections which were already closed. * native resolver: fixed name resolution stalling, in a multi-request to multi-origin scenario, when a resolution timeout would happen. ## Chore * refactor of the happy eyeballs and connection coalescing logic to not rely on callbacks, and instead on instance variable management (makes code more straightforward to read).