httpx 1.8.0 has been released. ``` HTTPX.get("https://gitlab.com/honeyryderchuck/httpx <https://gitlab.com/honeyryderchuck/httpx>") ``` HTTPX is an HTTP client library for the Ruby programming language. Among its features, it supports: * HTTP/2 and HTTP/1.x protocol versions * Concurrent requests by default * Simple and chainable API * Proxy Support (HTTP(S), CONNECT tunnel, Socks4/4a/5) * Simple Timeout System * Lightweight by default (require what you need) And also: * Compression (gzip, deflate, brotli) * Streaming Requests * Authentication (Basic Auth, Digest Auth, AWS Sigv4) * Expect 100-continue * Multipart Requests * Cookies * HTTP/2 Server Push * H2C Upgrade * Automatic follow redirects * International Domain Names * GRPC * Circuit breaker * WebDAV * SSRF Filter * Response caching * HTTP/2 bidirectional streaming * QUERY HTTP verb * Server Sent Events * Datadog integration * Faraday integration * Webmock integration * Sentry integration Here are the updates since the last release: # 1.8.0 ## Features ### New plugins #### `:server_sent_events` plugin The `:server_sent_events` plugin provides a convenience API to deal with `text/event-stream` requests, on top of the `:stream` plugin. ```ruby session = HTTPX.plugin(:server_sent_events) sse_response = session.get("https://example.com/event-stream", event_stream: true) sse_response.each_message do |message| puts message.id puts message.event puts message.data end ``` You can read more about it in https://gitlab.com/os85/httpx/wikis/Server-Sent-Events . #### `:cache` plugin The `:cache` plugin allows caching responses. It exposes some options to determine some of its functionality, i.e. whether a request can use a cached response, whether a response can be cached, whether a cached response is still valid, etc. This functionality was extracted from the `:response_cache` plugin, which now uses it under the hood. You can read more about it in https://gitlab.com/os85/httpx/wikis/Cache . #### `:ntlm_v2_auth` plugin The `:ntlm_v2_auth` plugin is now available. It implements the most recent version of the NTLM authentication scheme supported by Microsoft products. You can read more about it in https://gitlab.com/os85/httpx/wikis/Auth#ntlm-v2-auth . ### New timeouts #### `:total_request_timeout` You can use the `:total_request_timeout` to time the time it takes a request to get its final response. This includes when your requests follows redirects (via the `:follow_redirects` plugin) or is retried multiple times (via the `:retries´ plugin). #### `:ping_timeout` Defines the number of seconds a connection waits to receive a ping response when probing for a connection for liveness. Defaults to 2 seconds. ### `:ssrf_filter` plugin new options #### `:extra_unsafe_ranges` A list of extra unsafe IPs or IP ranges to the default deny list. #### `:safe_private_ranges` A list of IPs or IP ranges which are allowed and would otherwise be denied. ### `:auth` plugin new option #### `:reset_auth_header_expires_in/at` The `:reset_auth_header_expires_in` and `:reset_auth_header_expires_at` options enable discarding an authorization token an X number of seconds after it has been generated, or at a particuar point in time, respectively. This is useful when the token is dynamically generated, so that you can preemptively renegotiate a new one. The `:oauth` plugin makes use of these fields, alongside the `expires_in` claim from token responses, to refresh the token as soon as the it expires. ### `:max_response_body_size` Can be set to the maximum number of bytes a response may have, after which it'll return an `HTTPX::ErrorResponse`. The limit is enforced based on the `content-length` header and as bytes are received. ### `:max_response_headers` Can be set to the maximum number of headers a response may have. ### `:max_response_header_value_size` Can be set to the maximum number of bytes a header value may have. In cases where a header field may be spread across multiple entries (ex. `"cookie"`), the limit is enforced on the aggregate byte size. ### resolver file cache By specifying `:file` as the resolver `:cache` option, the DNS entries will be cached in a known location in your file system. This will allow sharing entries across processes within the same machine to reduce overall DNS traffic. You can use it via: ```ruby session = HTTPX.with(resolver_options: { cache: :file }) session.get("https://example.com") ``` ## Improvements * `http-2` minimum version is now 1.2.0, which brings performance benefits around frame parsing and other common operations. ## Bugfixes * several fixes to make `httpx` usable inside a fiber scheduler in ruby 4. * `:proxy` plugin: unescape user/password from proxy options before reusing it (to avoid using percent-encoded values when p.ex. generating base64-encoding for basic auth). * `:retries` plugin: fixed the polynomial and exponential backoff `:retry_after` strategies calculation. * `:tracing` plugin: fixing span start time set up when request is sent to a closed connection, or when a long-lived connection will be probed for liveness. * datadog adapter: fixed integration with the `datadog` gem v2.34 or higher. # 1.7.8 ## Bugfixes retries: do not calculate jitter when there's no retry_after interval # 1.7.7 ## Improvements * ssl cert info will also be redacted when `:debug_redact` option is enabled. * fix performance regression of #receive_requests recent loop change (which also improved performance of previous baseline). * header/data logs indexed by request (instead of parser). ## Bugfixes * proxy: fixed issue when basic auth to proxy failed with 407 and it'd raise NoMethodError when retrying (marked as unretriable now). * selector: prevent removal of wrong selectable when previous closed via `#on_close` during selectables traversal. * http1: only reset requests which weren't completed (prevent issuing errors twice). * http1: forego clearing buffer on reset (avoid deleting potential next request bytes). * response_cache: reset cached response age on successful cache revalidation, as per RFC. * rate_limiter: do not apply jitter when delay comes from `retry-after` header. * digest: digest header parsing issues results in an error response (instead of an exception raised). # 1.7.6 ## Improvements * `datadog` adapter: support setting custom `peer.service`. * stopped doing `Thread.pass` after checking connections back into the pool. The goal of this was to allow threads potentially waiting on a connection for the same origin to have the opportunity to take it before the same thread could check the same connection out, thereby preventing starvation on limited pool sizes; this however backfired for the common case of unbounded pool sizes (the default), and was specially bad when used for multiple concurrent requests on multiple origins, where avoidable lag was introduced on the whole processing every time a connection was checked back in. Thereby, the initial problem needs to be solved at the mutex implementation of CRuby. * connection: after keep alive timeout expired, try writing `PING` frame to the socket as soon as it's emitted (previously, the frame was being buffered and the socket would be probed for readiness before writing it, whereas in most cases this is avoidable). ## Bugfixes * reuse the same resolver instance when resolving multiple concurrent requests in the same option (it was previously instantiating different ones based on option heuristics, such as when requests had different `:origin`). * on successful early name resolution (from cache or hosts file, for example), check the resolver back to the pool (the object was being dereferenced). * http1 parser: turn buffer back into a string on reset, instead of preserving its current type (in cases where it was a chunked transfer decoder and connection was kept-alive, it was keeping it around and using it to parse the next request, which broke). * http1 parser: enable pipelining based on the max concurrent requests, which the connection may have had pre-set to 1 based on prior knowledge of origin or other heuristics. * http1 parser: when disabling pipelining, mark incomplete inlined requests as idle before sending them back to the pending queue. * http1 and http2 parser: remove references to requests escalating an error; in scenarios of connection reuse (such as when using the `:persistent` plugin), these requests could be, on a transition to idle state, resent even in the absence of an external session reference, causing avoidable overhead or contention leading to timeouts or errors. * connection: rescue and ignore exceptions when calling `OpenSSL::SSL::SSLSocket#close` or `Socket#close`. * connection: when sending multiple requests into an open HTTP/2 connection which keep alive may have expired, only a single (instead of multiple, as before) `PING` frame will be sent (in the prior state, only the first `PING` was being acknowledged, which made the connection accumulate unacknowledged ping payloads). * connection: on handling errors, preventing potential race condition by copying pending requests into a local var before calling `parser.handle_error` (this may trigger a callback which calls `#reset`, which may call `#disconnect`, which would check the connection back into the pool and make it available for another thread, which could change the state of the ivar containing pending requests). * connection: skip resetting a connection when idle and having pending requests; this may happen in situation where parsers may have reset-and-back-to-idle the connection to resend failed requests as protocol (such as in the case of failed HTTP/1 pipelinining requests where incomplete requests are to be resent to the connection once pipelining is disabled). * connection: raise request timeout errors for the request the connection is currently in, instead of the connection where it was initially sent on and set timeouts on; this happens in situations where the request may be retried and sent to a different connection, which forces the request to keep a reference to the connection it's currently in (and discard it when it no longer needs it to prevent it from being GC'ed). * `:auth` plugin: when used alongside the `:retries` plugin for multiple concurrent requests, it was calling the dynamic token generator block once for each request; fixed to call the block only once per retry window, so retrying requests will reuse the same. * `:retries` plugin: retry request immediately if `:retry_after` is negative (while the option is validated for numbers, it can't control proc-based calculation). # 1.7.5 ## Improvements * `:tracing` plugin: make `Request#init_time` a UTC timestamp. ## Bugfixes * fixed handling of conditional responses which was making a batch of concurrent requests being handled serially after they failed. * `datadog` adapter: use `Request#init_time` as the span start time, which will fix the bug where the span wasn't including the time it takes to open the connection (TCP/TLS handhshakes). # 1.7.4 ## Features ### Tracing plugin A new `:tracing` plugin was introduced. It adds support for a new option, `:tracer`, which accepts an object which responds to the following callbacks: * `#enabled?(request)` - should return true or false depending on whether tracing is enabled * `#start(request)` - called when a request is about to be sent * `#finish(request, response)` - called when a response is received * `#reset(request)` - called when a request is being prepared to be resent, in cases where it makes sense (i.e. when a request is retried). You can pass chain several tracers, and callbacks will be relayed to all of them: ```ruby HTTP.plugin(:tracing).with(tracer: telemetry_platform_tracer).with(tracer: telemetry2_platform_tracer) ``` This was developed to be the foundation on top of which the datadog and OTel integrations will be built. ## Improvements * try fetching response immediately after send the request to the connection; this allows returning from errors much earlier and bug free than doing another round of waits on I/O. * when a connection is reconnected, and it was established the first time that the peer can accept only 1 request at a time, the connection will keep that informaation and keep sending requests 1 at a time afterwards. ## Bugfixes * fix regression from introducing connection post state transition callbacks, by foregoing disconnect when there's pending backlog. * transition requests to `:idle` before routing them to a different connection on merge (this could possibly leave dangling timeout callbacks otherwise). * `:brotli` plugin was integrated with the stream writer component which allows writing compressed payload in chunks. * `:brotli` plugin integrates with the `brotli` gem v0.8.0, which fixed an issue dealing with large payload responses due to the lack of support for decoding payloads in chunks. * http1 parser: reset before early returning on `Upgrade` responses (it was left in an invalid "parsing headers", which in the case of a keep-alive connection, would cause the next request to fail being parsed). * `datadog` adapter: fixed initialization of the request start time after connections were opened (it was being set to connection initialization time every time, instead of just on the first request before connection is established). * parsers: also reroute non-completed in-flight requests back to the connection so they can be retried (previously, only pending requests were). * `:proxy` plugin: do not try disconnecting unnecessarily when resetting may already do so (if conditions apply). * `:proxy` plugin: removed call to unexisting `#reset!` function. * `:proxy` plugin: also close wrapped sockets. * connection: on force_close, move connection disconnection logic below so that, if requests are reenqueued from the parser, this can be halted. * connection: when transition to `:idle`, reenqueue requests from parser before resetting it. * implement `#lazy_resolve` on resolvers, as when they're picked from the selector (instead of from the pool), they may not be wrapped by a Multi proxy. * allow resolvers transitioning from `:idle` to `:closed` and forego disconnecting when the resolver is not able to transition to `:closed` (which protects from a possible fiber scheduler context switch which changed the state under the hood). # 1.7.3 ## Improvements ### cookies plugin: Jar as CookieStore While previously an implementation detail, the cookie jar from a `:cookie` plugin-enabled session can now be manipulated by the end user: ```ruby cookies_sess = HTTPX.plugin(:cookies) jar = cookies.make_jar sess = cookies_ses.with(cookies: jar) # perform requests using sess, get/set/delete cookies in jar ``` The jar API now closely follows the [Web Cookie Store API]( https://developer.mozilla.org/en-US/docs/Web/API/CookieStore), by providing the same set of functions. Some API backwards compatibility is maintained, however since this was an internal implementation detail, this effort isn't meant to be thorough. ## Bugfixes * `http-2`: clear buffered data chunks when receiving a `GOAWAY` stream frame; without this, the client kept sending the corresponding `DATA` frames, despite the peer server making it known that it wouldn't process it. While this is valid HTTP/2, this could increase the connection window until a point where it'd go over the max frame size. this issue was observed during large file uploads where the first request could fail and make the client renegotiate. * `webmock` adapter: fixed response body length accounting which was making `response.body.empty?` return true for responses with payload. * `:rate_limiter` plugin relies on an internal refactoring to be able to wait for the time suggested by the peer server instead of the potentially relying on custom user logic via own `:retry_after`. * `:fiber_concurrency`: fix wrong names for native/system resolver overrides. * connection: fix for race condition when closing the connection, where the state only transitions to `closed` after checking the connection back in to the pool, potentially corrupting it if another session meanwhile has picked it up and manipulated it. # 1.7.2 ## Bugfixes * `:stream_bidi` plugin: when used with the `:retries` plugin, it will skip calling callbacks referencing state from the connection/stream the request was moved from. * `:auth` plugin: fix issue causing tokens to be concatenated on non-auth errors. # 1.7.1 ## Improvements * fixed timers handling in the selector loop which caused them to be traversed-for-drop twice on each tick. * connection: take proxy connecting states when transitioning to `:closing` state. * refactored name resolution cache to a cache adapter API with a default memory cache which keeps the current behaviour and will allow to add others. * a new option, `:resolver_cache`, was added, which is `:memory` by default. * this fixes an issue introduced for multi-ractor support where the cache store would not be thread-safe when used in a non main ractor. * `:auth` plugin: when loaded with the `:retries` plugin, and the auth value method is dynamic/callable, will recover out-of-the-box from 401 HTTP responses by retrying the request with a newly-generated token. * the `:rate_limiter` plugin will use this work to retry rate-limited responses without having to set setting `:retry_change_requests` to true, thereby eliminating a potential issue with non-idempotent requests. ## Bugfixes * HTTP1 parser: clear buffer on reset. * http1 fix: handle the case in `#handle_error` where the response is an error response * `:stream_bidi` plugin: fix internal state preventing bidi requests from being retried. * `:stream_bidi` plugin: will only allow initial request body being passed using `:body` param (others, like `:json`, will raise an exception) * https resolver: return `:idle` on `#state` calls when no connection is available (sometimes called in internal log messages). * selector loop fix: when there are no selectables and an interval is passed, sleep instead of returning (thereby avoiding potential busy loop).