[python{,-experimental}] Obey floating point timeouts provided to RESTClientObject.request(...) (!7154) · Merge requests · OpenAPI Tools / openapi-generator

Merged Administrator requested to merge github/fork/cjgibson/cjgibson/allow-float-timeout into master Aug 07, 2020

Created by: cjgibson

This PR allows passing of a fractional number of seconds as a timeout when making network requests through the RESTClientObject. Currently timeout durations provided as a float are silently ignored instead of being enforced:

RESTClientObject timeout behavior

>>> # Google uses the OpenAPI generator for their Python client
>>> from kubernetes.client.configuration import Configuration
>>> from kubernetes.client.rest import RESTClientObject
>>> rest_client_object = RESTClientObject(Configuration())
>>>
>>> # unexpectedly returns a response
>>> rest_client_object.request('GET', 'http://neverssl.com', _request_timeout=1e-9)
<kubernetes.client.rest.RESTResponse object at 0x7fd6a968cbd0>
>>>
>>> # fails with a timeout error, as we'd expect
>>> rest_client_object.request('GET', 'http://neverssl.com', _request_timeout=(1e-9, 1e-9))
Traceback (most recent call last):
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fd6a96a5050>, 'Connection to neverssl.com timed out. (connect timeout=1e-09)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/kubernetes/client/rest.py", line 214, in request
    headers=headers)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/request.py", line 76, in request
    method, url, fields=fields, headers=headers, **urlopen_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/request.py", line 97, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/poolmanager.py", line 336, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='neverssl.com', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd6a96a5050>, 'Connection to neverssl.com timed out. (connect timeout=1e-09)'))

The underlying call to urllib3.PoolManager.request will happily accept a float timeout duration, as shown:

urllib3 timeout behavior

>>> # urllib3's PoolManager provides the underlying impelementation for
>>> # OpenAPI's RESTClientObject
>>> from urllib3 import PoolManager, Timeout
>>>
>>> # the following is what we'd expect when passing 1e-9 as our _request_timeout
>>> pool_manager = PoolManager()
>>> pool_manager.request('GET', 'http://neverssl.com', timeout=Timeout(total=1e-9))
Traceback (most recent call last):
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fd6a96a5790>, 'Connection to neverssl.com timed out. (connect timeout=1e-09)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/request.py", line 76, in request
    method, url, fields=fields, headers=headers, **urlopen_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/request.py", line 97, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/poolmanager.py", line 336, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/cjgibson/miniconda3/envs/Users.cjgibson.code.rapid/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='neverssl.com', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd6a96a5790>, 'Connection to neverssl.com timed out. (connect timeout=1e-09)'))

PR checklist

Read the contribution guidelines.
Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
If contributing template-only or documentation-only changes which will change sample output, build the project beforehand.
Run the shell script ./bin/generate-samples.shto update all Petstore samples related to your fix. This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master. These must match the expectations made by your contribution. You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*. For Windows users, please run the script in Git BASH.
File the PR against the correct branch: master
Copy the technical committee to review the pull request if your PR is targeting a particular programming language.

Fixes: https://github.com/kubernetes-client/python/pull/1069

@spacether