Mercurial > hgrepos > Python > libs > ConfigMix

annotate configmix/extras/aws.py @ 283:503768f91a05

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
More granular configuration of retrieving AWS metadata: retries with backoff setting
author Franz Glasner <fzglas.hg@dom66.de>
date Mon, 07 Dec 2020 01:51:24 +0100
parents da1596034954
children 4aaf74858d07
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
282
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 # :Copyright: (c) 2015-2020, Franz Glasner. All rights reserved.
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 # :License: 3-clause BSD. See LICENSE.txt for details.
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 """AWS namespace implementation
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8 """
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
10 from __future__ import division, absolute_import, print_function
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
11
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
12
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
13 __all__ = []
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
14
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
15
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
16 import requests
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
17 import requests.exceptions
283
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
18 import requests.adapters
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
19 import urllib3
282
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
21
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
22 _MARKER = object()
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
23
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
24
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
25 URL_META_INSTANCEID = "http://169.254.169.254/latest/meta-data/instance-id"
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
26 URL_META_REGION = "http://169.254.169.254/latest/meta-data/placement/region"
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
27 URL_META_AVAILABILITY_ZONE = "http://169.254.169.254/latest/meta-data/availability-zone"
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
28 URL_DYN_INSTANCE_IDENTITY_DOC = "http://169.254.169.254/latest/dynamic/instance-identity/document"
283
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
29 TIMEOUT = 2
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
30 # See https://gist.github.com/doublenns/7e3e4b72df4aaeccbeabf87ba767f44e
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
31 RETRIES = urllib3.Retry(total=3, backoff_factor=0.3)
282
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
32
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
33 _meta_instanceid = None
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
34 _meta_region = None
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
35 _meta_availability_zone = None
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
36 _dyn_instance_identity_doc = None
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
37
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
38
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
39 def _get_text_req(url):
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
40 with requests.Session() as sess:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
41 try:
283
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
42 a = requests.adapters.HTTPAdapter(max_retries=RETRIES)
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
43 sess.mount("http://", a)
282
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
44 resp = sess.get(url, timeout=TIMEOUT)
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
45 resp.raise_for_status()
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
46 return resp.text
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
47 except requests.exceptions.RequestException:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
48 return _MARKER
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
49
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
50
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
51 def _get_json_req(url):
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
52 with requests.Session() as sess:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
53 try:
283
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
54 a = requests.adapters.HTTPAdapter(max_retries=RETRIES)
503768f91a05 More granular configuration of retrieving AWS metadata: retries with backoff setting
Franz Glasner <fzglas.hg@dom66.de>
parents: 282
diff changeset
55 sess.mount("http://", a)
282
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
56 resp = sess.get(url, timeout=TIMEOUT)
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
57 resp.raise_for_status()
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
58 return resp.json()
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
59 except requests.exceptions.RequestException:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
60 return _MARKER
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
61 except ValueError:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
62 # JSON error
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
63 return _MARKER
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
64
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
65
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
66 def _get_meta_instanceid():
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
67 global _meta_instanceid
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
68 if _meta_instanceid is None:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
69 _meta_instanceid = _get_text_req(URL_META_INSTANCEID)
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
70 return _meta_instanceid
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
71
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
72
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
73 def _get_meta_region():
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
74 global _meta_region
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
75 if _meta_region is None:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
76 _meta_region = _get_text_req(URL_META_REGION)
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
77 return _meta_region
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
78
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
79
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
80 def _get_meta_avzone():
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
81 global _meta_availability_zone
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
82 if _meta_availability_zone is None:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
83 _meta_availability_zone = _get_text_req(URL_META_AVAILABILITY_ZONE)
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
84 return _meta_availability_zone
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
85
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
86
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
87 def _get_dyn_instance_identity_doc():
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
88 global _dyn_instance_identity_doc
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
89 if _dyn_instance_identity_doc is None:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
90 _dyn_instance_identity_doc = _get_json_req(
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
91 URL_DYN_INSTANCE_IDENTITY_DOC)
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
92 return _dyn_instance_identity_doc
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
93
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
94
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
95 def _awslookup(name, default=_MARKER):
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
96 if name == "metadata.instance-id":
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
97 v = _get_meta_instanceid()
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
98 elif name == "metadata.placement.region":
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
99 v = _get_meta_region()
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
100 elif name == "metadata.placement.availability-zone":
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
101 v = _get_meta_avzone()
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
102 elif name.startswith("dynamic.instance-identity."):
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
103 idoc = _get_dyn_instance_identity_doc()
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
104 if idoc is _MARKER:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
105 v = _MARKER
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
106 else:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
107 v = idoc.get(name[26:], _MARKER)
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
108 else:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
109 v = _MARKER
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
110 if v is _MARKER:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
111 if default is _MARKER:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
112 raise KeyError("key %r not found in the AWS namespace" % name)
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
113 else:
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
114 return default
da1596034954 Implemented an "AWS" namespace to retrieve some AWS-specific metadata
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
115 return v